 Hi. Welcome to using Dev Space to usher in an era of peace for our developers. My name is Ratsim and Rava Chandran, in short, Raj. I'm a software engineer that specializes in cloud computing. I started my career as a site reliability engineer where I manage Kubernetes clusters and ensure the readiness of the systems. And then I slowly pivoted towards developer experience. I'm here to talk about the developer experience journey and how we improved them last year at my previous company. So these are the following developer challenges that I'll be talking about in this session. And we really got to know about these developer challenges by talking to the developers, putting out surveys in our organization, trying to get the results regarding how they feel about our development clusters. And we really tried to understand from their point of view. And I've marked them over a happiness-over-time graph so that we can showcase the developer experience journey and how we address each of these point points and how it changes over time. So starting from the top, we have shared development environment in our cluster. And that makes it really painful for our developers to test their changes in the environment. They also face the problems with local environment setup hassles that they have to do a lot of complex setting up of their tools and packages to get things started in their local environment. Combined with those two pain points, they end up taking a longer cycle time to develop tests and push code to production. And finally, they also tend to face a lot of problems with setting up the environment for running end-to-end tests in our system. Now, a quick show of hands. How many of you have faced one or more of these problems at your organization? It's good to know that. So let's start off with the first pain point of our environment of shared development cluster. And before I start off with that, I want to give a brief background of our developer environment. So we have our Kubernetes development cluster that runs on Kubernetes. And our development cluster is shared with 100 developers in our organization. And we have main or core services that are deployed on one namespace in our development cluster, that says core namespace, and their names service A, B, and C. They are core services that power our main applications. And most, if not all, the developers use these services in their day-to-day work. And there are also other non-critical and non-core services that are deployed in other namespaces. Now, taking a deeper dive on the core namespace, we have the services deployed as Kubernetes deployments. And they use the main branch for stability reasons. And we have a Kubernetes ingress, which has the endpoints, devcluster, slash, SVC, A, B, and C. And that's what developers use to access the services. And with that being said, let's take a scenario where a developer wants to deploy a test branch on this development cluster. A developer deploys on, say, service C. They deploy a test branch. But since test branch is not stable, it breaks the service C. And it becomes unavailable not just for one developer, but all the developers in our organization. And as you can see, as you can imagine, that caused a lot of friction with all the developers in our organization. One of the solutions that our developers came up with was to use Google calendars. They would simply create a time slot just for themselves for the entire development cluster, and they would test the changes. And developers would often, they may override deployments, or they may not follow this Google calendar. They might forget, and they might override the deployments. And that really causes a lot of friction in our development cluster. And the developer experience team took a look at that, and we felt like, what is going on with this Google calendar? And we took a look. We talked with the developers. We understood their pain points, what their expectations, what their needs were. And we looked at potential solutions, and we came across Dev Space. Quick show of hands. Does anyone know about Dev Space? Cool, quite a bit. That's good. I'll give a brief overview of Dev Space. So Dev Space is a CNCF project created by Loft Labs. It's an open source tool. And it can be considered like a good analogy would be, it's a Docker compose for Kubernetes. Essentially, it's a command line tool that allows you to develop and deploy applications on Kubernetes clusters. It provides a lot of functionalities like abstracting the Kubernetes commands. For example, port forwarding, logging, really abstracts a lot of Kubernetes functionality such that developers don't need to remember those Kubernetes commands that runs underneath the hood. They would simply use Dev Space command to use port forwarding or logging or any other functionalities that they would use it in their day-to-day work. So that really reduces the cognitive overload for their developers. So in this diagram, we have a Dev Space configuration. The command line tool will simply build the image and deploy to the development cluster using Helm or Kube-CTL. Now, how do we use Dev Space to address our shared development environment problem? We implemented something dedicated ephemeral testing environments using a feature of Dev Space called Dev Space Deploy, where the developer will simply create a unique namespace for themselves on the development cluster and deploy their changes on that environment. And we implemented Dev Space Deploy for our core services. And Dev Space Deploy also has a dependency feature such that the developers can deploy multiple services such that they can simply use it on one repository and use dependency feature to deploy multiple services rather than going to different services and deploying it. So here, a developer would simply clone the repository, create a unique namespace using Create Space, and simply run the command Dev Space Deploy to test their branch. So let's revisit the scenario where a developer deploys service B and C on a unique namespace. They would create a namespace, say developer X, and they would deploy their branch, Dev X branch, on service B, and they use dependency feature to deploy service C as well. They also get a unique Ingress object, which is devx.devcluster.svc, B and C, so that they can access their environment using that object. And they can share that URL with other developers as well for collaboration purposes. In hindsight, we pretty much created a way for each developer to create their own namespaces and deploy their own changes. So they're not stepping over each other or overriding deployments on the development cluster. So what does that bring us? So one of the main benefits is no more use of Google calendars. Nobody needs to block off time slots for accessing the development cluster. It also brings better visibility. Each, as I mentioned, each developer would get a unique Ingress object, unique URL, and they can share that and collaborate with other developers. At the end of the day, this makes it a happy environment. It's not a conflict ridden. All the developers are working on the same development cluster, and they can deploy their changes at their own time and at their own pace. So we address the shared development environment with DeskBaseDeploy, and developers are becoming happy. So next up is the local setup hassles. So let's take a scenario where a new developer joins our company. They get a new laptop. They're itching to start to develop, start to write code. But before they do that, they need to first set up tools, say Docker, AWS CLI, a lot of command line tools. They need to set up packages, write versions of the packages like Python, PIP, Flask. And they may need to set up more than one service. If they're setting up services locally, they may need to repeat the same steps of installing tools and packages for those specific services. And after a period of time, they may need to do this again if they're changing laptops, if they're changing the local environment. And that's a very painful procedure. We also find a few challenges with that. One of them is when you're hosting the local services locally, we have a high resource usage. Some of the services, they may use more memory or CPU. And so there's resource contention in that local environment. There's also a challenge of outdated documentation. Say we have a new service that's deployed, we have documentation for installing tools and packages. But as time progresses, tools change, packages change, but the documentation does not get updated. Then the new developer joins the company. They look at the outdated documentation. Local services are not coming up. They ask other developers, time is wasted, resources are wasted. And so that's one of the main challenges as well. And we also found this challenge of works on my environment syndrome. What I mean by that is when they have a local setup running, they have a local service running on their laptop. They start developing, things are working great on the laptop, but when they deploy it on the Kubernetes environment, things break. Everything seemed to have worked on my environment. That's what they obviously, that's why we named the syndrome. So there's a drift or this difference between a local environment and the production environment or development cluster. So again, the developer experience team had a look at that. We saw that paint point and we wanted to address that paint point. And what we found out is that DeskBase has another feature called DeskBase development mode, where it allows it to deploy remote development mode environment on the Kubernetes cluster using the command DeskBase Dev. So when a new developer types in the command DeskBase Dev on the local environment, what that does is that it deploys the service on the Kubernetes environment and it connects that service in the Kubernetes environment to the local environment such that any changes that made locally will be synced to the service in the Kubernetes cluster. And you can see the changes in near real time using the URL devx.devcluster slash svcc in this example. So they create a unique environment and they deploy the service on the Kubernetes cluster and they can sync the local changes to the Kubernetes cluster. And it automatically hot reloads the service. We can also connect the integrated development environment like an ID using such as VS Code with remote debugger such that you can insert breakpoints and endpoints on the Kubernetes environment. So again, very similar procedure. A developer would simply clone the repository, create a unique namespace for themselves and use DeskBase Dev to deploy a remote development environment. So what are the benefits? One, we don't need to set up tools, packages, or services on the local environment. We are developing on a production like cluster. So no need for local development. This also brings us low resource usage because none of the, you're not hosting any services locally. And it also provides easier onboarding. When a developer gets joins in your company, they simply need to install DeskBase and they will run the command DeskBase Dev in installing tools or packages or all sorts of services to set up their local environment. So coming back to the developer experience journey, local setup hassles is addressed using DeskBase Dev. Now, we've made some improvements and definitely the cycle time would be shorter, but we wanted to take it a step further. How can we use these building blocks to make the cycle time even shorter? And the way we did it was to bring DeskBase deployments to our CI or continuous integration workflow. So essentially what we do is that for each pull request, we would deploy DeskBase environments. We've deployed and then provide that unique environment for the developer on the workflow. So let's start with an example. Say a developer, they want to start developing. They clone the repository, create a test branch, develop using the DeskBase Dev feature. And once everything looks good, they would create a PR. Once they create a pull request, they simply type in the command slash DeskBase deploy on the PR comment and that kicks off GitHub actions that we created, DeskBase deployed GitHub actions. And what that essentially does is that it creates a DeskBase environment for that pull request, deploys the changes in that pull request into the unique DeskBase environment. And here we use the IngressObject URL, repo-prnumber.devcluster slash SVC A and C. We also collaborate with our developers such that they want to deploy multiple services on the environment. So one of the conventions we use is that we look for the same branch name as the one created for the PR. We look on different services. If a branch name exists, we would deploy that service on this developer environment as well. So for example, service C is being deployed in this environment. And developers can use this URL to collaborate with other developers. They can review with other developers. They can, and if it's not good, they will probably go back to the DeskBase Dev, develop, make some changes, redeploy their changes to the GitHub actions, and then deploy it on the DeskBase environment. And then they continue testing as well. So most of the collaboration demonstrations, everything's happening as part of the CI workflow. And then if everything looks good, they would merge this. So one of the benefits for it is that it's part of the CI workflow, everything is happening as much faster cycle time. Collaboration is happening with other developers on the pull requests. And it also provides a faster review cycle. Reviewers can simply see the environment. They can validate the changes. They can verify it. And if everything looks good, they would approve it. And in the end, this results in a faster development cycle. Here we have a screenshot of the pull request comment of when a DeskBase environment is deployed. We provide the links. We provide the nitty-gritty details, such as QPSito commands, if they need to look at it and link into it. We also provide Slack channel if they need to contact us, if there's any issues or any feedback. So we made the longer cycle time shorter by bringing DeskBase into deployments per pull requests and developers are really loving it. And we have the final pain point of having manual end-to-end tests. Developers in our organization, even if they get a unique DeskBase environment, they would still need to configure the environment, load the data, and run end-to-end tests. These steps were totally, they were manual, and they felt like there was a lot of pain points for that. And the developer experience team, since we already built these workflows, these new workflows by bringing DeskBase deployments to pull requests, we wanted to address this by taking it a step further. So what we would do is once we deploy the DeskBase environments for the pull requests, we automatically configure the environment, load the data, and run end-to-end tests as part of the CI workflow itself. So again, going with developer, they will clone branch, develop using DeskBase dev, and they would create a PR, and this time they run a different command called slash e2e run, and this kicks off get-up actions that we created. And very similar to the previous approach, they would deploy, it would create a unique DeskBase environment and deploy services onto that environment, configures the environment with loading up data, all the manual tests, manual steps are automated, and e2e tests are run against that environment and provides the results back into the pull request. And again, we get a URL so that developers can test against that here, same as before, repo slash dash, a PR number. And again, developers can test it, they can develop using DeskBase dev, and they can redeploy their changes using the get-up actions. And this part would be considered DeskBase deploy. So what are the main benefits? Well, they don't need to do any more manual end-to-end test configuration, they would simply need to kick off these get-up actions and that will automatically configure the environment and run e2e tests against it. It also provides on-demand testing capability, they can simply run the commands on the pull request and it will automatically run it for them. And it also provides a gating mechanism to merge. If all the end-to-end tests passed, allow the pull request to merge, if not, don't bother, or we can add in more logic or implementation to that. Here we have a screenshot of the commands slash e2e run and then it provides the results saying which end-to-end tests passed and which ones failed. And it also provides a link to the logs. So again, we address the manual end-to-end tests and it is solved by adding in end-to-end tests as part of DeskBase deploy per PR. So how did we roll out? It took us a few quarters, two engineers, myself included, and it took us for proof-of-concept implementation and we implemented it for each core service. After we implemented it, we created thorough documentation and self-how-to guides, how to get things started without the developers depending on us. If they did need some help, we did walk-throughs and discussions and that way that also helped us learn about how they are interacting with the new workflows and we iteratively improved upon them. And finally, we did presentations on our organization-wide meetings so that all the developers got to know about the new workflows and we really care about it as our internal product so everybody got to know about the new workflows and processes and get used to it. With that being said, we also had a few issues with our rollout. Some of the lessons we learned is one of them is since we deployed a desk space or we implemented the new workflows ourselves, knowledge was siloed to our team. So if there were any issues, developers, they were dependent on us to solve the problems. And also, there were issues where developers tried the new workflows and the new processes. Some things didn't work and they went back to their own old ways. We didn't get to act on it proactively. We didn't get to know about if any issues happened. We didn't know about it before the developers telling us. So we resolved it using a few things. One of them, we built monitors, alerts and we built visibility dashboards so that we would know the health of the system or the desk space deployments or any of the processes before the developers notify us so that we can be proactive in fixing them and make sure the systems are stable and the workflows are working as intended. We also found desk space experts so that developers were interested and they got to know about desk space and they became experts and champions for the specific teams. So they built custom workflows for their teams as well. And we also got a lot of support from Loflabs. If there are any issues, we raised the questions or we raised the bugs to them and they were able to resolve as soon as possible. And if there was any issues, any questions regarding implementation, they were able to give us some tips and tricks. And reception was, since we rolled them out, over 200 ephemeral environments have been kicked off. A lot less Google calendars has been created. Developer experience improved as well as code quality has improved as well. For the next steps, we wanted to have a wider adoption. Not all the developers are still using it so we wanted to cover more developers using, some of the most of the developers are using desk space deployments but some of them are still using their old ways of local setup. We wanted to get a wider adoption. And finally, we also wanted to standardize these workflows, these processes across all the other services, not just the core services, but non-critical other services and other namespaces as well. So the developer experience is standardized across our entire organization. So recapping, we had all these issues with shared development cluster, local environment setup, and those made the cycle times longer, along with the manual end-to-end tests. And we were able to address each and every single paint point using our own way of implementing desk space deploy and desk space dev, and our ways of integrating as part of our CI workflow. And with that being said, I'll be ushered in an era of peace for our developers. Thank you.