 Hi, welcome to my talk navigating the shift to remote developer environments like a year of insights and challenges My name is Andre Marcello Tanner. I'm a staff DevOps engineer at Ada support a Toronto Canada AI company You can find me on the hang-up slack or the CNCF slack and I also previously gave a talk at KubeCon Amsterdam about how we do ARCO CD and get ups and disaster recovery. So please check that out Now this talk is a continuation from a talk also done at KubeCon Amsterdam by my teammate Raj Using death space to usher in an era of peace for our developers. So you want to see Where we were back then and how we got started on this journey. Please go check out that talk after this So I want to start with a statement Local development is outdated remote development rules to present and then after this talk, let me know How where we are in Latin? What do you think? Let's start with some definitions and the two most important ones I want to find is local and remote development Just to make sure we're on the same page So local development is when you read your service on your local machine like your laptop and You don't really need an internet connection to get feedback during your development's life cycle Remote development on the other hand is where you're running your application probably in a remote server in the cloud or on-prem And you need to have that internet connection running to be able to get feedback in your development So that's the two definitions. I want to highlight here So what was the problem at ADA a year ago and why did we look for a remote development? Environment solution. Well, we were company about seven years old We've had lots of changes over the years lots of developers lots of services people coming and going come changing teams and You know every time you make a new service or things change in that service the complexity grows the development becomes harder and so testing and working in those services Really became a bottleneck So you want to find a solution to make it easier for everybody to develop all those services and be able to move faster and some of the problems we we saw was like People were developing locally and then when they would deploy to the dev environment to test it out with our other depend services things would break because other people would be using those services or They're test what they develop locally wouldn't work in the Remote environment because it's not the same as their local machine Even if they're using Docker So there would be various changes and think things would break and see if we even had a second development cluster just to Make testing easier, but that didn't really solve the problem and so really had a scheduling problem here where people wanted to test and make sure their their thing works in Their their feature works before getting it out to production But they want to make sure nothing else is broken So they ended up with schedules that they were using and we were using like a Google Calendar to schedule testing on our dev environments and it really slowed down development, you know, it was more like a Not a great solution And so we look through many solutions to solve this and we we land on dev space Dev space is an open-source project by started by loft labs and it's now a CNCF Sandbox project We started using it before it became a sandbox project and it's really awesome in that it allows you to deploy to any remote environment Kubernetes environment Whether local or remote using a single command dev space deploy similar how you might do get push or Docker run Dev space makes it really simple to do that and We could use our existing Kubernetes Manifests we were using helm. We can also use customize or plain Kubernetes manifests and so It worked really well. We didn't have to learn like a new paradigm for developing new templating language and Another thing we really liked about is the way it deployed in the pet dependent services because we had Modeleth and dependent microservices and Making all those work together was quite a challenge. We actually had our own homegrown shell script solution before this made of, you know Bash line bash scripts that in our CI service that would try to piece all these remote deployments together and make them work It's really hard to maintain and once we got dev space working an open source project That did the same thing where it was it basically delete all our custom Scripts that we used to use for remote deployments and just use dev space Another two things that are great is it takes care. Oh, it does it You don't need, you know Server-side daemon to run. You can you just need access to a Kubernetes cluster even a local cluster or remote cluster And it's totally client-side. It even has a client-side dashboard And It can take user input like ask the user questions you need to customize the deployment with some user Something from the user. You don't have to like have them memorize a long command of Parameters and I'll show you what that looks like in a second So this is how we implement dev space. We put a dev space.yaml file in all of our core repositories and We specify the helm chart and the values we wanted to override there and also like Other things you can customize about it. I mean that there's they have great documentation available and so What we did was when someone dev space can build images using canico or Docker build kit, but we chose to have our images built by the same process we do for production our CI process. So What people do now to what our developers do is they just clone the repo Or their branch and then they they run dev space create namespace and then their their name of their unique Namespace that they want to develop in and then they just run dev space deploy and out it goes and So what that looks like here is you can see here we just ran dev space deploy We asked the user some questions. What branches do they want to deploy of some dependent services? what what URL do they want to make this deployment available at and then Takes those in saves them so you don't have to enter it the next time and then just run the helm chart deployment and Give some helpful links after that that we added in that makes it easier for the developer to know where to find its logs and dashboard and everything So that that's dev space in a nutshell. And so this was our remote environment solution that we were able to deploy Using dev space we took it one step further where Since it's a single command dev space deploy We ran that from our CI which is in GitHub Actions now and we use just standard GitHub Actions for doing PR comments and When when everybody adds a PR comment to a PR a pull request Slash dev space deploy it runs the same command in our CI and deploys to our cluster and Then gives you some nice comments with a feedback is what what we made here but this really lowered the barrier to entry and Made and this actually became our most popular Method of using dev space at our company You know because you didn't even have to have access to the cluster or set that up or even download the CLI even clone the repo. You just had to go into a pull request and Add a comment and we use pull requests every day as part of a developed lifecycle of our company. So Yeah, this became really popular And so what are the wins what would be win after a year using dev space? Well There were thousands of deployments using dev space. It's used every single day by a lot of our developers Anybody can just join when they join a company join a team or start and on a service that they haven't worked on before They can get it up and running in less than five minutes And that that's huge like before these like set up your local environment with that service get all the dependencies installed To fix the issues that are that are there that took like one to two weeks You know and when you have that local the environment working You didn't want to touch it because if it broke that all you have to fix it and you know that it works with my machine problem didn't really exist anymore, you know and Also because we lowered the barrier to entry and you saw there via the PR comments Actually anybody who even wasn't an engineer or company like maybe a designer or product manager Or even our C-suite could come in and like add a PR comment slash that space deploy and get the service deployed or test out the changes and so this made it really good for feedback where Someone would open up a change or even non-technical people could make could look go into the code and deploy that change Without having to understand all the complexities of actually getting that service running because it was all basically abstracted into that devspace.yaml file and Lowering that barrier entry lowering the complexity to make these remote deployments and making easier to test Made complex big changes much simpler. So things like upgrading we use Python at our company and Upgrading Python versions is a big deal and it made this much simpler even for our largest services So we used to do like big changes very slowly and do a lot of testing locally and slowly roll that out to production But because of how we implemented devspace Our developers were able to make big changes to our services That had you know big blast radius, but they were able to test and work out all the issues in the remote environments And get things out faster And like I mentioned because they weren't relying only on their local machine to test changes They would catch issues in these remote devspace environments that would previously only be caught in production because Our development environment or remote clusters are a lot closer to our production environments. They're Kubernetes, they're running on the same, you know cloud using all the same services. So There were there was huge value there and we had a lot less incidents And so another big win was the data that developers were able to access since it we these remote environments had you know production like data or data that was very secure and Well managed If you if you depending on how your company works It may have to adhere to certain regulations around customer data or around user data. And so Doing that in local environments is really complicated, but then With remote environment, it's a lot lot easier. So that's one of the I think our developers love about a remote environment is that the data is easy to access and easy to use Also because it's not just one one local machine like the laptop you and when you change branches, yeah, you lose the old changes You can basically have multiple deployments going at the same time So you can take one branch deploy that using devspace change go deploy another branch or change to another namespace or environment and you don't have ten of them for all it matters and people and Basically context switch or have people test this one move on to the next one Have people test that there, you know, they can have multiple parallel streams work going on at the same time Which is a lot harder to do with a single local machine Another big advantage, but this is probably a given but like using remote Environments give you access to everything else that is out there in the cloud or your other dependent services that are in your environment things like GPUs are very easy to use With with the cloud and so that's all available via remote environments where doing these things locally like running our databases locally is It's possible and there are solutions out there things like AWS local, but it's a lot harder and Most companies nowadays or at least our company is a cloud native sass service So it really doesn't make sense to try and do a replicate all cloud native develop services locally, right? And so it's a lot easier to do that remotely and It comes in it has a better result And so after we deployed dev space other developers started Wanting to implement based on their services that didn't have death space yet. And so we were able to expand a search of death space By having other engineers Come and learn how we how we did it and implemented in their service and we helped work with them so we saw this also as a win of spreading the knowledge and the usage of Death space and remote environments and became like an expected standard at a company for all services. So When you're working in a service or feature People know they could they can always use depth space deploy to Work on it and develop Rather than trying to figure out how to deploy it locally, you know, it's now a standard but What are the learnings like what was the hard things that we learned and We had to fix and I think this is where the meat of the talk is is So the first thing was like remote your remote environments have to be 10 times better they can't just be two or three times better than What your developers were using before it can't be just like a long list of commands or Something that breaks off and you know, it can't has to be because it's because they imagine they've been doing this Probably local development for like 10 plus years in their career before you implemented remote remote environments and now you're asking them to just use this new way of developing and so We're able to solve this by making a lot simpler to deploy. I show you the PR comments Remove as much points of friction as possible and so that made a lot easier for people to use it where Even if they still like local development because it was familiar to them They knew the simplicity of using the remote environments and the value was was there so That that was our first, you know big learning Second one is like the quality has to be always kept high Distribute systems are hard things fail Versus a laptop. It's easy to fix. It's it's it's a controlled environment. So When you when you run this run this like a service like as a Any other service you run at your company, you know, you have to have monitoring alerting when there are errors you have to fix it And even though death space is a client side thing we have a support channel where when people run into errors that aren't working or they can't get to work we help them fix it because It's it they may not always know how death space works or how how to fix problems. And so We want to be proactive and see if it fails to deploy Find out why and fix that, right? and if you don't, you know people will complain and the loudest one and and Your remote environment solution will have like a negative Stigma attached to it. So you really have to keep Marketing and selling the value of your product your remote development environments to your teams to your company As you know probably For everyone complaint there are like 10 Successes that the users don't say anything about right? It's only the so when people complain That is feedback that you should you should take into account and and you know But also don't forget to keep marketing and telling people the value of what's going on and How it's being used successfully and how many successful deployments it it is and how much value, you know How much time it's saving so have those metrics to your observability observability platform We use Kubernetes metrics for tracking how many deployments are made How long they're up How long a PR takes to go through to production and all that and we regularly Talk about that with our teams and our leadership to make sure we know where it's successful And over time we've seen the number of errors decrease and the number and the success The successful deployments increase. So that's where you want to get to Another thing is I mentioned before about the data so Always keep in mind the data of That your developers need when they make these deployments like Because if they just do the deployment to the remote environment and there's no data there's an empty slate It's not really that useful for most of the time And so for us we have we came up with custom solutions that bring to our company of how we deploy and get access to the data that our users need for the development so I Think about how to do this and this problem is solved outside of dev space outside of how you deploy your remote environments So you really have to think hard about where to get the data from and how to make it easy to access for your teams Another one your early adopters like how users are critical to the success of your Remote environments, so we had early users early on who helped us test and figure out all the bugs and give us feedback and then We we they we help them you know like advocate for this to their teams so At our company we call them in for champions and they help go to teams and find out Problems and take them back to us and we help them. So, you know There are these people within various teams that want to help you that want to learn how to develop this way and you know Take the chance on it. So find those people and work with them really closely so that you can your remote development environments will be successful also Don't make your team the single point of failure. So Also teach people how to get around things and teach your your advocates how to fix things when things break, you know and Having using an open source product that has documentation out there and many use cases and and Is a really miss one way to do that also But we also have documentation things within our company that help people use depth space in our remote environments Here's the one is that the job is never done. Yeah, so I've been doing this for a year improving it It wasn't roses at first, but it's a lot better now and a lot more a lot of people use it at our company every day There will always be people who don't want to use it. So When you do get that negative feedback, that's good signal because they're willing to say something versus the people who just Stop using it and don't say anything. So work with those people To get feedback find out why don't I like using it where doesn't fit the use case and and also do road shows with teams because the same like with them with the Selling use case you just got to keep selling this right and so we also have regular feedback cycles with teams to get there to find out when things aren't working and We're also looking for more ways to improve the use the use of our remote environments. We also implemented and With our quality team implemented end-to-end tests to make it easier to test to run end-to-end tests On any branch and using depth space. So that was another way. So we're using and we also implemented part of our ML testing processes too, so We'll keep finding ways to add more value to it and you'll you'll get more and more Usage of it over time if you keep finding keep talking to your users Another one is here you're not going to solve everything with it you won't totally get away from local but Try to solve the the hardest problem and where there's most value We could have implemented death space at our company on a simple, you know cookie cutter template service that Maybe one team uses that's really simple to to implement But that wouldn't have much value overall for us if we did that So we implemented on the biggest all this hardest services at our company And it took a while to get working But when it did get working and after fixing bugs of how it works, you know It provided the most value and so instead of people having to go through a really hard time deploying it their local machine Now most developers could Could do this every day using depth space and that's where we found the most value And so really look for where is where is it most where you're gonna make the biggest impact With your remote environments and start there Yeah, it's and There will always be use cases that won't be covered by your mode environments so That's okay, but try to find, you know, the biggest ones, you know, the parental principle 8020 To go back to local development, it still will be there There will be people who love using it and we'll still want to use it and we'll still want to sell their local We found that the the death space dev It has a development mode that syncs to a remote container Had a lot of issues because of the remote connections and that failing out and Wasn't as snappy as just, you know developing locally on your IDE and getting immediate feedback using something like hot reloading So we're still looking into ways to solve this there are solutions out there like dev pod By loft lab that do solve this so That quick feedback loop in your IDE is still something That is hard to solve, but I think We're gonna find a way to do that so local development will still be there, but that's fine, but So what's next like I mentioned We've been looking to dev pod, which is another open source project by loft labs And it's sort of like get pod and get hub code spaces, but totally open source Where you could deploy and run any branch on an eye in your IDE When it's sort of like the same way you do development locally, we could standardize that and Make it also Use the same way to do development remotely. So we're sort of blurring that line And so if developer wants to do deployment locally That's fine. We'll have a standard way to do that And if they want to take it to remote and get all the value that's already there Then they can do that in the same way So they could deploy locally using Docker and then they could deploy remotely to Kubernetes and We've already done a POC with dev pod for this and it turns out to be pretty good So we're looking to get that implemented next Also, we want to standardize dev space and how it's used and customized There is that YAML file, but we don't think it has to be so unique for every single service So we have an internal IDP or internal developer platform at our company and we want to standard make it standard so that Every service can be developed and use and use dev space and Take advantage of remote environments Another two things here is dev space is a really versatile tool. You don't have to use it for just Kubernetes deployments For production, we do have like Argo CD and GitOps But we've been thinking of well, what do you do when Argo CD and the GitOps process fails and you still need to get Something out to production Well, dev space can be a really great tool to simplify that for developers If they in case they need to break the glass and deploy the production and skip the whole Process in case though that those things are down, you know, so That's one thing that we're looking into and then also for standardizing workflows Dev space is sort of like a workflow tool If you look at the docs has really great docs and that so We're thinking about replacing things like our custom make files and and other Short-gut scripts and just putting that into dev space Basically using dev space as the one command to rule them all Could be that way where we're looking into that so Like I said in the start local development is outdated Remote development rules the present. Well, local development still exists but in the past 12 months we've gained so much value from our remote develop environments and We are so much better off than before so I Think this is where everybody should be and should get to and If you have questions find me on the CNCF slack or scan the QR code here and leave feedback on my talk Thank you very much