 Hello, and welcome to this talk. My name is Edidia Ngasipo. I currently work as a senior developer advocate at Ambassador Labs. I am also a senior ambassador, and you most often find me writing, building, or sharing knowledge with people in the tech community. I go by DD codes on all my social media platforms. My name Edidia Ngasipo can be hard to pronounce, so people call me Edidia. In case you can't pronounce Edidia, Edidia is fine. On all my social media platforms, I go by DD codes, except LinkedIn, which is a combination of my first and last name, Edidia Ngasipo. In this talk, I'm going to share several case studies on how the CNCF tool teller presence has enabled cloud-native companies to catch bugs before production, accelerate their inner dev loop, and improve their developer experience. These case studies are based off three companies, cultural code, which are the creators of the Things app, Voiceflow, and a fintech company who opted not to share their names. Just like most companies, the development teams of these companies had certain Kubernetes challenges that they wanted to fix or improve their development workflow. Take things for instance, like I mentioned, this is an application created by the team at Culture Code. Things enables you to plan your day, manage projects, and make real progress towards your goals. The development team at Things uses Kubernetes in AWS, they have multiple services, and they also have multiple managed databases. And if you work with cloud-native applications, you know that most often than not, they always comprise of so many microservices, and most times, these microservices work interdependently and communicate with each other to achieve larger business goals. And because of this multi-directional service-to-service communication, it's always best to test your local code changes against the services and external dependencies they rely on before pushing into production, because you want to ensure that it's going to work as expected when you eventually push it. And of course, this can be quite challenging. You deal with things like limited resources, inconsistent data between production and non-production environment, and even managing distinct configuration for separate environment. For the Culture Code team, one of the things they really struggled with was the inconsistency between their dev environment and production environment. They would make a code change, test it with the setup they had, and assume everything is going to work as expected. I feel like most of us have gone through that as well. But then when they pushed to production, they realized, oh, something happened, there was some sort of missing configuration that wasn't in the actual setup that they had. And it really impacted them, because whenever they shipped features to their end users, it would oftentimes not go as expected. So they wanted to address that problem. They really wanted a dev environment that would allow them to test their code changes against external dependencies without the fear of things going wrong or not matching up with their shipping or not matching up after they shipped to production. So they decided to use telepresence. Before they used telepresence, this is what their workflow looked like. They were unable to actually test in an environment that was similar or the same as their production environment. So they would deal with certain challenges like misconfiguration, bugs in production. But after they used telepresence, they were not able to have a development environment that resembled production very well. In the awards, these are some of the things they said. We can now develop against the same database running in the exact same configuration. And when we push something to production, we know that it's going to be 100% compatible with the environment. They didn't have to deal with surprising missing environment variables. They didn't have to deal with incompatibility between the production and mock database. And most importantly, they were able to catch bugs before production because they shifted testing left. So it means that because they now had an environment that was really similar to production, they could test against these microservices, test against the databases, and be sure that when they push this thing to production, it's going to work as expected. The next case study is from the amazing people at Voice Low. Even though I don't know what Voice Low is, it is an application that enables conversational AI teams to design, test, and ship, chat, and voice assistants faster. Like many of us, Voice Low builds their application as a monolithic application. And then they eventually decided to migrate to microservices. And like you would expect, they moved from having one big application to multiple microservices, multiple databases, and Q engines too. Like Voice Low, most organizations adopt cloud-native development practices and technologies because they were told they were going to enable them to code and ship features faster, help with scalability and flexibility, amongst other things, of course. Even though Kubernetes helps you to achieve these things to a certain level, what nobody tells you is that it actually impacts your inner development loop tremendously. For instance, let's say this is what a traditional inner-dev loop looks like. And a developer had to spend six hours per day coding. And they had a traditional inner-dev loop of five minutes. So it means that they spent three minutes on coding, one minute building and reloading to test their code changes, and then one minute testing that code change. So let's say they had 10 to 20 seconds on committing their code changes. So it means with this workflow, they would have been able to make at least 70 iterations of their code per day. And the only developer tax here is committing code, which is actually not that serious, because it doesn't take time. You could do it in a manner of seconds, maybe minutes, depending on how you're trying to craft the commit message. However, when you transition to cloud-native development, it dramatically changes, right? Coding remains the same. You still spend three minutes, for instance. But then now you have to wait for your container to be built, push to a registry, and then deploy it into the Kubernetes cluster before you can see the impact of your code changes. So it means that the iterations that you had per day, which was 70 previously, could now reduce to something like 40, right? And the developer tax here now is having to containerize, push to a registry, and deploy into your Kubernetes cluster. Most people don't think about this as a big deal. They're like, oh, sure, let me just wait. Go grab some coffee or something. But sometimes your application could be really big, that it takes minutes in order for you to finish building, right? Or let's say you have bad internet connection, so it's a lot slower than usual. So the time that you could have used to actually focus on coding and testing the impact of your code changes, you're just there waiting for your application to be containerized and then push to a registry before you can see the impact of your code changes. So for voice load, they wanted to change this. When they were working with monolithic applications, they were able to see the impact of their code changes as soon as possible. But then now with microservices, it wasn't like that. Of course, they still wanted to enjoy the benefits of using Kubernetes. I mean, all of us want that. But then they also wanted to have a fast in-app development that would make them see the impact of their code changes quickly. And as such, eventually push product to their end users a lot faster. So they explored several Kubernetes in-app dev tools, like Starfold, Octato, TelePresence, and Dev Space. So after this exploration, they decided to go with TelePresence. And here's why. According to them, and in the awards, that's why I added the codes. They said other solutions introduce more complexity to our in-app dev loop, like adding files syncing with the cluster, or checking changes in the Docker file, rebuilding it and pushing it to the cluster. Whereas what they really wanted was a tool that would not add an extra set of development to the already complicated in-app dev loop. I would only make them have to run their local service locally while every single thing was already in a remote Kubernetes cluster. So and TelePresence helped them to provide that workflow. So by using TelePresence, the voice load team moved from having to, originally, they would have to build the application, push it to a registry, and then deploy it to the Kubernetes cluster before seeing their changes. And this made their in-app dev loop slow, like I mentioned. But with TelePresence, the need to constantly build and redeploy test images was removed. And this was able to enable them to speed up the fidelity of their in-app dev loop. So it was now a lot faster because instead of waiting to build, push, deploy it to see, all they needed to do was run a TelePresence intercept command and it will connect their local development environment to remote Kubernetes cluster interacting with other services. So they could now test to see that this service that they're working on would work as expected when they push to production. And if something did not work as expected, what they needed to do is just go back to their IDE and change it. But if you had to go through the build, push test cycle, you would have to come back, make all the code changes, rebuild again, and again, think about it from this way. Imagine if this application is very big, or let's say your internet connection is not great, or your laptop is slow, maybe it does not have a lot of memory or something, because if you think about it, even when you're doing normal things like responding to Slack messages, maybe trying to get something running or maybe owning Docker desktop, some of those things make your laptop start to hang. So imagine not having to always redo the same process over and over again. So with Voice Low, they were able to adopt TelePresence and this made them not have to go through that same process again. All right, the final case study is from the Fintech company. Like I mentioned, they were not okay with sharing their names, but this Fintech company provides personal accounting services in the APAC region. One of the core problems they had was that their Kubernetes development environment was negatively impacting their developer experience, and they wanted to fix this so that their developers in their team would have a good experience when developing products. So they explored several options. One of the first things that they did was go through the local development route, yeah? And the thing with this is that you get to enjoy, of course, the many benefits of local development. You can use VS Code, set breakpoints, have hot reloading, and even see your logs faster, right? Like all of those good stuff that you could have when you were developing locally. And it was also cheap too, because they didn't have to spend lots of money buying Kubernetes clusters. The dev loop was really fast as well, because everything is running locally. It means that when you make a code change, you can see the impact of that code changes immediately, and once you're comfortable with it, you can then push it to production. However, the maintenance was super high, right? Because you always have to maintain your mock APIs, and even your scripts, for instance, to ensure that everybody was running the same services or databases on their local Kubernetes environment. And because you're running every single thing on your laptop, your laptop starts to get hot. Even when you're responding to Slack messages sometimes, you will start hanging. So imagine running an entire cloud native application that would oftentimes have hundreds or fifties of microservices. At some point, your laptop would not be able to run it anymore. And as a FinTech company, it didn't take them too long to realize this. So they had to move from local Kubernetes environment to the remote Kubernetes environment. Here, they didn't have to struggle with resources. They had all the space, all of the storage. They had all of it. However, it was very expensive because every developer now had to have their own remote Kubernetes cluster. And we all know how that can be expensive when you think about it. But the maintenance was low. They didn't have to constantly maintain mock APIs or scripts or any of those things. They could set up CI systems that would automatically update their remote Kubernetes environment. Cluster, rather, when someone made a code change or made changes to the services. And your workstation temperature was normal. Again, nothing is running on your computer. It's all in the cloud. So you're good. You can respond to those Slack messages, go have coffee. Your laptop is great, waiting for you to come back. But one of the drawbacks to this, in addition to the expenses, was that the dev loop was really slow. Like I talked about with voice flow, you always had to wait for your application to be containerized, pushed to registry, and then deployed. And this would oftentimes take a lot of time in your development workflow. And I mean, we know how fintech companies are, right? When there is a bug, you want to move as quickly as possible so that maybe you don't lose a lot of money or you fix something before it affects your end users. So this dev loop was really slow for them. And they wanted something different because they were on a mission to actually improve their developer experience. So they thought, OK, local development is great. I could have debugging options, set breakpoints, hot reloading, all of those good stuff. However, I can't run a lot of these things on my laptop, right? And then again, there's remote development. I have all the space I could ask for. I could actually test my code changes directly with the actual dependencies, right? But then it's actually too expensive. And I always have to go through the build-push test cycle to know, OK, what if I could merge these two things, right? Get the good size of local development and the good size of remote development and merge it into something. So they kind of did like a research on the possibility of doing that. And then bumped into an article on telepresence which explained how we were trying to address that problem, right? Because our engineers at the team also struggled with the same thing. And that was one of the motivations behind building telepresence. So with telepresence now, it offers something called the Remote Call Development Environment. And I'm going to explain the benefits of that in a second. So in the Remote Call Development Environment, it's more of a combination of the local and remote. You can think of it as the best of both worlds, right? And what this does is that it's actually cheap. Find your microservices and external dependencies are still running remotely. But then developers no longer have to use specific remote clusters per person, right? They could now have to use a shared remote cluster through telepresence shared isolation feature. And the maintenance was still low because, as usual, you're still using CI systems to automatically update your remote Kubernetes clusters whenever someone on your team made a code change. Your workstation is normal because even though you're running some bits of it locally, you're only running that service that you're making a change to. So let's say you have three microservices in your cloud native application. And there's something wrong with one of the service, and then you want to make changes to that service. So you're only now expected to run that service alone locally, whereas the other two services will still run remotely. So your workstation will not be really impacted as opposed to having to run the three services on your computer. And the dev loop was super fast. And I'm going to tell you why, because now they didn't have to go through the build-post test cycle. What they needed to do was connect the local development environment to the remote Kubernetes cluster. So it would oftentimes feel like those two things are in the same place. And you can test the impact of your code changes against the actual services and dependencies in the remote Kubernetes cluster. I'm going to explain more about this in a couple of minutes, explain how telepresence works, show you a demo of telepresence, and hopefully you'd get the picture when that time comes. But pretty much this development environment gives you a combination of both worlds. It means that you'll be able to seal at break points. You'll be able to see your logs a lot faster. But you don't have to struggle with the whole resource constraints of your computer. And then you don't have to constantly have to maintain your mock APIs or scripts or any of those things. You are testing against the actual services and dependencies in your cluster. So without telepresence, this was how the team looked like. They did not have a really great developer experience. So they bought the operational and resource button of running all your services locally. But with telepresence, the need to do that was removed completely. They only had to run the service that we're making a change to. They couldn't utilize the benefits of the local development environment and the remote development environment. They had to choose one. Remember, they tried local first. And then they had these problems. And they're like, oh, no, let me go to remote. But with telepresence, they were able to merge the best of both worlds together. Without telepresence, they always had to code, build your container, push the registry, deploy and wait, and then test. But with telepresence, all they had to do was code, create an intercept, and test. So far, I've shared how telepresence has enabled three companies to catch bugs before production, accelerate their inner dev loop, and also improve their developer experience. So you probably want to know how it does all of these things. What exactly is this telepresence? How does it do these things? I claim that it does. And that's what I'm going to explain. So telepresence is a CNCF tool. It's open source. And you can think of it as a tool that tries to make Kubernetes development teams to develop and test on Kubernetes faster. Like it simplifies that process. Yes, enjoy the benefits that come with Kubernetes. But don't spend extra minutes or hours waiting for things to load, when you can actually focus on the most important thing, which is coding, shipping, and sending it to your end users. And it does this by connecting your local mission to a remote Kubernetes cluster via a two-way proxy mechanism. So that way you can access remote services, as if they were running locally, and then route remote traffic to your local services. So this enables you to create a seamless remote local dev environment, where you're enjoying the benefits of local development and remote development as well. So how does telepresence work? In this example, you can see we have two services here. Service A prime, which runs locally on our computer. And Service A, which runs in the remote Kubernetes cluster. So typically, the ingress sends a request to Service A, because, of course, that's the one running the remote Kubernetes cluster. What telepresence does is it sets up a sidecar agent, which would then check that request and confirm if the request comes with an HTTP header. And if it does, then it's going to send it to the traffic manager, who would then reroute the traffic to Service A prime running on your local machine. But if it doesn't have a HTTP header, it's going to send it to Service A. To external users and every other person, they don't understand what's happening. Nobody gets a warning saying, something has been changed. For you here, who is on your laptop and has access to your remote Kubernetes cluster and local service, you can actually see these things. And I'm going to show you a demo of how telepresence works. And this is not just for one developer. Like in this case, I showed you one local service running on a computer, but it may not always be the case. You may have more than one developer on your team, or hundreds of developers as well. So telepresence provides an isolation to your request. So in the diagram, you can see we have Service A prime, which runs on Alice's laptop. And then Service A2, which runs on Bob's laptop. Of course, it follows the same process as well. The ingress sends a request, and then it hits the agent that is deployed on the service you're trying to intercept. It looks through where is this request coming from. So typically, every developer has an authentication token. So it would then look at that, and then it said, hey, is this coming from laptop A or laptop B? So for instance, in case of Alice and Bob, it would then send the traffic to the traffic manager. And the traffic manager would then send the request to either Alice's laptop or Bob's laptop based on who sends the request. So even when there are multiple developers on your team, you can still utilize telepresence without stepping on each other's toes. So let's assume that the request was sent and did not contain a HTTP header. It's going to go to the actual Service A, running the remote Kubernetes cluster. But then if it does contain that HTTP header, it would come to your laptop. So external users using your tool are not going to be impacted. You can just focus on testing, adding your breakpoints, doing everything you want to do without necessarily impacting your team members, or causing something to go wrong in production or for your end users. So one second. Let me switch to demo. All right, cool. So this is an application called emoji voter. And this application, you pretty much come here, vote for your favorite emoji. But someone had complained that they could no longer vote for the Donut emoji to my manager. So my manager reached out to me and asked me to go and fix it. So I tested. You can see the laughing emoji is working. But if you click on the Donut emoji, it shows an error saying, hey, this is not working as expected. As one of the software engineers who worked on this application, I know that devoting service is the service that handles this request. And if the Donut emoji is not working, then it must be from the Donut function. So what I want to do is to confirm that my hypothesis. So I'm going to go to my terminal. And here, I've already connected my local development environment to the remote Kubernetes cluster. If you look at line 7, you'd see connected to context default, meaning that I've been able to connect my local Dev environment to the remote Kubernetes cluster running this emoji voter application. So right now, the service I want to change is devoting service. So I'm going to run a gRPC call command against that endpoint. And once I do that, you see that it's going to throw an error confirming that, oh, hey, that error actually comes from the Donut function in devoting service. I'm just going to highlight that so you see the error I'm talking about. All right, cool. So I go to my IDE and open the API folder. And I'm going to go to the API.go file, because that's where all of these functions are written. So quickly, you'd see that the vote joy emoji returns joy, and the vote emoji returns that. However, the vote Donut emoji returns an error. So what I want to do is actually change it to return the Donut emoji. So I'm going to quickly do that, and then I'm going to save my code changes. Of course, now, if I go back to the actual application and try to reload it to test to see if the application is working, you'd see that it's not going to work, obviously, because this change that I made is actually happening locally. It's not deployed to the Eumoc Kubernetes cluster or anything. But if I come back to my terminal and run that gRPC call command again, you see that it's no longer trained that error, which shows me that, oh, I've actually been able to fix this thing. But following best practice, I want to ensure that this local service, devoting service that I've changed, is going to work with the other services in the remote Kubernetes cluster so that when I deploy it, the application will still work as expected, and the Donut service, sorry, the Donut emoji is going to work when it's being clicked on. So I'm going to clear my screen, and then I'm going to now run the telepresence intercept command, and then I'm going to pass the devoting service, which is the one that I want to intercept, and then pass the pot that I'm trying to target. So on doing this, it's going to reroute all of the traffic from the devoting service on the remote Kubernetes cluster, so the devoting service running on my local mission. You can see here that it says, oh, it's on pot 8080, and then it says all of the traffic has been intercepted to this service. So now if I go back to that application, click Reload, and then come to vote for the Donut emoji, you'd see that it's going to work as expected. It works. So right now, this application is actually getting the request from the Donut, from devoting service on my laptop, not the devoting service in the remote Kubernetes cluster. So now I've actually been able to test and see that this devoting service I just made a change to would actually work effectively with the other services in the remote Kubernetes cluster. But of course, this example shows global intercepts in telepresence, and global intercept pretty much routes all of the traffic to your local service. But like I said, sometimes you don't want this thing to be seen by everybody, you're just trying to test alone. So telepresence has another feature called Personal Intercept, which will only reroute a subset of the traffic to your local mission and not all of it. So if Alice goes to the website, she's going to see the Donut emoji not working. But for me, I'm going to actually see it working. So it only routes that request to the HTTP error that I've added, not every other person. So it doesn't impact every other person. So in order for me to do this, I'm going to run the telepresence leave voting command. That command pretty much ends the intercept. So on reloading the application and clicking on the vote on the Donut emoji, you'd see that. Let me quickly go back. All right, so I run the telepresence leave voting command. So that stops the intercept. So I'm going back to this application. I click reload, and then I try to vote for the Donut emoji. Again, you'd see that it's not going to work. So now it has pretty much rerouted the traffic back to the actual voting service running the remote Kubernetes cluster. All right, so now I'm going to log into Ambassador Cloud. Ambassador Cloud is what enables you to use personal intercepts and is also going to generate a preview URL of these code changes that I have made. So when I send this preview URL to my manager, who asks me to go check this book, she'll be able to review the code changes I have made and pretty much share her feedback. And I could always go back to my IDE and check it and make the code changes. All right, so let me click on this to show you. All right, I just highlighted a session that said, in order for you to use a personal intercept, you need to authenticate by logging into Ambassador Cloud using the telepresence login command. So I'm going to quickly run that. All right, and then I'm now going to run the telepresence intercepts command again. So when you log into Ambassador Cloud and then run the telepresence itself command, it's going to add you for a couple of questions based on your ingress. So in my case, I'm using Ambassador Edge Stack. So I'm going to add that at the port and also choose if I'm using TLS or not. On doing this, it's now going to generate a preview URL, which I'm going to copy and run. Once I copy this preview URL and run, I'll now be able to see the impact of this change that I have made. So devoting service running locally on my computer and the other services running in the remote Kubernetes cluster without impacting any other person. So if you now go back to that application, it's looking to look the same way nobody's impacted. But I, who I'm doing the testing, can actually test it. So I'm going to copy the preview URL now and then go to my browser, paste it, and it will pretty much be the same flow. So right now, it would just work for only this preview URL. Anybody who goes to the other one is not going to work except this. And then if I can then copy this preview URL and send it to my manager for how to review the impact of the code changes. And once everything is set, I can push it to GitHub and eventually push it to production. So without having to go through the build-post-test cycle or waste time waiting for my application to build, I was able to test my code changes immediately, share it to my manager for how to give me feedback and deploy it to GitHub when I was ready. So in conclusion, Kubernetes and containerization has solved many problems that businesses face today. It gives you scalability, flexibility, and a couple of other things. But it actually does make development workflow a lot slower than it was. And in order for us to solve this, we need to bridge the gap between remote Kubernetes clusters and local development. And this is exactly what Telepresence does. It gives you the best of both worlds. Telepresence is an open-source project. It is a C&C project as well. So you can try it out, and you can also contribute to it. I've added links to all of the necessary information you need here, the GitHub link, the product page. We also have a partnership with Docker. So Telepresence is available on Docker Desktop as an extension. So you could also check that out as well. We have a pavilion at K13 in case you want to speak with someone on the team, or you want to visit our booth at Ambassador Lab, which is S7. You could also join our Slack channel at 8, our.ios.slack, if you have any questions, and our Twitter account at Ambassador Labs. Thank you so much for joining us.