 Hi, everyone. Thanks for joining me here today. My name is Josh Hendrick. I'm a senior solution engineer at RookOut. And today I'm going to talk to you a little bit about debugging Kubernetes applications on the fly. So what we'll talk about today is starting with really the evolution of software development and why organizations today are considering adopting Kubernetes as their platform of choice for running distributed cloud native applications at scale. We'll look at some of the challenges that organizations typically face when they're making those decisions. And probably some of the questions that you should ask yourself if you're thinking about moving towards Kubernetes or maybe you've already started it. We'll look at some of the challenges developers often face when they're debugging Kubernetes applications. It's a different approach from traditional application servers that many might be used to. So we'll talk a little bit about that. And finally, we'll talk about a potentially better approach to debugging, one that takes advantage of modern debugging tools that can really save developers a lot of time. And at the end of the day, of course, make your customers happy because you're solving their issues at a faster, more rapid pace. So we'll jump into it and talk a little bit about how things have changed recently. So there's been a number of changes in the recent years in terms of how we build, develop, and deploy software applications at scale. A number of years ago, we were building these tightly-coupled, single applications which required these big bang releases. It required organizations to align across different teams to come together to put out these what we now call monolithic applications, right? These softwares that were composed of these components that couldn't be interdependently deployable. Everyone had to get together and release once, twice, maybe four times a year. And release processes were hard and complex. What we realized very quickly is that we needed a better approach. And the approach that was initially taken was a move towards what we called service-oriented architectures, where we started to decouple these application components, allowed them to be a little bit more interdependently deployable. They were more core-screened services that communicated over APIs to one another. And that brought us to really where a lot of organizations are today. And if you talk to the vast majority of organizations, they are moving or have moved towards a paradigm where we now call microservices, right? Which consists of having a number of loosely-coupled, interdependently deployable components that really fit well into agile development paradigms, where you have teams that are responsible for building one or more different services. And it makes life easier in that those individual services can be deployed independently of one another. If one component goes down, it doesn't take down the entire application, and really allows for running applications at scale in a little bit easier approach. And Kubernetes has really been the platform of choice for those organizations that are looking to build microservices. And we'll talk a little bit about why that is. And before I jump into that, it wouldn't be a tech conversation if we didn't throw in a Dilbert. I like this one and thought it was quite funny, because it really resonates with, I think, the initial thought of a lot of organizations. Do we need to re-architect our applications, or can we just take them, put them in containers, and move them to the cloud, or move them to Kubernetes, and get the value that we're looking for? A lot of times, as you probably know, it's not quite that simple. You can't just containerize something, push it to a new platform, and expect to get all the benefits and all of the value out of it. Requires a little bit more thought in terms of re-architecture, and really educating everyone from the organization on new approaches and new paradigms. So if we look at some of the recent software development trends, there's been a major move towards microservices. Organizations are re-architecting their monoliths or starting new Greenfield applications with microservices in mind, taking these monoliths, doing a lift and shift with some level of re-architecture towards these cloud or on-prem deployments. A lot of the, and actually all of the major cloud providers now, have these managed Kubernetes offerings, which allow organizations to deploy their applications in either an on-prem or cloud-based Kubernetes cluster. A lot of organizations are even moving their CI-CD workloads onto platforms like Kubernetes, which allow really for a lot of resiliency and scale as they build and deploy those Kubernetes applications. And the adoption is really growing incredibly fast. So organizations have been adopting this for the last number of years and even more are jumping on lately. Why is that? The main reason is it gives organizations a lot of value in terms of running those applications at scale, really things like elasticity. You can automatically scale your application as more customer demand grows. So Kubernetes is nice because it can automatically do that for you based on the specification that you tell it. You tell it how you want to run your application and it handles all of that underlying scheduling and deploying of your application for you of different containers and different components so that they're running in the right places at the right time. It allows for self-healing as well. So you can automatically replace any failing containers or Kubernetes pods that are running. So if something goes down, it can handle automatically bringing those back up. And it tries to do that automatically for you and it's flexible. It allows you to run on-prem and the cloud. A lot of organizations are looking at hybrid deployment models where you're running some things on-prem, some things in the cloud. And it's really an amazing tool if you take the time to do it right. So there are adoption challenges. Any time you're moving to a new platform and paradigm like Kubernetes, you're definitely going to face challenges. So there's things like you have to learn how to set up your Kubernetes cluster. So it's new education for your operations and DevOps team who are responsible for managing these clusters. You first have to move your application into containers if you haven't done that. That's definitely the first thing you should think about. You have to learn how to deploy those containers into Kubernetes. So if you have CI CD components and tools, it requires probably and possibly re-architecting those to make sure that you can deploy efficiently and in an automated approach. Kubernetes under the hood uses YAML files to manage really the specification of how you should run those Kubernetes applications. A lot of times, app organizations are packaging up their Kubernetes apps using something called Helm charts. Think of Helm charts like the package manager for Kubernetes applications, if you will. Then you have to automate all of this stuff. You have to constantly improve, provide feedback, and of course, all while delivering business value to your customers. And of course, it requires a new approach to debugging when things can go wrong. So before you just go out there and move your application to Kubernetes, you really need to ask yourself a number of questions. Is Kubernetes actually going to reduce your costs? It doesn't necessarily offer that immediately. It doesn't necessarily give you immediate efficiency, but it offers you new levels of scalability that you can get by choosing to use it. And as we talked about, if you're not containerized, that's definitely the first approach, and you're not really going to gain much if you're not first containerizing your applications. Looking at breaking down your monolith, simply doing a simple lift and shift onto a different platform may cause you more headache than value. So you need to spend the proper time re-architecting your application. And it will allow you to move faster. It requires new build and deploy capabilities. You need to learn and understand a new platform. All of these things come into play. And is it better than what you currently have? So are you going to see immediate performance improvements, or will that cause you new problems that you're not necessarily aware of yet? So all in all, it's not a magic bullet that's going to immediately solve all your problems, but it can absolutely help you scale and provide a lot of value if you're willing to put in the work. So looking at debugging applications that are running in Kubernetes, the developers that we talk to still tend to think of this as a big challenge. So it's a new approach that developers need to understand and learn. Simulating the exact situations or conditions that allow you to find the root cause of a problem can often be a challenge. So the traditional approach of debugging, and especially debugging locally, can be difficult. When you think of how you traditionally debug a lot of organizations, go and look at application logs. They're used to going into a virtual machine or an application server, looking at logs and trying to figure things out. And oftentimes even debugging remotely can be a challenge. Applications are often locked down in clusters. Developers may not have access to the Kubernetes clusters. There's security mechanisms and things in place that can make it hard for developers to even get the information that they need. And then there's the fact that you have to learn new approaches to actually get that information. There's new tools that developers need to become familiar with. Cube Control or Cube Cuddle, whichever you like to call it, has basically a command line tool that allows you to interact with the Kubernetes cluster, get information from your Kubernetes pods, get logging information back, and a lot of organizations choose to lock those down for developers. So debugging can be a little bit slower than in traditional approaches. And when we look at some of the limitations of debugging locally, this is where there's oftentimes a big challenge. A lot of organizations that are used to traditional application servers, developers can spin up those application servers on their local laptop, run the application, and debug in a fairly similar environment that they would have in their test environments or even in their production environments out there. With Kubernetes, a lot of the times it's a bit different. You can't oftentimes run a full-blown Kubernetes cluster on your local laptop. So the environments that are running out there in your cloud are often using higher-powered servers that can be difficult to reproduce. There's a lot of different options for that. There's things like MiniCube, which allow you to spin up Kubernetes clusters and a number of different others on your local laptop, but there's gonna be a lot of differences in terms of actually having these same configurations and same conditions that allow you to easily reproduce defects. One solution that a lot of organizations take is an automation script, which allow developers to spin up all of the different microservices you may have on their laptop or their own machine. In that case, developers basically control the configuration, versioning, branching of the script, which can be a challenge and don't give you the exact configuration conditions that you often need to reproduce issues, but it can oftentimes be a good start, at least in a development environment. And there's open-source solutions like Docker Compose that are out there that allow developers to have a script which can run one or more microservices, databases all at the same time and allow you to spin it up from the command line with a single command. So developers can run a single command, spin up all the microservices in one go and have the application running locally, but the underlying infrastructure that's running those components is completely different than your Kubernetes cluster a lot of times. So it allows you to find some issues, but not always all of the issues. Really ultimately the system that you care most about really lives in your production environments. There's also issues with debugging remotely, right? So when you think about how you go and access your applications, if you look at Kubernetes, it is accessing a Kubernetes pod can be an unstable operation. Kubernetes behind the scenes is responsible for how and when it schedules your containers to run. You may SSH into a running Kubernetes pod, Kubernetes may kill it before you have the chance to get the data. So you need to have the proper logging and infrastructure in place to make sure you can get the data that you need. Traditional logging and log files of course is an approach most organizations are taking today to save information, application specific data that they need to debug things. There's other approaches like taking Kubernetes traffic and redirecting that to a local debugger that might be running, which is great for lower level development or lower level test environments, but production environments that are locked down, that's oftentimes not an option because that would be a major security vulnerability. There's also other solutions and things that can make it easier to get debugging information, things like Istio or LinkerD, which are service meshes that are oftentimes ideal places to put in debugging information or tracing capabilities, but at the end of the day, they don't always give you the ability to get down to the root cause of an issue that may be within your code. They can give you insight into performance slowdowns and things like that, but not always code level defects. So it can highlight, for example, this specific microservices is running slow, but maybe not exactly why. That may still require a developer to go back and dive into the code to find the root cause of a specific issue, right? And given that developer spends so much of their time going in and trying to fix bad code or debug applications, really having the right tools to optimize that time that they're spending debugging remotely is oftentimes critical, right? So there's the typical things like configuration drift that you need to keep in mind. You need to be able to simulate the right data conditions within your application. And so debugging remotely is nice, but there's oftentimes a lot of concerns that cause slowdowns or don't allow you to get all of the information that you need in order to debug those applications effectively. So really in cases like this where you're looking to debug remotely, there are more modern cutting edge debugging tools that in a lot of cases can really remove the need for development teams to have to go in and create, specialize defect reproduction environments. A lot of organizations are taking their production environments when they find a defect, spinning it up in a lower test environment, trying to reproduce the issue there. And there's a lot of back and forth and time that can be wasted when doing those things. So what we'll look at next is a little bit of a more modern approach to debugging Kubernetes applications. So there are a new set of debugging tools that are out there called remote debugging applications that allow you to debug applications really on the fly without having to make changes to your code or create defect reproduction environments. They allow developers to go in and immediately get the data that they're interested in really, really on demand while your application is running which allow for really developers to go in and increase their velocity, reduce the mean time to reproduce defects. And at the end of the day, make your customers a little bit happier because you're finding and solving their problems more quickly. So before we dive into how those tools work, just wanted to go through a typical process of how developers typically debug and a lot of the different steps that are taken when they're trying to get the information that they need from their applications. So if you think of a traditional process, debugging logs, right? Most organizations and most developers use debugging logs as the primary method of debugging. And what happens when a developer wants to resolve a defect, they'll go in, they'll find the logs that they're interested in, they'll search through to see if they can find some hints or some clues that give them an indication of what the problem might be. And if they can't find the information that they're interested in, what they'll have to do is they'll have to go back to the code, they'll have to potentially add more log lines to the code. They'll have to then go once they've added the log lines, run some quick local tests to make sure they didn't break anything. They'll have to push that code to their version control system, open a pull request, which will then oftentimes kick off their CICD processes, which will build their application, run test cases. Oftentimes, they'll need to get approvals when that happens. So somebody may have to go in and manually approve a specific step before the build or deployment can happen into test environments. They'll have to oftentimes wait for the next deployment window if there's shared test environments. Once it's actually pushed into their test environment or their production environment, they'll then have to wait for the data. And then they'll have to go back into the log, search for the data. And if they don't find the data they need, it's kind of an iterative process that they continue on in terms of trying to get clues into what's actually going wrong. And so at many organizations, this can take anywhere from hours or even days or weeks, depending on the amount of regulation or lockdown that might be in place. So it definitely brings up the need for a better approach. Something that reduces the amount of context switching that often happens, the amount of wasted time that developers have. Something that they can use to actually just go in immediately while the application is running, add some additional logging statements, get the data from their application that they need while it's running, and then be able to go in and resolve the defect from there. So really what these debugging tools and these remote debuggers that we're talking about allow you to do is really quickly and instantly collect remote data on demand from your application. And so that's really a nice segue into talking about these remote debugging tools. So with these new modern distributed applications that can have many different components running, there's a lot of places and a lot of areas where things can go wrong. And it's not often easy to find the source of those problems. So using remote debuggers and what we're looking at here is kind of a simplified view of the Kubernetes architecture. If you're not familiar with the Kubernetes architecture, there's a lot of different components involved, but in general, there's a couple of key things. There's a command line tool which allow you to interact with the Kubernetes cluster in an API server that allows for communication back and forth. There's a scheduler which handles scheduling your Kubernetes containers and pods on different nodes within your Kubernetes cluster. And there's this master component that manages all of that. So where your application is actually running are on these nodes and these Kubernetes pods that are scheduled to run and scaled up and down on demand within your cluster. So the way these remote debugging tools actually work is they allow you to go in and install a remote debugging SDK or agent that run really alongside your application as another dependency to your application or as an agent, depending on the language that you might be using. And they allow developers to go in and remotely connect to your application and collect data instantly and on demand without having to change the application code, without having to wait for those slowdowns by contacting production support teams or operation teams to get the data that they're interested in. So what did they allow you to do is to go in and set these data collection points within your running applications to collect snapshots of data from the app. And that consists of things like local variable data. So you can see all of the local variables just like you would in a local debugger. You can view stack traces, profiling information, tracing information. And that's all packaged up and delivered to the developer on demand wherever they need it. So, and these things often run across the entire environment. They can run in development environments. They can run in staging or test environments and even in production environments for these production-grade remote debuggers. So it's something that really allow developers to take the most advantage of their time, make your customers happier and reduce the overall time it takes to identify and solve some of these defects. So there's a lot of different options out there. If you're not using a remote debugging solution today, they can really allow you to have a overall better approach and give your customers a little bit better satisfaction when they're using your application because you're solving their defects really on demand and as quickly as you can. So that's it. That's what I wanted to talk to you about today. If you have any questions or you'd like to follow on and get any more information, I'm happy to chat with you at any point in time. Feel free to send me an email, Josh at rookout.com. If you have any questions or comments and I'd love to talk to you more. Thanks again for tuning in. Bye.