 Great, so I Was in a talk earlier today in the go dev room from Andre where he was talking about advanced techniques of debugging go and he asked the audience, how do you debug and everybody shouted out print line and It's quite clear that debugging, you know kind of lost its way over time But at the same time it's not because we've got this great dev room and there's been amazing talks from people talking about the debugging tools that are out there and I'm gonna go through this is mainly a tool talk. I'll do some demos, but the tools are still in their infancy I'd say and Debugging isn't gonna be your first problem when you get into kubernetes. Your first problem is deployment You want to focus on your CI CD process ideally with some GitOps techniques and then once you have a good flow of getting code into production you then need monitoring and There is a whole dev room on monitoring and tracing open tracing is a fantastic technique for finding out what's happening in your application Once you got all that going then on your day three if you're that fast you can start debugging and Just so we're all on the same page. I want to just do some basic kubernetes talk Hopefully, it's the end of the conference and you've been able to pick up at least enough to where you don't need this But just so we're all on the same page while I keep going through The container is the core. Well the pod itself I guess I think of as the core component in kubernetes and in a pod you have a container running You can have multiple containers running Ideally, it's preferably just one container, but there's no restrictions. You can have containers You can have volumes with data in them or not as you like. There's networking capabilities that kubernetes provides you and This is kind of your your basic concept of what you want to do with kubernetes is run your docker image Run your application in a docker image doesn't have to be docker either, but primarily it is and so those pods then run inside a node and the nodes would be Typically an actual machine and you can have multiple nodes and distributed across all sorts of Areas and each pod can then be in different nodes and distribute out in all sorts of different ways kubernetes is there to provide The orchestration you tell it what you want you give it the configuration This is how I want my environment to be like and kubernetes does its best effort to give you that So if a pod crashes if your application has a crash and it shuts down Kubernetes will do its best effort to spin that up if the node gets lost Yeah, if you're in the cloud and they just take that VM away from you kubernetes will do its best effort to shift those pods That were there to a new node or other nodes you do have so again. That's it's a best effort technique Based on what you want so Hopefully that's kubernetes in a minute and now into debugging and Everything that I'm going to show you today. I do not recommend to do in production all these tools Have huge security risks let alone just taking your whole system down So I gave a talk yesterday about ephemeral environments. I'd love to plug that There's a lot of tools on how you can get set up in a non-production environment and I mentioned Andre's talk earlier today, and there was a great talk from Lucas about One of the tools I'll talk about later that you can check out and Yeah, I mentioned that like There's this one technique of just logging out what you want to find in your application But the debugging has so much more power that when you can utilize it you can really get the best development process available So with the tools I want to start with two that I particularly find more like they're my favorites They're the first ones that I got these to work and when I was able to set breakpoints in my IDE on something running in kubernetes I thought this is great stuff, and I did it with the use of one tool was ksync and the the concept is that you have your file that runs in your pod and you know it's just an html file and you build that docker image and You put that file in it then you run your pod and that file sitting there in the clouds or in your cluster and You have that file in that docker image and you want to change that without having to recompile a whole docker image and everything And so ksync will sync a directory that's on your local machine with the Directory that's in the pod and so then on your file and your local machine You can change the file and it'll get synced into that container your docker won't restart It just updates the file and you now that have the change that you want it Available to you in your running system Once you have this going you could then connect a tool called squash Which is basically a wrapper around debugging for kubernetes It'll spin up another container in its own namespace which opens up the port connections And so then your IDE locally will have that port connection configured So then when you're running the application and hit the point where you would have a breakpoint It would then trigger in your IDE and stop the pause the application while you develop squash is limited to delve for go and Java and GDP They had Plans to support Python and Node.js this year this past year and didn't reach those goals So of course they're open for contribute contributors to help get there But yeah, so as I said this was when I got these two tools working and they each do their own thing So they don't have that extra complexity when something goes wrong if something goes wrong with one of them You know, this is where I need to focus my attention You don't have one tool trying to do 70 things and you're like wait. What is it this? No, it's there. Yeah, so So that's again the benefit of having these two separations There's another tool called telepresence which when this works, this is even better this when this works This is probably the best The concept is that you have traffic coming into your cluster into your pods You know communicating with other pods in your network and coming back out and you know You want to debug something in that one of these pods? so with telepresence you create this tunnel into your local machine and your local machine then has access to everything else that's in the cluster and so as I mentioned about not doing this in production if you do this in production and Traffic comes in here and then goes to your local machine and you're not running the app on your local machine There there's that's the end of the traffic, right? Or if you have a break point sitting on your IDE and somebody hits that break point Yeah, that's the end of it. Like nobody's gonna get back the results that they're waiting for Telepresence because it creates this interesting networking at least when we develop it or we use it with our go applications We have to create a new VPN for this network and You can't use another VPN while you have telepresence running So if you have as we have VPN for all our outside developers to be able to access the internal networking They're not able to use the VPN and develop with telepresence. It's not compatible for two VPNs at the same time at least with that technique of creating the network that uses Mac OS and Go and so there's the documentation on telepresence gets into its Limitations and what you can do and how you can do it with different IDs as well and integrate it with your ID directly So check it out if you can get it to work. You're gonna love it If you can't then try one of the other tools There's also an important aspect with telepresence is when you start the application locally, you know If you have the settings in your pod that say, you know, which ports something connects to like all the configuration that's Available for the Docker image that's running in the pod You need to know when you execute the command locally Yeah, if you need to know that this here is called, you know DB on port 306 when you execute locally You have to know that it's DB 306 and so forth Okay, so the next tool to talk about is scaffold which basically combines the two of case-sync and squash into one tool and It's available and IntelliJ and VS code as a plug-in and It works fantastic when you have just this smaller like the one application with less dependency structure and It'll manage the syncing of the file into the pod If it it'll know that if you have a compiled language that it'll need to spin up the new docker image It'll spin that up for you put it in the place that you tell it to and pull it down and start the pod again and have everything Running locally in your IDE and you get a lot of fantastic usability when it works and It was interesting because I have this demo that I'm going to go through with this application And I was trying to get scaffold to work with it because I thought oh this is you know a great tool This is how you should use it, but I couldn't actually get it to work on something that had this Pre-built helm charts that are I already had my application. It was already there It you then have to do everything manually and once you start doing things manually some things don't work And then you got to figure out how I did so I need to get there Maybe I didn't spend enough time with it all these tools there as I said they're in their infancy They're gonna take some time to get rolling. Yeah, so don't expect to just flip it on and ah, this is wonderful so the the last one I wanted to talk about was the new IDE from Eclipse that actually runs in the Kubernetes cluster so all that networking and mismatch and trying to figure out how to sync files you don't have to do because your ID is actually running there with the pods Started up with the code that you are developing right there. So the ID is a browser based interface, but you just interact directly with it and get all the all the magic there Lucas had given it gave a great talk on this a little while ago and It helped answer why when I tried to set it up two days ago. I still was like, hmm, how's this gonna work? What's going on here? It's brand new. I think it's just been out a few weeks. So it'll be good pretty soon, I think based on what I'm hearing and the the one limitation now is that if you're doing it in like your local machine if you're spinning up a mini cube as an example and then Installing it that install process takes a little bit of time as well And so it's not a quick up and running experience at least if you're doing it locally Yeah, if you have a cluster in the cloud where it's running You don't need that extra time of getting up and running But if you have that cluster in the cloud other developers can then interact with the same code base I believe so demo time I Learned a lesson and recorded my demos But what I haven't done was figured out how to actually Give a demo that based on a recording. So we'll see how this goes and yeah, it's not going good yet because Share the screen that should Okay, so here I'm gonna be demoing k-sync and I start out by running mini cube on the machine and I'm using a slightly older version of Kubernetes and not very older Google only released it a like a few weeks ago. So Made it available in GKE I mean a few weeks ago, but that's because the The application that I'm going to be installing there doesn't have the new API versions configured So the app only works with one dot one five and before So now everything's up. We have our our cluster. We have a default namespace and everything the kube system namespace and I Got a helm installed locally, but not in the help in mini cubes or install that I Have armador installed which again was in the talk I gave yesterday on how that works So now I'm going to need to pull some code down and there's this example voting app that I went into that has these different components and The idea is that I'm gonna clone this repo. I'm gonna be a developer of the voting app I'm gonna be you know this front-end developer working on a Python application and so I need this set of code here and So I clone that code copy it into another directory and then I Also want to have the helm configuration for that code and of course docker doesn't give that in their sample file, but Code fresh heads originally wrote it and then I added it into armador and the documentation there so We can easily pull down the The helm chart with the armador configuration and merge that into the voting code So as a developer of the voting app, this is kind of the this would be the Code I would have locally With you know my app.pyton my docker file my helm configuration In the helm configuration I have the armador file with the dependencies that I need and The docker file that adds this app and Here we can see that there's you know cats and dogs that are available for voting and In the example of using ksync I'm gonna want to switch from being able to vote for cats because I'm just not a big fan of cats But I love bunnies, so I'm gonna work on making that happen So first we're gonna install the whole environment Of all the components we need so there's the five charts to be installed one of them is the voting app and The others are the Postgres the Redis the results app and the worker those got installed Then we can see they're installed but not yet running So we'll wait for them to get running which 39 seconds 50 seconds whoo that went quite fast time's flying so everything should be running the Database is still initializing Postgres is still a bit slow, but we should be able to now load up the front end So we'll use mini cube to open the front end There's the cats and dogs We'll load up the results so we can see yeah, I voted for a dog and Here we go in and change cats to bunnies Maybe nope not yet first we want to set up ksync so It's already installed, but it's not initialized into my cluster so that runs and then I'd start watching and so ksync will now just watch what's happening in the cluster and Then I'll have to create an actual instance to I Don't know to be watched I guess and from that I'll need to know the oh Maybe I'm showing you something else first Yeah, so here you can see that there's in the coupe system So ksync has to have these extra privileges to which you again wouldn't want to include in your production environment so here it's just sitting there waiting and listening and It's gonna need the The app selector for the pods so the voting app has the the label that Will be needed during the create Yep, so first I show you some documentation on ksync show you how I Google to get documentation and Quickly read through it all. Yep. Yep. Yep. Yep. Yep. There we go get that selector and so We need to take a look What our labels are in our pod? There's a lot to read through here, but there's the voting app and there's the selector and Let's speed this up because we might be low on time Hopefully I can talk fast enough where I make a directory. So I have this theory where With ksync, you know, it syncs the folder from your local machine to the folder. That's in the docker image and and the docker image You know, we have hopefully just a small subset of what what you need to run Yeah, so in the actual code base you have your docker file you have your application you have all your configuration you have your ID settings and Hopefully your docker image is compiled and built without all that extra stuff and just the application you need And so when I do a ksync, I don't want to sync my directory of my code I want to sync just an empty directory and get all the stuff from the docker image first done to my machine and Then copy things from my local environment into that and this is maybe because we work primarily in go language is a compiled language and You need to compile it anyways. I'm not going to have it in my local in the directory of my code base So I'm gonna have to copy it over anyways And so here I'm going in and I'm showing you what's in the folder that syncs and then what's in the actual docker image there Now, of course, they built the docker image with the docker file My old ideal of you know separating out what's there is lost on this technique with this example But you got to do things right if you want to be right now baby steps There we go. Now I changed some bunnies and I saved the file again. It doesn't sync anything yet because I only saved it into that local directory but then I copy it into the ksync directory and We see that There it goes. It copied the file Reloaded the pod didn't actually stop, but there's our bunnies we can vote for bunnies and Well, we voted for cats that didn't work so I think that's the end of this demo and then Oops So let's try to fix the the bunnies on the back end and My theory is that it's a problem in the worker and so we have the the voting app in the front end in the worker that converts it into then show in the result app and The worker is a Java application. So in theory in this I could have used to squash. However I'm not a Java developer. So I ran into a whole bunch of problems getting that to work so in this demo, I'm going to use telepresence and Be able to spin up the worker On my local machine while it's connecting to the Redis and the Postgres and in the cloud So I start up IntelliJ go into the worker code And again, I took it from the example like I took their code You know, so this is me as a developer of the worker I want to debug the worker and I go into their source code. Yeah, there's just one file and It does the connection to the Redis and to the database and then it just waits for things to happen and In a moment, I'm going to set up the configuration to be able to debug Don't know what I'm doing there There we go set a break point and then Go into the menu and Okay, so debugging it's waiting for Redis right because I started the application But I don't have the connection to Redis. I'm just running an app on my machine so now I need to turn tell the presence on and What I need to do that or so I have this Redis It's actually interesting so I already have the telepresence command set up I'm going to swap out the deployment of the worker to be a tell it presents instance and so I'm going to do that. I have to get my password right and then telepresence starts up Then I was going to show you about how there's you know Ways of doing this with minicube. There's plugins for development with IntelliJ There's a whole lot of different options here and here you can see that I'm using the TCP connection with the VPN So that has the limitation that only tell one tell the presence and So I'm still waiting for Redis What actually happens with this instance is that the Redis DNS name doesn't Resolve so I have to change the Redis to be the port I think this is just a Setting on my local minicube thing, but the DB Resolves so there we go. We're connected. We got Redis and the DB running and Then we're going to set a breakpoint to see what happens when we vote and we're going to go and we're going to vote we're going to vote for a dog and It doesn't change because it hit the breakpoint here. Yep, there it goes the votes in the queue and to speed things up Yep, so we vote for B. We can see that there's It's actually coming in just as the letter a and not as bunnies or cats or anything. So It's not even a bug in the worker. It's a bug in the front end application. That's I didn't change what I needed changing Yeah, so that was a nice demo. I have five minutes left to try to do a demo of squash that It's not here. I Uploaded a zip file with the this code that I've got So VS code I've got just the simple application with a kubernetes configuration of just a service and a pod Running on port 880 the docker file is really straightforward. I uploaded the the default docker file into quay And the application is really I was looking at this I must have taken this from somewhere because it doesn't make any sense, but it gets to the point across and so if I first run it could fly and So and then I got a port forward so we can see that His ad is true. So I'm trying to add option one to option two So I'm trying to add two plus two and it equals zero. All right, so clearly there's a bug there and so now I need to first set up the casing because again go as a compiled language so I need to compile the Changes and have it with the debug available. And so let's do a case sync watch and a case sync Create Nope, that's not it So it's in that name call a calc. Yeah, this should be good. So now I've got this thing Yep, there we go running case sync get yeah, so now that's We're in the code so if I do make Case sync this is just going to build the application and save it into the casing folder which I think should have worked and The port forwarding still there and so in the code in here I can do I already installed the squash plug-in into VS code I Yeah, that's basically all you need to do with that and delve needs to be installed and then you start up squash and It squash is running in another namespace So I'll get namespace so there's the squash debugger runs there and As I started up it says hey, you know, I've got all these namespaces. Where do you want to debug something? I got it running an sq and here's the example app and which debugger do you want to use delve and So now it should just start up there we go and I can set breakpoint somewhere in the code and around here after this print calculating and Load the app It didn't work. What if I change? This and do another make that's probably why it's not actually updating the binary I think I'm a little low on time here to find out why this isn't working But again, I uploaded this code base and so based on part of what I said you might be able to piece this together on your own one last thing to show you is kubectl exact minus IT minus n sq spin H L s minus all so here we can see that the The application was updated Well, you know an hour ago, but no real time based on time zone The file got synced. It just didn't change Times up Thank you