 Hello. Welcome. Hey, my name is Linsa. I'm the moderator for this session. I work for a small company, Solo.io, with specializing in application networking. We still have some spaces up front if you are brave enough to walk on the first row, second row. So, please welcome, Erin, to present to you a scene that is believing debugging with ephemeral containers. Thank you. Okay. Let's get going here. So, let's see if I can make this work up. Getting a little ahead of myself here. There we go. Make sure PowerPoint is going to do what I need to do. Okay. So, I'm going to be talking about ephemeral containers today, of course. I'm Erin Alpar. I work with Castin, software engineer with Castin. I do software engineering sort of stuff. I primarily focus on the data layer persistence. So, today I'm going to talk about ephemeral containers. So, and what's important about ephemeral containers is that the, they are more than just containers that stick around for just a little while. They are a very important piece to Kubernetes architecture. I'm going to give you a little bit of that and hopefully reasons why, as well as a lot of the technical background that makes ephemeral containers work. And hopefully, by seeing that technical background, you'll get an understanding of what their power is. So, anyhow, so let me proceed. This is going to be a pretty technical presentation. So, I encourage you to download the slides. They're up on schedule for this. I just happened to update them about a half an hour ago. So, if you've already downloaded them, try to download them again. The content remains largely the same. So, the ordering of a few of the slides and I've emphasized some details in the new ones. So, if you're following along, I encourage you, definitely encourage you to download the slides. I've got a lot of console-y type text in here. A couple of slides. The fonts are small. So, I'm hoping that folks can see them yet another reason to download the presentation. Anyhow, let me get started. Before I get started about talking about ephemeral containers, I'm going to talk about what you're going to need in this presentation. This presentation is both me presenting and talking about it, as well as a sort of self-driven tutorial. So, I'll talk about that as we go along. To make this tutorial work, you're going to need, and this is designed to do after the presentation. I don't try to follow along while I'm talking because we've only got 30 minutes since it could go pretty quickly. I'm using kind in this presentation. Kind is a local cluster. The reason why I'm using kind is because I'm going to need direct access to the node and I'll basically need privilege access to the node. So, the easiest way for me to do this is to start up a local cluster. In this case, I'm using kind, which happens to be what Kubernetes uses for their own testing. It's fairly featureful and it will certainly do the trick for our presentation. It does use a docker. So, there's implied that you'll need docker installed to make this work. I think that's pretty standard at this point. I'm also going to be using Postgres. Postgres just happens to be an open source application that I can install with Helm. It doesn't really matter what you use. But the examples I'm going to be using are going to be using Postgres. Okay. So, you're going to see a bunch of slides that look like this. So, and the blue, you see the dollar sign there? That is a command prompt. That's in bold, right next to it. Blue text. That is something you can copy and paste into a shell and you should get output that looks a little bit like the gray text that follows it. So, voila. So, this is my command to create a kind cluster. Just type that in. Kind create cluster. It'll go about its business and create a new local cluster for you. After you've got a new local cluster installed, you can install Helm. So, this is a command to install Helm. I use Bitnami repos. It's in there. Create a namespace for it and then use Helm 3 to install that. At that point, you should kind of be all set. You got kind installed. You got Postgres installed. You might want to check your versions just to make sure that you've got the correct stuff installed. I am running with Bleeding Edge. This is another reason to use kind is that you can get the latest versions of Kubernetes as well as, of course, Kube controls. Be compatible with that. Then I can go ahead, make sure I've got my pods running. Postgres will run with a single pod. Okay. So, you're all set up. So, now for ephemeral containers. So, ephemeral containers are new. They're beta in 1, 2, 3. This means that if you install Kubernetes 123, they will be enabled by default. So, you're not going to have to set any feature gates or anything like that, which is nice. All of the platforms, as far as I know, have 123 on their edge release. So, if you want to play around with ephemeral containers on something like GKE or what have you, either it'll just be installed by default or go take a look at the edge releases and you should be able to find it for installation. It should be fairly straightforward. So, with all this, ephemeral containers start with debugging. These are the cases that you'll primarily see on all the example sites for ephemeral containers and a vast majority of the literature that's out there is about debugging using ephemeral containers. So, I'm going to start with that. I'm going to talk just a little bit about debugging in general in Kubernetes, very high level, and then I'm going to do a deep dive into ephemeral containers and not so much the process of actually debugging your programs, but the stuff that makes debugging work. And I'll get into a few examples. So, there's, again, there's a lot of implications that go far beyond debugging with ephemeral containers. Okay. So, this is debugging by attaching. Hopefully, everybody's familiar with this. This is where you might have a pod that's having problems and you've deployed this pod with, like, a Debian release or maybe even a Red Hat release, and you just shell into it directly and you run commands and try to figure out what's going on. Here's an example of doing that. Ta-da. Kubekettle exec and we just run a bin shell or what have you in the pod so we can go ahead and log in and debug into it. So, you can debug by attach or you can debug by copy, which I'm not sure everybody really knows about, but this is using the actual debug command. It allows you to create a copy of your pod. So, pods are the smallest unit of scheduling. So, with the current infrastructure, what you have to do if you want to run a new process in a pod other than exacting into it is that you're going to have to create a copy of it with a new container or with whatever debugging tools, a new image or whatever debugging tools that you want. So, this is debugging by copy and what this does is it creates a copy of the container and it runs that. It's a little awkward, but it works. If you don't know about it, I encourage you to look it up. So, debugging without a femoral container so we can either attach, which I covered, which we can either copy, which I also covered, but this is a little, if you debug by attaching, you need to include all of your debugging tools in your container so you can attach to it. If you want to change that container to include additional tools, you're going to have to copy it. And this requires doing some foo magic on your workloads to make that work. So, like if you're working with a deployment or you're working with a stateful set, you're going to make a copy of the pod and then the stateful set's going to look at it and see that it's replicas of change and then it's going to start up another pod and you're going to have to set your services to point at the right pod. Anyhow, it's a little bit of a mess, but you can, once you get the hang of it, it works. Ephemeral containers get around this, and I'll talk a little bit about this. This is how you debug by ephemeral containers. So, you've got Kubernetes 1, 2, 3 installed by kind. You've got Postgres in there is my previous example. Copy and paste that command in there and what you get is an ephemeral container. Busy box. Standard set. So, this is a, it actually runs a new container. The pod stays the same, a new container is added to it, which is really handy. This is a little bit of a, a little bit of a special container. It's probably not going to be obvious immediately because all containers within a pod get to share the same network resources. So, so anyhow, by default, you'll see network resources shared. That's, that's really nice. And this is this ideal case for doing more complicated debugging. And as we'll see, we can get really, we can get really complex. We can get really complex with debugging, not just simply network resources. So, here we go again. Here's another example of debugging with ephemeral containers. Again, you can copy and paste this. You see the usual prompts. You know, if you don't see a prompt, hit enter. Now, when you're looking at the pod, the pod resource, the pod spec, things are going to change. So, I want to talk a little bit about what it actually looks like in the pod spec. So, you can see that after you've executed a ephemeral container. And then I'll kind of revisit this and talk about it again later on. So, this is the ephemeral container spec. So, I've run the previous command, boom. And what I get is a new ephemeral container that will be evident in the pod spec. Here's the ephemeral container spec. So, I'm going to go ahead and get the pod as YAML. And down here, I know the second. We have a spec. We have our containers there. And we have a new property, ephemeral containers. This is a zoom in to that. And again, here we see it. So, this lists all of my ephemeral containers for which I have one now. So, that's executing. There's another important part of this. And if I get it again and I look through it, and again, I blow it up. We also, on the status property, the spec, I also get a ephemeral container entry. So, we get two new entries in the resource, the YAML, for the resource. One is just regular ephemeral containers that tell us what ephemeral containers should be started. And then I have the status end, which gives me the current status of the ephemeral containers. So, I'm set. I can debug with ephemeral containers. I know what to look for on the spec. So, I can see what the status is of ephemeral containers, see which ephemeral containers are running. And again, the spec is changed on the pod as it's running, which is unique. That's one of the things that's really nice about ephemeral containers. But what is a ephemeral container? So, this is actually going to be a majority of the presentation. And I'm going to try to spend some time on this. A container can be thought of as a sandbox, right? So, you're in the sandbox, you're executing commands, and what you can see is limited. You can't see the entire system. You can't see into other pods, right? You can't really see into other containers, actually. You're basically, you're in a container. It can be seen as a sandbox. What makes this work is Linux namespaces. So, now, these are not Kubernetes namespaces. These are Linux namespaces. They've got namespace in the name. They're rather different. So, I'm going to talk about Linux namespaces in this. So, Linux namespaces control container isolation. Linux namespaces are the things that don't allow you to see outside the container into the rest of the system. So, you can play with these namespaces so you can see various parts of other containers. So, it's really nice. And this is really what ephemeral containers are about, is controlling Linux namespaces. I'll talk a little bit about this. There are many namespaces. Here's just a few. Mountpid. These are roughly listed in the order of importance. Mountpid, Net, IPC. There's a bunch of them. There's some resources at the end. I'm going to talk about these and there's some resources at the end, and I'll talk about those too. Each namespace is identified by an iNode. So, these are kind of those funny file system things. So, there's an iNode in the file system. It's a virtual file system that represents these. I'm going to talk about the pid namespace, the mountain namespace, and the net namespace. I think those are probably the ones that you're most worried about. And there's also some quirks in there that you should know about when you start to play around with these. So, as I said before, you're going to need privilege access to the node to make this work. And here's an example of this. So, what I'm doing here is one of my first examples of Linux namespaces is that I'm going to log in to the control plane node. This is basically when you use kind, it is the only node you got by default. And I'm going to take a look at the namespaces for pid1. Pid1, of course, is the net. And I can see those by ls-l slash proc. Everybody should know what slash proc is. One, which is pid1. And then for each process in proc, there's an ns. And under the ns directory, you'll see all the namespaces listed out. Over here, on the left, we've got, or the right, rather, we've got a listing of i-node entries with their corresponding namespaces that they're associated with. So, that is in proc. For the process, under the ns directory, you'll see a bunch of files, and those are all related to namespaces. I can get access to if I have a privilege shell on the node. Here's one of those zoom-ins again. This first entry, by the way, that's going to come up later on. So, just keep it in the back of your head. But these are the namespaces for pid1. So, now, if I, that's from the node perspective, what I can do is I can go in and do the same thing and take a look at my Postgres processes that are running on the node. So, that's, sorry, the previous slide was from the init perspective. This is taking a look at Postgres, which is running in a container within the Kubernetes cluster. So, here, I get my pid for Postgres. I ps it as an example. And then here, what I do is I get all the namespaces for the Postgres. And I can list all of those out. Now, these are the namespaces that the container is actually going to use to run Postgres in. And here's a zoom-in of that as well. The identifiers for this aren't so important. And I'll, we'll see more of this in just a second. So, those are how you see namespaces. You can look in proc. You can see which namespaces are identified with which process. In the example that I had before, we looked at pid1, and then we looked at our Postgres namespaces for our Postgres process. Those will come up in just a second. Looking at them doesn't do you much good. What you really want to be able to do is execute something with a certain set of namespaces set. This is what a command called nsenter does. It allows you to enter a set of namespaces and execute commands as though you were in that namespace. So, when you're in a container, this is really what you're doing is running a shell with nsenter. So, your perspective on the system changes according to what namespaces you'll see. This is an example of running nsenter. I'm here. I'm going to run nsenter in the Postgres namespace specifically for network. So, nsenter dash dash network. And then I give it a namespace file in proc that says start up a command using whatever namespace using the network namespace for whatever Postgres is using. And here I just run sstcp-l just to see what's listening. And here we go. If I want, it's a little tedious to list out all of the namespaces that you want to use. So, there's a shorthand it's called target. Target basically allows you to give it a PID and it will enter the namespace of that process. It's really handy. So, if you have a shell somewhere or you have some sort of command and you want to see the universe from the viewpoint of that process, this is how you do it. You specify target, give it a PID dash dash all and then run your command. So, I'm going to show you what things, what doing this looks like from the node executing a big in a namespace versus what it looks like from the container. So, here's a kubectl exec. Here I'm execging a command in the default container in my pot. So, ps-e, yeah, if I just want to see what the processes are. So, here I go, ps-ef, excuse me, those are the processes, should look familiar. And this is the view that we get if I exec directly into the node and then I do an MS enter, give it a target of my Postgres process and I do a ps-ef, basically the same thing. You'll notice if I switch between these two slides, the output is basically exactly the same. So, this, the NSNR command basically is the same, as I said before, is the same thing as running within a container in a pod. So, this tells you that basically this is essentially the same system that your node uses to build up containers. Okay. So, that's sort of the background. I'll get into some more details. And again, a little bit like that blue, the blue text I was telling you about and entering in that. I'm going to use this comparison technique of moving between slides in a few cases to show you what changes when you do certain commands. So, ephemeral containers. When you start in ephemeral containers, network namespaces shared by default, really handy, there's a bunch of other namespaces that are also shared. This is certainly the most important, I think. You can share other namespaces. So, you can kind of tweak things and change your perspective from the container. So, here's a namespace when debugged by attaching. So, this, I'm going to go directly into the Postgres node and I'm going to go ahead and do an ls-l self NS. So, self, if you haven't heard of that, is just whatever the current process is. So, in the container, I'm just doing a PS. Now, I'm doing this next command is a kubectl debug. So, this is a ephemeral container debug, right? Again, into Postgres. So, before, this is like exacting into the container and running a command. This is starting a ephemeral debug container against the same pod and against the same container. So, here I go ahead and start it. Ephemeral container with an image of a busy box. This will start a, this will start a new container and there it is, defaulting debug container name to debug blah blah blah blah blah. We get the usual thing, press enter. And if I do that same command, I pretty much get the same output, but a few things have changed. So, what I get is I get a set of new namespaces. And what's highlighted in yellow are the namespaces that change. So, when I start a new ephemeral container within the pod, I share all the same resources in that pod with the exception of C groups, my mount, my PID, and my PIDs, right? So, when I PID in my container, I'm going to, when I do an ls-l or something like that, I'm going to see my own set of processes, right? When I look at my mount points or when I look at my file systems, I'm going to see my own files. C groups has to do with permissions. I'll talk about a little bit in a minute. But as you can see, my net, the between the two yellow things right there, my net namespace remains the same. So, when I do like an SS or an IF config or something like that, I'm going to see the same network resources as the rest of the pod would normally see. There's time and not so important. The bottom one for username space is actually pretty important, which means you're going to be sharing the username spaces of the pod. So, the user IDs that you're going to see are going to be the same as what you would get in just all the other containers in the pod. So, this is by default. What I can do is, opposed to just starting up a formal container, the default container, is I can set up a specific target container. So, here we are doing a kube-cuttle debug and I'm specifying a target, postgres. That is basically the same format that you used in NSender, right, dash-dash-target, and you gave it a process. That's kind of what we're doing here. So, notice here that fewer of my namespaces have changed. So, this means I'm sharing more with the original pod. And what's most important is now you're sharing the pid namespace with the pod. Excuse me. You're sharing the pid namespace with a postgres container, which is very helpful. So, all of a sudden when you go into this container, you're going to see all of the same processes that your target container is in your target container. And largely the view of the world through your ephemeral container will look just as if you were logged into that target container postgres. So, ephemeral containers versus kube-cuttle-execing. Kube-cuttle-execing into a pod or into a container, you will basically see all the resources in that container, but you'll be limited by mounts, that is, by the tools that are available in that container. So, the files will change. You will get network namespaces in IPC and UTS and so on and so forth by default, which is great. It's very handy for doing regular network debugging. So, you can share the pid namespace by using target. I'll talk a little bit about that more. So, let me talk a little bit about ephemeral containers and how to get access to more. So, you may want to get access, for example, to file system mounts and a few other things, and I'll talk about that. So, ephemeral containers are a sub-resource. Technically, they're basically a new resource off of pods. It is now a sub-resource. So, if you want to address ephemeral containers using API calls, you're going to have to use sub-resources, which means that you're going to have to use raw or you're going to have to use custom API calls. I talked about this before. Every time you run an ephemeral container, it shows up in the ephemeral container property, and the statuses, the dynamic statuses, exit codes and so on and so forth for those containers is listed in the ephemeral container status. Oh, yes, unique names must be used. I'll talk about that a little bit more in just a second. Here's a handy command for listing out ephemeral containers. I get all my ephemeral containers and I get all my ephemeral container statuses. As I said before, not all properties are available from kook cuddle. You can run kook cuddle exec to start up ephemeral containers, but there are some namespaces that you don't have access to from the command line. There's mounts and mounts, excuse me, mounts, pids and something else not coming to mind, regardless. We'll see that in a second. Oh, yes, you cannot remove or change ephemeral containers or ephemeral container statuses. So, you'll see this once you try to access the sub-resource directly. You can add ephemeral containers. You cannot remove ephemeral containers. Now, remember that every time you start in an ephemeral container, it's added to the ephemeral containers in the pod and the ephemeral container statuses. This means as you start ephemeral containers, the more ephemeral containers you start against the pod, the greater, the longer that list will be of ephemeral containers. So, I'm running a little short on time. I'm almost at the end of the presentation, so I think it's going to work out well. So, this is an example of a custom API call to create an ephemeral container. In my case, I do not have access to volume mounts using the regular kook cuddle debug command line. So, this is an example of how you do that. Since it's a container, you have access to all of the volumes in the pod. So, what I can do here is do a custom API call, create a custom ephemeral container with a mount to create to access my mounts. Now, there's a good reason why ephemeral containers don't allow you just to mount the copy over the mount namespace of the pod is because the proc amount is an important mount and that affects the rest of the system. So, you have to be choosy about which file systems you mount and that's why you can't just go ahead and share the mount namespace. And if you like, I encourage you to play around with it because you'll get some interesting results. But regardless, you'll have to pick and choose your mounts. This is one way to do it. This is an example of the once again the comparison of the namespaces between the mount, the, excuse me, the ephemeral container I just started with a new mount. Notice we get a new mount namespace. So, it's basically the same as I had before when I started ephemeral container by default. But I created a brand new namespace so I can mount my volume mount that's in the pod. So, that's useful. Here's taking a look at some of the mounts that I have. There's my new mount slash mount. And I can go ahead and I can list that while the files are there. So, if I can mount, if I can do custom mounts, then I have access to some of the data that's in my pod. Once you do that, you can start talking about doing really interesting debugging. And there's also a privilege mode that allows you to do even more interesting debugging. So, after you've gotten, been able to do your own mounts, been able to get access to the network namespace, be able to get access to the PID namespace. And for some things, getting access to the privilege namespace, which I'll talk about in a second, you can do some very interesting debugging. If you haven't already heard about this, it's called Netshoot. It's a container that's specifically designed for debugging. You can start it up as just a regular container, but its utility is very limited as that. Once you start using it as a federal container, its utility really becomes evident. So, it's a system level diagnostic container. It contains tools like S-trace, L-trace, TCP dump, what have you. It's very nice. So, this is an example of starting up a container using Netshoot as a privilege container. So, I'm skipping a few steps here, but that's okay. So, here, along with my volume mounts, I'm adding a security context and setting privilege to true. So, this is basically running this container as root. Even though the container I'm debugging is not a privilege container, I can run it a federal container as a privilege container. Here, I'm going to run Netshoot, and then when I do that, I can do all sorts of interesting things on my target container that you'd never be able to do in a regular container. In this case, I can run S-trace, for instance. So, I can look at system calls that are executing. I can look at all the network resources. If you've got them out set up, you can take a look at the data that's the changes in data that your target container is making to the file system. So, it gets pretty complete at that point. Here, I'm going to do a similar comparison here. Here's my classic kubectl exec, excuse me. Here's my classic kubectl exec. So, this is from slide 40. This is when I logged in originally to the pod to just see all the namespaces as if you just exact into it. This is the one using my privilege container, debug container 2. There, you see those two new namespaces, the C group namespace and the mount namespace. Take a look at that C group namespace. That happens to be the same namespace as PID 1 in that original example in slide 30. So, C groups control permissions. So, that is the namespace. I think of namespaces as being what you can see in C groups as basically being the dials. So, here, we're sharing the same control as PID 1 basically in that pod. So, that's what a privilege container is for. I see. We're doing good. So, with all of that, given that you can get complete access to your pods through ephemeral containers to the point of having access to basically privilege access, I encourage you to look into C groups. This allows us to talk about distrelis. So, distrelis is when you deploy containers that only contain the application that you're going to want to run. They don't contain any of the utilities. So, with ephemeral containers, you can start to start containers beside your application containers with commands like LS and PS and what have you, and given that you can have up to, you know, this sort of privilege access to the pod, you can then run those commands against the containers just as if you were in the container itself. So, this loosens the requirement that you have all the tools required to debug or even really use your containers in spec files within the containers themselves. So, that is basically it. Here's a list of resources. I encourage you to take a look at these guys. They're fairly valuable and they're pretty comprehensive in terms of C groups, which is the next important topic, which I didn't have time to cover, as well as what makes containers work and some of the intentions behind ephemeral containers beyond simply debugging. So, I'll open it up for questions. I think I've got just a few more minutes. Yeah. Thank you so much, Erin. That was a great explanation. I think we probably have time to take one question. I don't see any questions online. So, I'm going to walk to the gentleman. Hello. I'm just missing a piece from your presentation. What happens to the ephemeral container once you're done debugging? It remains as an entry in the pod. It goes away. You get an exit code. So, the question is, what happens to an ephemeral container once it's done? If you exit an ephemeral container, the entry remains listed in the pod. Nothing you can do about that. You will see an exit status within the ephemeral container status that says whether it exited with a code of one or zero. You can get some log information from it, which is handy. If you opened a terminal to that ephemeral container, if you attempt to attach to it using kubectl attach, you'll actually get the last screen you saw. So, it's kind of a double-edged sword. It allows you to see any errors, last errors that were output. But at the same time, if somebody has access to kubectl attach, they could see some sensitive information that might be listed there. Thank you. Oh, and once you restart the pod, the ephemeral containers will go away and you get to start over from scratch. Okay, with that, let's thank Erin again for a wonderful presentation. Good job.