 Welcome to KubeCon on the last Friday. I hope you guys had a good week. I know I did. It's nice to be in person once again for these talks. So times that I cherish the most is being able to see all you folks and communicate with everybody here at the con. So you're here for KubeKuttle said what? Because we need to interpret sometimes what KubeKuttle is trying to tell us about our applications. My name's Chris, Christopher. I shorten everybody else's name, so I just go by Chris. I'm a senior cloud native engineer at RXM. I'm an instructor and I'm a consultant. So when I'm not in a class with a bunch of students, I'm usually working on a project to help people adopt Kubernetes and cloud native applications and things like that. And I like to talk, so that's why I'm here. And I've done it in the past. So you can check out some of the other talks that I've had the last couple of years here at KubeCon, although it was virtual. OK, so before we get started on interpreting what KubeKuttle is trying to tell us, we need to think about what Kubernetes is doing to deploy our application. And so this is a resource creation flow that shows, often, we're the user here in the upper right-hand corner or upper left. And we're a little bit disconnected from the actual deployed application that happens many steps down in this flow. And so there are some swim lanes that happen here. And so we've delineated those to show that you create the deployment as the user, it's then, or the controller, let's say, it's then the controller's job to actually submit the API requests to create your applications or to create your pods. And then the scheduler has to do its work to make sure that those things get home. And then the KubeLit has to kick in and communicate with the container runtime and actually create containers and have running processes. And in any one of these swim lanes, something may go wrong that you may not anticipate. And you'll need to figure out what's going on. So we're going to look at each one of these swim lanes in turn here. So Kubernetes resources, all of them have things like conditions, phases, states, that some of which are summaries. That will give you a little bit of information, some clues about the status of your application. Other things are very detailed. And it will give you exactly the message that you need to know about your application and what's happening with it. So they'll all give you some sort of clue to help you with that. When you first learn in Kubernetes, it can be a little bit difficult to know where to look and what to pay attention to. I create a pod or I create a deployment and it says resource created. Oh, cool, everything should be great, right? Well, no, not really. A lot of stuff has to happen in the background before that actually becomes true. And so if you don't know how to dig in and actually find that information, then you're sort of left stranded and wondering why can't I connect to my application? Kubernetes said it was created. So we're going to go through those steps here. So things like get with events, get the object itself, give you some clues. It's always a jumping off point. We get the object to get some sort of summary information and then go from there. We can use modifiers like minus 0, so outputting YAML, outputting JSON, and then getting specific status information using something like status.conditions and extracting the actual message from the condition of the object. And of course, kubectl describe is very helpful as well. So we're going to look at some practical examples. We're going to start with the first top two swim lanes here. So we're going to focus on, as a user, we create the deployment and then what happens, or the controller rather, and then what happens when that controller fails to do what we expect it to do. And so in some cases, we can get some details from a command like kubectl describe. So for deployment, it has some transient states that it will express. So for example, available, this can be a little bit complicated because it requires understanding of the setting which is max unavailable and the other setting which is the minimum ready seconds. So some number of pods has to be available for a certain amount of time based on what was expressed in the deployments, essentially rolling update strategy, which determines whether or not the minimum number of replicas are available to clients. And so this will tell us yes or no. Is the thing actually available to clients? Or are there not enough replicas to make that available? Progressing happens when we have new RSSs. So a new RSS comes around when we have a new deployment or a rolling update. And so in any of those cases, we'll see the progressing status condition show up. Or if we have a scale up or scale down, we're making changes to the existing application. And then that shares a condition replica failure with the replica set. So both deployments and replicasets will express replica failure if they can't create a pod for any sort of reason. Stateful sets and jobs don't have as much detail. So stateful sets have a status.conditions part of their object. But it's not easy to find. And it's not very detailed. So it's just not as good as the deployment when it comes to being clear about what's going on with pods under a stateful set. And then job has one that's very transient. That's whether or not something finished with a zero or non-zero. But if it's still running, it doesn't even present that to you with kubectl, right? So it's not queryable at that point until it does its work. So we're going to look at a practical example of these guys. So I've got a couple of yamls. I'm just going to apply them. And then we're going to see what happens. So we've created a bunch of controllers, all the ones we mentioned. And so I'm just going to kubectl get the deployment, the RS, the stateful set. And I'm using the short terms because I'm lazy. And job, right? And so this is kind of our jumping off point, right? I have a couple of deployments. You can see that they're not whole. I've got a stateful set. It too is not whole. It has a lot less information for me to understand why it's not whole. And the same thing with the job, it's not done yet. I don't know if it's in progress or if it can't create a pod or what's going on with it, right? So there's a limited amount of information that gets displayed here. But if we go through each one, we can get a sense of what these different columns mean and how they can help us. So for example, ready, this is important concept in Kubernetes' readiness probes. And so at this point, we don't know if two of our pods have passed their or have not passed their readiness probes or if they just don't exist. So we can look at something like available and up to date to help us with that. So if the pod doesn't exist, it's not going to be up to date. And we don't have a new revision of this deployment, of course, because it's brand new. So we know that we were able to create two pods, but two are missing. We still don't know why. The RS's actually have a little bit better details because they've got the desired versus current and ready. And so now I've broken out a little bit more detail information. We know it's not just that the pods aren't ready, but that they really just haven't been created because the RS has four pods that it wants to create, but only two exist for the one RS. And then this one has zero for whatever reason. Staple set, again, we don't know. They're just zero of three ready. There could be a pod there, but the readiness probe may not have passed. Or again, there's no pod that exists there. So what we need to do is dig deeper and ask these specific controllers about their statuses. So I can do kubectl describe on one of these controllers. So I'll do deploy. And we'll do the dep ex. And so our only condition that shows here available as false means that, again, we don't have enough replicas. So this ties into the max unavailable. This is just the default. So we only have two or four pods. So of course, we've breached this threshold and we're below the max unavailable. And so we expect this to be false. Replica failure is true. Here's our first indication that the controller is failing to create pods because it has this set to true. And it's still trying, of course, because it's under progressing, set as true. But the only event we really get here is that the replica set scaled up to four. So what's going on here? We don't have enough clues. So I need to go to the replica set to really get the information. And of course, I get a huge amount of events. So the replica set, remember, in the flow is the actual controller responsible for creating the pods. So the deployment did what it was supposed to do. It created the replica set. But the replica set is not being able to create the pods because I actually ran this in a namespace that has a quota that would be deliberately violated so we would actually see something that was wrong. So knowing where to look is key to this. If I'm trying to look at my deployment, when what I'm looking for is missing pods, well, the deployment's not responsible for that. It's the RS that's responsible for that. And so if I go directly to the RS, I can see events like this. Now, when I don't have that two-tiered relationship between controllers, of course, I can go directly to the controller like a staple set and get the same message, right, that there's a quota in place that is preventing that as well. And the job's the same way. So if I queried the job, you'd see the quota. Now, there's one that we didn't see that is not part of this quota scenario that I've set up. So I'm just going to list all again. So there's one more deployment here called SAEX that is failing to create one pod. And it looks very similar to the other problems that we're having. Is it a quota issue? They look very similar, right? If we actually look at that, again, deployment's not going to tell me much, right, because it's not what's responsible for creating pods. So just wanted to reiterate that. So what I need to do is go to the RS. So now in this case, it has nothing to do with the quota. This is a failed service account. So the pod spec is relying upon a service account that doesn't exist. Again, I made this scenario artificial just so we could have something to look at in our troubleshooting scenario. But just looking at the get output doesn't give us these details, right? Being able to go into the actual events of the controller and the correct controller will give us the information that we need in that scenario. So going back here, when we look at the deployment stateful set with something like kubectl, get a lot of the issues that we're going to have are going to look identical. And we're going to see later on with some of these other scenarios that they're going to look identical to those scenarios, too. Replica sets give us the best details about things like ready versus current, not matching desired. Pods are going to be missing, right? So I can't query pods that don't exist for their status information. I have to rely on what the controller tells me because I can't query something that hasn't been created yet. And you can actually see the message field that gets populated as a related condition. So those events will populate into the message field of the controller. So common causes here are things like quotas, missing service accounts, as we saw. Now, if your controller can actually create pods, then we're moving down a swim lane. If we see that pods are created, then we need to go and look at pod objects for their status, their phases. And so some of the macro states that exist for pods include things like pending. So this is when the replica set has actually successfully submitted the pod spec to the API. But the pod can actually be created. Remember that pods are represented by the pod sandbox, which is a container. And so the first bullet here that says that the containers have not been created includes that infrastructure container, that pod sandbox container. Running, then, is where we actually have scheduled it to a node, but the containers can be in any state. So they can be restarting. They can be running. They could have gone into a crash. In some cases, we'll get that actually as a message. Succeeded and failed is very straightforward. Zero non-zero exit codes. And then unknown is not something you'll see very frequently, but it's a lot of times deals with the communication between the host where the cubit is running, where that pod is, and it's not updating the status. If we go a little bit deeper, so these come with kubectl get pod. That's, again, a good starting point where we jump off. But if we get something like kubectl describe, we can get Boolean conditions that let us know if something's ready, if the containers have been launched or they have failed to finish via initialized, whether or not all the containers in our pod are actually ready, passing their probes or not, and whether or not it's been scheduled. And this will give us some more details. So if we look at the next swim lane here, there's the scheduler responsible for finding pods that have not been bound to nodes once the controller has submitted them, and then binding them to nodes. And so if we look at some examples, I'll clear my cluster. And I'm going to need to change namespaces so I get out of the quota so we don't deal with that. So we'll switch to a different namespace without a quota. We'll apply a new set of pods here. And I'll just dump everything because what we're going to need to do is look at the pods themselves. So we'll use get all. Now, if we, again, go back to the controllers where we started the first time, these conditions look fairly similar to what we saw when there was a quota problem or a service account problem. So they're not going to be very helpful, right? We have pods, but they're not passing readiness checks. And so that column of readiness includes everything up to the point where that probe passes. So a lot of conditions are trapped. We already saw upstream that other conditions are trapped by that. So we can't really rely on the controllers to give us that information. We need to go directly to the pods in this case. So I can use get on a pod. We'll use the res ex example. And if I go to something like a YAML output, I can see that under the conditions for my pod. I do have a false for the pod scheduled condition. And then I get the actual message as the reason. So this will take the event that traps. The reason why the scheduler couldn't schedule it and inject it so I can extract it directly. So I can actually traverse this output if I wanted to with the JSON path outputter. Go status conditions and extract this message and find that each one of my pods have some sort of scheduling issue. And so if we go through each one, I'll use describe on the other ones, we can see that nodes aren't available because of things like resources or volumes. And because we're dealing with the scheduler and pods, the messages are going to be related to the pods. I didn't even bother going to the controllers because they would just lead me on a wild goose chase. So if I go up one level, let's do describe. They did what it was supposed to do and that's the event that we see. It created the pods that it needed to create. It's just the fact that the scheduler can't reconcile the use of the resources. So going back to the pending symptoms, the controller outputs are going to look identical to what we saw before when the controller was having an issue. But the pods are going to actually exist as opposed to before when pods just didn't exist. It's going to return pending, of course, in this scenario. And then you will see pod schedule is false. As part of its status condition, you can get that message field. But there's not going to be any containers. So the difference between something that you need to troubleshoot at the container level versus the pod level is that essentially, container status will be no, there are no containers because you can't create containers unless the pod has actually been scheduled. So some common causes here, things like insufficient resources, if you don't have enough CPU or memory. Things like node selectors. So if I use a node name, a node selector, an infinity, that would prevent that. And then volumes. These are kind of the primary three things that the scheduler used to schedule a pod. The last one here, priority class preemption, is kind of an advanced use case that I didn't go over here, but can look very similar to the first, which is the insufficient resources, because you'll have a bunch of pods that are running that may be a lower priority class. And so they will then, if a higher priority pod comes along and resources are scarce, those pods will get evicted in favor of one that has a higher priority and needs those resources. In which case, the events you'll see will look like insufficient CPU or insufficient memory, because the new pending pods that got evicted, there are replacements for the old ones that were running at one point. But unless you see that actually happen in sequence, it's hard to tell that it even happened at all, because the new pods have no relationship to the old pods that were actually running. So that can be a tough one to look through. So running. So once we have pods that are actually scheduled and running, we're getting down to the bottom swim lane here. Containers each have their statuses tracked. So this includes in containers. This includes init containers. This also includes ephemeral containers that that's enabled on your particular cluster. Again, you can use things like the container statuses to group this information and look through it. There are three different types, waiting. This is kind of the default state of a container. So anytime it's waiting to actually be created, if it's mounting volumes that config maps or secrets, or if there's a crash loop or an error, it will go into a waiting state. Running is anything that's, of course, got a process. And if there's any hooks that have been specified, then those have completed successfully. So you can determine that. Okay, my hook was ran successfully. And same thing with terminated. If you have a pre-stop hook, you won't see terminated until that completes. And then you'll get your zero non-zero. So that sounds pretty straightforward, right? Everything's fine, right? You've got a controller that has created a pod. The scheduler has scheduled to a node. The Kubelet has created the containers. Why are we even talking about this? Right? Everything's just fine, correct? Right, well, if that's the case, then we actually wouldn't be here talking about it. So let's clear our cluster again and we'll apply our new set of pods and see what sort of things we get into. So it's a little bit fast with the get all. We'll run that again. Right, so again, looking at the controllers, right? If we go from the top down that we've done before, they look very similar to other cases, right? It's a hard place to start to debug from. But if we look at the pods directly, we get a little bit more details. We know that they have one, in one case, two containers, but one of them is in peril. So in this case, we get an extra status message. This was not included in the official pod statuses. This is actually to help us. So we know there's something causing an error here, causing a container to fail and get restarted. And so just having a running in that case would be a lot harder to troubleshoot. So there are other cases that we'll look at at the very end of this that help us debug by replacing the macro states of just like pending running with these other detailed status messages. So although we haven't really looked at that yet, there are ways that we can populate the status section and give us more clues when we wanna have a easier time debugging, right? So in this case, I can look at my containers in the container status array. And the easiest way to do that is getting the pod and we'll do the restart pod. And you can see that we've got two containers in here and we've got a restart count of four and a restart count of zero, right? And so this one's not ready and this one's ready. And so we can very easily see that the database here is failing and then we have the message for the waiting state. So that's in the waiting state because it's in the crash loop and the related message right in front of us. If I were to instead, let's say describe this pod, I get similar messages from things like the events but back off restarting fail container, which I don't know, right? Because it's just a generic message but by leveraging something like minus O YAML with the get, I can relate the actual status message to the container that has the problem, right? And so being able to customize what I see coming back from Kube CTL helps me to debug faster. So I know we're running out of time pretty quickly. Now there's one use case that I wanna point out here. This one seems to be healthy and running just fine. This is an example of a liveness probe and so that's easily determined from just describe. You'll see that as an event. And since we only have one container in there, anyway, you can see that the readiness probe has failed for that one. So easy enough to get from events. But going back here, nothing seems to be wrong with this container, right? It's running, it's ready. The controller is happy. But if I show you where it's running, so it's running on worker one and I go over here and I stop worker one from functioning in my two node cluster. If I continue to get this pod, I know that pod's not running but Kubernetes is telling me that pod is still running because remember in the swim lanes, the agent responsible for reporting these things upstream is the kubelet, I just disabled the kubelet. So I have no indication that this pod is no longer available to clients from this standpoint. But if I again dig in, I can see that the readiness condition has set the false because the node controller found that the heartbeat of the kubelet where this pod was running has decayed long enough to declare it unready. We don't actually know what's wrong with this pod because the kubelet is no longer communicating that but to be safe, we stop sending traffic to it. So this will affect things like end points controller. So this pod would no longer receive traffic from clients but if I go back to my get, still running. So you have to know where to look sometimes. Now at a certain point, this is gonna go to terminating because Kubernetes will eventually give up on the pod or give up on the node and terminate all the pods on that node because the node is not coming back. However, if we just simply bring the node back, kubelet will heartbeat again and then the readiness that we saw in the describe will actually flip back to true and it will continue to receive traffic. So sometimes running doesn't actually mean running. Okay, so turns out all was not well. So some of the symptoms and some of the differences between the symptoms. So again, controllers, not gonna be very helpful. Running is gonna be set on your get pod, ready is not gonna have equal values for your current desired. Schedule will be true because we're past the scheduling swim lane. Things like ready and containers ready are gonna be false because those containers are gonna be unhealthy. And so you can find messages like containers with unready status will give you container names within things like different outputs and the different states of the containers of course will change based on cause. The difference between something like a probe failure and an actual error is that running will be true for a probe failure, ready of course will be false. Something that's crashing will be in a waiting state whereas they'll both share the ready equals false state, right? Now with a node failure again, controllers are gonna look normal because we think the thing's still going. Once that timestamp decays about 40 seconds, you'll see a mismatch between ready and available to show up in your cube CTL output, right? Running and ready is gonna be there and that's gonna be completely misleading. So common cause as we saw with this is a node failure, like kubelet is down, can't report in. So what happens when we saw one example of this, right? Error versus crash loop back off. What happens when we have other examples? Clear this cluster again and we'll apply the other, this is Kubernetes helping us. So the status condition of these pods has to do with what errors, right? They're actually in depending state. So if I look for this, if I use this command JSON path where I get the items, so all my pods and I look at the phase of all my pods, they're actually all in a pending state. But Kubernetes is helping us by saying, well, this one which is trying to mount a config map can't get that config map. This one has some sort of problem with an image, right? A mis-type in an image or some other issue. This one is not so clear, just container creating forever. So I actually have to dig in on that one. And now in this case, it's a secret, right? We're trying to mount a secret for the pod, it's not working. So something like where a container gets stuck creating forever, it's got a dependency and that dependency has not been fulfilled. Now in other cases like config maps, better details, right? You can see it just with kubectl get but in this case we have to dig in a little bit more. Okay, so, right? So even though the status.phase is always gonna be one of the standard phases, getting the container status, state.reason is gonna give you more clues as to what's actually happening and Kubernetes helps us by surfacing these types of messages for, and we didn't cover this here because it was in the previous section so that you can help you debug the commonly occurring issues that exist. So now you kinda know what kubectl said. So thank you guys.