 Hello, my name is Brian Borum and I'd like to show you a new way to visualize what's going on inside your Kubernetes cluster. First of all, let me introduce myself. I work at Weaveworks, the GitOps company. I've been involved in open source software for decades now. Right now I'm a maintainer on a couple of CNCF projects, Cortex and CNI. I worked on lots of other ones and contributed to Kubernetes. But what I'm really passionate about is understanding what's going on inside computer systems. A lot of times I'm saying, what is it doing? Why is it so slow? And so really interested in tools to understand those things and get a better idea of what's going on. What we're going to talk about, what are events, these things that the idea is based on? What are traces distributed tracing? What's that all about? And I'm going to show you a program I wrote to implement this idea and it's called K-Span. If you're watching this talk live at KubeCon, any questions or comments, please type them into the question box. They're watching along with you and I'll get to as many questions as I can at the end of the talk. So the first thing to talk about is events. Events are little messages, little pieces of information about something that happened in your Kubernetes cluster. And they are generated by some component in the system, here by Kubelet or by the scheduler within the system. And they're sent to the API server. So you would typically look at these events using the command line tool kubectl. So if you, at your Kubernetes cluster, if you type kubectl getEvents, you get a list in text form. But it can be a little bit hard to figure out exactly what it means. So I've drawn the events coming from different Kubernetes components. They can also come, these are supposed to be your pods in the system. Events can come from anywhere. They can come from your pods, from operators, controllers in the system. But in terms of what I'm demonstrating right now, I'm using the events that are built into Kubernetes itself. Let's take a look at some events. So this is what you might see if you were to run kubectl getEvents. Each event has a type, which is either normal or warning. It has a kind of cryptic reason code. It is about some object within the system. I mean here the first few are about a pod, then this one's about a replica set. This one's about a deployment. And then there's a message, which is meant to be more human readable. So each one of these is a point in time when something happened within the system. If we drill further inside, I've docked out the contents of one event in YAML here. And from the top you can see an event object is a Kubernetes object. So it has an API version, a kind, it has a name and namespace and the metadata. These are fields you find in pretty much any Kubernetes object. And events are no different in that respect. Each one has a unique name. They all live in a namespace. Then we see the fields I was talking about, the type. The same event might happen more than once. And Kubernetes will count them up for you and tell you the range of times they happened over. This particular one's only happened once, so the first and last timestamp are the same. The message, the reason we talked about a little minute ago. And each one also has a component that it comes from, the source of the event. This one came from KubeLit, which is the part of Kubernetes running on every node. And finally, the involved object. So this is the way that we refer to another object in Kubernetes. It's quite long-winded, but we need all the details. We need the name and namespace of the object. We need what kind it is and the API version we should fetch it at. So that's one event in quite a lot of detail. Next thing I want to explain is distributed tracing. So the idea here is when you're operating a big system with different components running on different machines, that to understand one operation as it flowed through the system, we can collect what are called spans from each component. And we give them an ID saying how they interrelate, a context of an operation. And then we can collect them together in a distributed tracing system. So I'm using one called Yeager, which is a CNCF project. There's a few other ones around. I think Zipkin was one of the first. There are commercial products like Honeycomb and Lightstep. So there's a few of these things. And what they do, they collect these spans. Think of a time span with a beginning and an end and something that happened in that time. And they can put on the screen this kind of Gantt chart view. So this is a timeline, time going from left to right and all the different operations that happened. And they're kind of a hierarchy, how they relate to each other. So that's really useful to understand as a tool to understand what is going on in your distributed system. Now, you might notice that I drew this picture without any spans coming from the Kubernetes components. And that is how things stand as of the time I'm speaking, that Kubernetes doesn't have the code to emit these spans. It will come, but just not yet, still under construction. A lot of debate about exactly how that should work. So the idea then I had was, well, we've got these events which are kind of similar to spans. They come from all different parts of the system and they tell you what happened. So can we turn those into spans and get the same kind of picture on screen? So the answer is yes. I've written a program called K-Span. And it lives at the Weaveworks experiments repo because it's still very much under development work in progress. But what it does, it listens to events and it emits spans using the open telemetry protocol, which a lot of distributed tracing systems support now. It takes a bit of work to match up all the events inside so we can figure out how they interrelate to each other. But I'd like to show you it now. Let's take a look at the demo. I've got a little Kubernetes cluster running and I'm going to apply a manifest. So brace yourself, YAML coming. Here we go. Don't be scared. All it is is we're going to create a deployment object and that's good to create two pods. And we're just using a demo image called pod info from my colleague Stefan at Weaveworks. But it's really just to make Kubernetes do something, generate some events and see what we can do with the visualization. So let me flip over to my terminal window. So I'm going to do a couple of things here. I'm going to print out the events as they come out and use the minus w for watch flag. So it's going to sit and wait for any events to come along. And on the other window, I'm going to apply that manifest that we just looked at. So let's see what it does. Okay, events coming out. It has created pod. It's pulling images. Okay, so more events. It has created two pods, started two containers. Well, that's what I asked for. So that's great. Now, that's the text coming out of those events. I was running caseband in the background this whole time. So let's flip over to my Yeager window. Yeager has this cute mascot. And let's see. So let's just refresh that view. Okay, so Yeager has now heard of Kubelet. And I'm going to ask it to find traces relating to that. Great, we've got one trace. A trace is a collection of spans. In this case, we've got 14 spans coming from, what is that, five different components in the system. Just a minute ago. So that's probably the one I want. Let's go click in on that. Wow, okay. Here is our Gantt chart view. Let me adjust the display slightly. Just click that one up, give me a bit more room. And drag this guy over. So let's see what was going on here. The whole thing took nine seconds to start two pods. We've got a hierarchy of things going on here. It begins with an update to the deployment. Well, that was me applying that manifest. And sure enough, it says it came from kubectl client-side apply. Next thing that happened, something in the deployment controller. We can click in, take a look at that. And we've got some more detail here. So K-Span has pulled out these details on the event, particularly the message. So the first thing was it scaled up the replica set. So that's great. Replicas set. This is just the way that Kubernetes manages pods in a deployment. And you might never know that without this kind of display of how things happen inside the system. So the pod was scheduled. That means assigned to a machine. And then we pulled the image so that I just cleared out the cluster, started out fresh. So it took a few seconds to pull the image down over the internet. And that's a very typical use of distributed tracing UI is to show the bit that took the longest time was pulling the image. Also that it pulled it twice for two pods. So it pulled the image. That's when it finished pulling it. Then it created the container. Then it started the container. And those things were very fast. And it did two pods, which is what we wanted. So there we go. That's the big picture, the visualization of a relatively simple operation. Which I asked for to create two pods. It turned out to involve 14 different steps in the system. Let's go on. Let's do something a little bit further. I'm going to go back to the manifest. So let's say I want the new version. I happen to know there is a 5.1.4. So let's go back to the terminal window. I've changed the manifest. So if I rerun that apply command, well, more events start coming out. Okay. So back over to Yeager. See what it shows us. Find the traces again. So there's two things have happened now. There's the one that we were looking at earlier. And this one, which is kind of just a few seconds ago. So same thing. Let me straighten things out here. So what happened here is that it started pulling a new image. And then it deleted one of the existing pods. So in this process of updating a replica set. Actually, I'm just going to refresh that. Maybe there are a little bit more. Yeah, more events came in, in fact, while I was talking. So that's kind of the nature of this thing, that it's asynchronous processing in the background. And the UI, the Yeager UI doesn't update in real time. But I refreshed it there. So now we can see a bigger picture. That it started by firing up a new pod. And then it killed one of the old pods. I killed another one here. And then started up the new one. So the process of updating a deployment from one version to another proceeds like that. And we're looking inside the operation of Kubernetes. By default, it starts a new pod and then kills one of the old ones and then starts a new one and kills the last one of the old one. So that's a little bit more detailed operation. You can still see the hierarchy of what's going on. And you can still see how these things relate to each other. But it just gives you a bit more of a taste of the kind of things you can visualize with this technique. So that was our demo. And we got a result on screen very much like this. Now let's drill into how K-Span goes about this. Well, here's the really big picture. Events come in one side and spans go out the other side. But how exactly does it work? Well, the first thing that happens when an event arrives is we take a look at it. And inside each event, just like we saw earlier, there's a reference to an involved object, in this case a pod. So we go to the API server and fetch that pod. And then we take a look at it. And what we'll find inside a lot of objects is this owner reference, which is a reference to something else, in this case a replica set. So we go fetch that. And that has another owner reference inside it, in this case to a deployment. We go fetch that. And when we look inside that, it has no owner. It's the top of the chain. It's the thing I, as the operator, applied the cluster to request my two pods. So that's what K-Span does. It walks this chain of references, of ownership of objects, and thus builds the hierarchy of spans. Let's go back to the architecture diagram. So events come in, but they don't come in in the perfect order. It's a distributed system. These events are being generated from all over it. And it's very much in the nature of distributed systems that things will arrive in an order that may not be convenient to you. So first thing we do is we stack up events that have come in on the basis that we might not be able to make sense of them yet. Each one as it comes in, we do that process of fetching the ownership chain that I just showed you. And we end up with a collection of all these things. Now, let me stress K-Span is not coded, well, 98% of it is not coded to understand things like deployments and replica sets. It just takes each object as it comes in and tries to walk the ownership chain and see if it relates to something else. So we end up with this kind of collection of objects and things which happened recently, which are sometimes pending events. And then at some point, say, when we walk to the top of the chain, we figure out that we've got a whole new operation. So we start a new trace. And now we can start assigning spans to these events and generating it spans to emit from the program. Now, the last thing that happens, we're not quite done yet, we hold the outgoing spans because the Kubernetes events that come in just happen at a point in time. They really only have one timestamp on them. So in order to generate the spans that you saw on screen with a longer length of time, what we do basically is watch for... Every time something new happens on an object, we call that the end of the previous one, and we mark the beginning and end times that way. It's kind of fudging it, but it makes for a much better, more understandable display. And you saw things like the pulling of the image. There is, in fact, an event at the start and end, so we can very easily map the time span of that thing to when the next event happened. So we kind of hold them at the beginning to match up to find out how they relate to other objects. And then we hold them just after we generated the spans because we want to put the end time on. We don't hold them indefinitely. If nothing happens for a few seconds, we send it off. We're trying to do a reasonable job trying to balance having a GUI that's reasonably up to date with trying to put the accurate information on the objects. So that's the caseband program. It is a little bit finicky exactly how to match all these things up, but after a few months I got it to work, and it's all open source, so you can take a look at it. Where can we go from here? Well, if you have thoughts, if you're alive at KubeCon, please put them in the comments, questions box. Otherwise, the link on the screen you can open up an issue and let me know what you think. But I'll just share a few ideas I've had about how this program caseband can be extended and integrated. So first idea is more detailed from Kubernetes. I said Kubernetes doesn't emit spans right now, but it can do. So let's say we got some really fine-grained detail information coming out of KubeLit. We could emit that as spans, and then a caseband could integrate together the information from events and the information from spans and put it all together into a much richer display. A more tailored GUI, by which I mean, we could draw this in a way that makes more sense for Kubernetes events. The generic display of Yeager and other distributed tracing systems is oriented in a particular way towards analyzing transactions in a system. So probably there are ways to put this kind of Kubernetes information about deployments and things like that on screen in a more focused and relevant way if we get some talented designers on the job. And lastly, what is upstream of the point where something happens inside Kubernetes? So if you think about rolling out a new version of the software, maybe something happens in your CICD system. You could get spans from there and link them all together. I work for Weaveworks, the GitOps company. So of course, we would expect the moment of which a change is applied to Kubernetes to come from a Git commit. So that would be this operation. And how did we get here? Did we go through a workflow? Was there an approval testing environment? All this upstream information in how did we get here and why did we apply this change? All seems very relevant to understanding your cluster. Well, thank you for watching. We'll flip to question answering live in just a minute. So if you do have any questions, you still have time to type them in. And I'll just leave you with a few links there. The code for the caseband is there at GitHub Weaveworks Experiments. OpenTelemetry, which is the protocol linking everything together and Yeager, the distributed tracing system I used in the demo. Thank you very much.