 Hi, my name is Nick. This talk is called in search of a cube cuddle blend command No, there's no cube cuddle blend command right now This talk is about why there should be one What it would do why it's hard to build and why you should care about it Let me start with why I care about it. I Work on tilt them. We want to make developing on communities a pleasant experience You edit your source code tilt updates your cluster We want that you to help you understand the progress. It's making and how that change is affecting your app Lots of tools have this problem and they all solve it in a different way. What does this mean exactly? Let me give you an example You apply a deployment. You want to check if it worked One approach to a cube cuddle blend command would trace each cause to its effect and tell you how far along the deployment is Developers often want this tracking in the reverse direction as well You know a pod is crashing. Is that pod from the deployment you just created or is it from an older revision of that deployment? Where did the different containers and sidecars of the pod come from? When we tilt team Started down this path We thought this would be easy. It turned out not to be so easy We tried a lot of approaches Many of them failed. It was pretty frustrating. I Don't want to put you through that. I Want to explain the problem to you in the way that I wish someone had explained the problem to us Because this problem has old theoretical roots We're going to talk about those roots We're going to take a tour of some of the tools that try to solve this problem and how they solve it and I'm going to talk about some ways future depth tools might make this easier if anyone here works on Kubernetes now, I don't want this to be just a talk about How to write tube cuddle blame if I wanted to keep cuddle blame. I wouldn't have given this talk I would have used that time to build my own cube cuddle plug I'm more interested in the abstract problem and its history Because if you understand the history you can make sense of the different approaches You can reach for non Kubernetes tools even non computers for inspiration on solutions And we might to see better Kubernetes debugging tools in the future So let's talk about why this is hard The fundamental Kubernetes architecture is a control loop The system runs in a loop. You declare a desired state What the loop does next is a function of the diff between the desired state and the current state I tried to describe Kubernetes to a friend. She Researches history of science She says oh Nick. I I know what you're talking about. We've been talking about that for decades Let me send you a paper She sent me this paper origins of feedback control by auto mayor Great paper old control loop papers typically have a ton of calculus because they're really about functions on differences This paper is about history. It's tons of fun very readable not much math You probably don't interact with water clocks and windmills day to day But you do probably interact with thermostats and they have this property that demonstrates the cyclic relationship between cause and effect that we're describing Let's say the current temperature is 60 degrees Fahrenheit You turn the thermostat to 70 degrees Fahrenheit You wait a bit Then turn it to 90 degrees Fahrenheit The heat is on did the first setting make the heat turn on or did the second setting make it turn on? What does it even mean for one change to cause the heat to turn on? The other important part of a control loop is that it's a declarative system It inherits a lot of the same problems that other declarative systems. You might be familiar with CSS is pretty widespread. You may be scared of CSS I was a UI engineer for many years and I think CSS is great I think it is one of the great declarative programming languages of all time Now when you look at the initial CSS proposal the inventories of CSS noted that it Hey, it has this nice property. It makes styles very pluggable It gives you a lot of room to change the implementation and for different contributors to contribute to the style and to optimize the implementation later The trade-off is is that CSS Like other declarative systems can be very hard to debug unless you have a good mental model about what the underlying system is doing. I Remember an old story from the Gmail UI team where someone added a bad selector that looked innocent and That selector killed the performance of the entire page. It took them weeks to figure out why Even to figure out which selector caused the problem Modern the CSS the buggers have come a long way to explain cause and effect You can see the order the rules are applied in where they're declared You can see rules canceling each other help and you can edit them in place This is frankly what we should aspire to in Kubernetes land But now we're going to take a look at what exists today in the Kubernetes ecosystem The sample app we're going to look at is a stripped-down version of tilt We build and push an image. We apply a deployment We track the deployment's progress and wait for the first pod to become ready Quick review of the life cycle of a deployment so you understand what's going on in these examples a Deployment creates a replica set and the replica set creates a pod Then the pod is the bit that runs your containers that runs your code We're going to look at how these tools differ the pros and cons and where they fall down If you want to follow along, there's a repo where you can run these examples. It's at the bottom of your screen there The first example I'm going to show you is what the tilt team did first Again, we thought this would be easy Each deployment would get a label. For example, we might apply the label deployed at 10 a.m. January 10th If you see a pod with that label, you know, it belongs to this deployment. Here's what the code look like This is the Kubernetes go client library client go client go Gives you an object called an informer and informer handles watching for Kubernetes objects retrying when there's an error and notifying your code about every change We can configure it to only watch pods with the label. We care about which is what you see in bold here Let's watch what this looks like. This is an ASCII cinema video of the Tool running our code is going to print an update every time the pod status changes and We see the deployment list is obsessed All of our examples have a crash flag that lets us see how it behaves when the pod crashes dash dash crash So we can watch this deployment come out and Eventually we're going to see it hit an error Great. Perfect. We saw it. So at this point We thought we had solved the problem. We were pretty happy Now this is very efficient. It only watches the pods that matter We need to inject labels, but that's okay. We can do that for the kind of tool we're building Then we watched it and everybody hated this and they complained all the time Because we had made a mistake We assume that Kubernetes would see the label on the pod template and would see that that was the only thing that had changed So it would update the existing pod with the new label That is not what happens The what happens and the way it's specified to happen is that the deployment controller will notice That the pod template spec has changed and replaced the pod entirely It was very important to people we learned that if nothing changes the pod should say as is it shouldn't restart and People hated the behavior where it started every time. So we looked for alternatives The first alternative we will look at is cube cuddle rollout. It ships with the cube cuddle The example code that we're going to look at uses cube cuddle rollout as a library Let's watch So it's gonna wait for the deployment rollout to finish and It's a little bit repetitive, but it's good success it deployment successfully rolled out Now let's look at what happens when the pod crashes. We start command with bash dash crash We build and push the image deploy the deployment wait for the deployment to finish and Wait and a kind out Okay, what happened When you dig into how cube cuddle rot is implemented, it's very simple It watches for changes to the deployment it checks the spec field on the deployment Against the status field on the deployment to see if it's finished Here is the code The full function has a few more if branches But it's not substantially more complicated than the checks that you see here the checks in bold They check that the number of updated pods is what we expected it to be What cube cuddle rot is really good at is understanding the planets Any information about a pod that's propagated up to the deployment it can find that out But because of the information we care about Isn't in the deployment status. It can't tell us why things are failing. Let's look at a different tool Let's look at helm Helm also has the ability to track progress. It has two flags dash dash above and dash dash watch That let you track the progress of a helm upgrade Just as an aside helm helm is pretty awesome if you want tips on how to use The go client for Kubernetes. I highly recommend you look at how helm uses it The code is very readable. The code is very well organized of The examples that I wrote for this talk. This was the easiest to write It uses helm as a library to watch progress and it's a couple of lines Let's Watch a video of what it does Again, we're going to build and push an image deploy a deployment and wait on the deployment to come up Deployment is not ready deployed successfully. Great. That is exactly what we wanted Let's try again with a crashing pod We're going to apply the new deployment with the crashing pod deployment is not ready deployment is not ready The pod is still not ready and it timed out. Okay, let's let's try to figure out what's happening When you unpack the helm code, what does it do? The helm code pulls the status of the resource it just deployed It has a big switch statement of all the different resource types It knows about it checks their status and it checks their children's status and it does a type specific check to see if it succeeded Here's what the code looks like It's a bit more sophisticated than the cube cuddle rollout command The important bit is the to get new replica set call in bold We're not going to show the implementation But what it does is it makes a best guess of what the current replica set is Based on the creation timestamp and whether the fields match the deployment Then it does a similar check to what cube cuddle rollout does to see if the deployment and the replica set are finished This is a fine approach. It's easy to understand and easy to reason about The downside is that there is no universal definition of done Each new resource type has its own definition in that big switch block that helm team maintains Those definitions are easy to add, but still you have to add them The third alternative we're going to look at is to spy trace Now we're not going to look at too much of the code I will warn you the code is very dense. It does a lot, but it's very comprehensive and you can learn a lot by reading it Let's take a look at a video of it in action Unlike the other ones we're going to pause in the middle of this video just to just to understand it a little bit better Because there's a lot Cues by a trace starts to hint that blame isn't a straight line It doesn't go straight from deployment to running pod We can see the current pod and we can also see the pod. It's replacing Let's unpause the video and watch it roll out now. What's going to happen is that? We can see the New pod has become ray and it has replaced the old pot Great Now let's look at what it means to crash Again, we're going to watch it roll out and We're going to pause at an opportune time right here Now you can see that the current rollout is crashing and it's crashing with the error message container completed with exit code 1 And you can see that because the current rollout is crashing the previous rollout is still up Revision 10 has not yet replaced revision 9. This is awesome. This is this is what we wanted Let's dig into how cubes by trace works Now the other tools we looked at track resources top-down. They started with a deployment and looked to its children Cube spy trace doesn't do that If you look at the bold above it's watching every pod in the namespace why So cubes by trace does both a top-down and bottom-up analysis First it creates tables of every type it knows about It looks at an annotation of deployment to find the current replica set Then it looks at all the pods in the namespace and at the owner ref at each pod They figure out which plot pods belong to that replica set. This seems like a lot of work But cubes by his insight is that you need to do both The owner references tell you definitively these pods come from this replica set But the deployment is mutable it has revisions and the owner references can't tell you Which revision of deployment created which pod they simply don't have that information To find out which deployment created which pod We need the deployment controller to pass some information down the object graph And we need to check that that info has propagated So this is great We have to build tables of resources up front. We still have to do a lot of custom analysis for each resource type This approach isn't easily portable to other types or to custom resources. I want to Briefly talk about what tilt does today, and it's not that much different than what cubes by does It's less comprehensive than cubes by which does a lot of different checks Tilt does use owner references heavily Every time tilt deploys a resource it grabs the UID that cube cuddle apply returns It watches all pods and every time tilt sees a pod it climbs the owner references to see if that pod Belongs to the thing that we just deployed Now we still need to do the top-down matching, but we wanted this to work for any type not just for deployments So we use a label approach It's a little bit technical, but I'll try to walk you through it We inject a label into the pod template spec. They are not random labels They're and they're also not auto incrementing revision labels like deployment controller adds their content based labels They're a hash of the pod template spec This ensures that the label changes if and only if the contents of the pod change and This gives us the semantics we want where Kubernetes will restart the pod only if the contents change Now deployments are special and that they can guarantee that this label will make it down to all the pods This isn't true for all resource types So tilt will check if the label made it down to the pod and if it did it we will only pay attention to pods that match the current template hash label Here's what it looks like We're gonna watch the video and again, we're gonna wait till the deployment comes out and we're gonna pause briefly Now when we applied the deployment we got the UID of that deployment and we also looked at the content based label on Each pod that we're watching and we saw that we actually saw three pods two of the pods had a Content label from a previous revision of the planet, so we're gonna ignore those But we are going to keep track of the new pod Let's wait for the sys aid and we're good Now for completeness, let's look at what this looks like when it's crashing Again, we're going to build the image push the image Apply the deployment ignore the pods that don't match the current revision and we can see the error of the pod that Corresponded to this revision So this gives us what we need. I am not super happy with the solution and I'll tell you why All the indexing and owner of traversing means this is very complicated and it's hard to make this perform well The doubt the upside is that it does work well with any resource types It allows us to track the process of custom research track the progress of custom resources It allows us to properly display events and events are super helpful for debugging So I want to close this talk with a small rant if you'll indulge me. I Don't want you to come away from this talk saying oh Cube cut a roll out is bad cuba spy trace is better tilt is the best All of these approaches have trade-offs. Some are simple and fast and efficient Some are more complex and give you more information, but are slower When I was looking at examples for this talk, I looked at cube cuddle tree a cube cuddle plug-in that visualizes the owner reference graph I Laughed out loud when I read this comment directly from the code Cube cuddle tree just grabs all the resources in a namespace It sets the QPS to a thousand so that it doesn't get throttled and This is not me shaming cube cuddle tree because this is what it needs to do I hope you see how much that stinks and it's not because anyone did a bad job It's because these are inherently hard problems to solve in a declarative system and to solve it We need to do some more work But if the underlying infrastructure gave us a little bit of help and did some things automatically for us That would make it a lot easier So what might that look like? Let's start with the bottom-up analysis. What can you figure out when you look at a pod? Right now Kubernetes gives you owner references, which are good But there is no API to easily traverse them the Kubernetes API is Very much optimized for querying types not for querying graphs of objects Which is good if you're trying to build a controller like a deployment controller But not so good if you're trying to attribute blame The other thing about the owner references is they don't really tell you which revision did what? They only tell you Which resource it came from then there's the other side What can you figure out if you're just looking at a deployment? What's weird is that there are a lot of common patterns That you see in different resource types Most controllers have a way to propagate information from the parent to the child To see which revision caused which child change And lots of controllers have ways to propagate status from the child to the parent But they all do it inconsistently they all reinvent the wheel every time even though we've kind of established the patterns by this point And we're getting to the point where it might be worth thinking about what would a generic API look like What are the common idioms that all controllers should be using to this? Lastly, I want to tease you a little bit The Kubernetes ecosystem is only going to get more complicated We're seeing more custom resources. We're starting to see mutated mission controllers, which are really cool But they modify pods and add sidecars and add volumes and other things But we can't tell they came from those mutated mission controllers What would a debugging experience look like that was as nice as the CSS debuggers of today That lets you see a hierarchy of rules that lets you see what change each one made and that lets you Attribute playing so thank you very much for listening to my talk Again, my name is Nick. I want to give a big shout out to Ellen Korbs and Sasha Mumbart's who helped me put together this deck and gave a lot of feedback on this talk that helped make it a lot better than What it was at the beginning. I Also want to thank all the projects I featured in this talk client go Cube cuddle helm and Cube spy. I hope none of them think I was criticizing them because I'm not I think these are all awesome projects And now we have some time for questions, so let's go to the questions