 OK, well, we're about to meet out. Hey, everybody. Welcome to This Week in Cloud Native with this episode number five. Good to see you. Good to see you. I'm saying hello to somebody who's coming to say hello. Good to see you. Darkside21. Awesome. Welcome, welcome. All right, so if you're in the chat and you want to say hi, just drop a note into the chat. And I can highlight it, ask your questions, anything else like that. Just let me know. Happy to raise you up in this way. If you are here for the first time, one of the most important things to understand is that I keep a weekly notes document. This is a carryover from something I did in TGIK, which was, thank goodness, it's Kubernetes. And so I've created for that, I've created a team account. And let's just go ahead and check that out real quick. Let me set up my environment so you can see it. So this is the Week in Cloud Native. If you go to hackMD slash at TWICN, you're going to see this week's notes along with any previous notes that came earlier in the session. These notes are editable by you. If you sign in, then I can see who's actually putting content up. If you don't sign in, you won't be able to see it there. But if you do sign in, then you'll be able to edit the content directly. Just notice something I want to change here. Give me one moment. So this is where I keep my notes and everything else like that. Anybody else who wants to say hello, go ahead and say hello in the chat, and I will raise you up. Hello. Good to see you. Welcome, welcome. So this will be an interesting session. But yeah, let's dig into it. So as you know, there's a ton of really great content on Cloud Native TV. I'm just one of the many shows. There are playlists for each of the content. So if you missed a show or you want to make sure that you catch a show, you can actually go check out the playlist. And all of our sessions are captured and then upstreamed into a YouTube playlist so you'd be able to see them later if you missed them. So don't worry about that. There's lots of content that happened. There was actually a really great CNCF face-off with Matt Stratton. It was actually, I can't remember what it was, like the Rogues Gallery versus, I can't remember what the other piece of it was, but it was a really funny show. You should definitely check it out. It was really a kick. Search magic with Siam, LGTM with Rockhood is doing OPA. And I think that Spotlight is also doing OPA. So lots of OPA love this week. If you're not familiar with OPA, it is the open policy agent. So there's lots going on there. Thank you, I'm doing great. Great to have you here. What else do we have? Oh, you know what I just realized? I wanted to change this real quick because I'm realizing it is redundant. There we go. Nope, that's not what I want. You get to watch me do this kind of stuff live. Perfect, there we go. So if you want to go to one out of our notes, you can see here in the bottom left-hand corner it says hackMD.atTwicken. That's where our notes are located. We have the cognitive classroom with Kat Kosgrove 100 days with Anias and Fields tested with Kazzlen. All of these great shows, definitely go check them out. One of the things I wanted to point out this week, there was a lot of hubbub of course on Twitter because that's what hubbub, that's what Twitter is for. It is a hubbub factory. There's always something going on. And this week, one of the things that was raised up was the API removals in version 122. Now I covered this in 116 as well in a session very like this where we talked about like basically how to identify those manifests that have been deprecated and things like that. And I figured I would dedicate this episode to that same sort of a piece of it. And we're gonna get into that here in just a little bit. But if you want to jump ahead or read ahead, you can always jump over to the Kubernetes blog that's blog.kh.io. And the first one here is a really well-written and well-thought-out post from Krishna Killari at Amazon Web Services and Tim Bannister at the Scale Factory writing a blog post on Kubernetes API and feature removals. I don't know if it's feature removals per se, but like API removals, portions of the API that are being removed from the generally available API inside of your Kubernetes cluster when you deploy version 1.22. And we're gonna kind of play with that and see what that looks like and how that feels during this episode. But if you want to read ahead, this is where you can find out information. So the version 1.22 release will stop serving. This is an important distinction. Stop serving the API versions that we've listed below. These are all beta APIs that were previously deprecated in favor of newer and more stable API versions. So some of the stuff that's been removed are things like the custom resource definition in this particular path, right? And we're gonna play with this a little bit. We're gonna get hands on with it. So if this is confusing to you at this point, don't worry, we're gonna get more into detail. I'm gonna show you just exactly what this means to you in your cluster during this episode. That's really my target for this episode. The API service, the token review, subject access review, certificate of signing request and lease. And this is gonna be a really big one. This is probably gonna be the one where people really get caught out is ingress, is no longer going to be available at extensions V1, beta one and is now or at the networking k8.io of V1, beta one. So we'll see what path it is available at and we're gonna dig into like how that works. So the Kubernetes documentation covers these API rules for V21, 22. And that's actually in the deprecation guide, which is a really great one. And you can actually see when things are removed and what things were removed at what versions. So if you wanna understand more about it, the docs are a great source of information. Looks like the next set of removals are going to happen in one, two, five. But that is some of the content that we're gonna dig into this week. One of the other things I saw, I mean, of course, one version 1.21 is out. So the next version of Kubernetes is gonna be the one that removes those APIs. So if you want to look into the change log, but difference between what happened in 120 and 121, this is always a good thing I try to spend an episode on. I'm not really not gonna do that in this episode, but right now, I'm just reminding folks, this is actually how you would go about reading the release notes for a given upcoming release. And so if you're looking to adopt version 1.21, this is where you will actually look for information. And it has a lot of really important information. Some of the major themes that are happening, some versions of customized or upgraded from version 203 to version 405, people who have used KubeKettle customized, you can see why that's an important one. The default container, hello. Good to see you, thanks for saying hi, hello. We have immutable secrets and config maps that now graduates to GA, which means the API path will change. Whatever you see graduates to GA, you know that there's going to be an API change, right? That's one of the ways that you can determine that. Structured logging in the KubeLit has changed, generic, fmro volumes, but there's lots of good content here. And one of the other pieces I really wanted to call out was this section of the release notes for every release. And one of the really important bits is this piece here, urgent upgrade notes. And I really like how they actually call this out. I feel like this is repeated every release of Kubernetes know you really must read this before you upgrade. So this tells us information that is important to us if we're going to upgrade from a previous version of Kubernetes to the current released version of Kubernetes. So definitely read this stuff if that's you, or if that's something that you're gonna be working on. So this was just a quick reminder of where to go and find information regarding upgrade paths and release notes for Kubernetes clusters. And so the release notes for 122 haven't been cut yet, but I imagine portions and pieces and portions of them are already in there. And we're kind of getting a little bit of a preview into what that looks like here of the deprecated API migration guide. All right, what is next up in our notes? I did see a new CBE being announced. And the way that I track this stuff is I'm actually a member of the security announce group. If security and Kubernetes is important to you, I recommend that you become a member of the security announce group, right? And if you click on over here, it's a Google group and you can go ahead and join that Google group and you'll be able to get notified when things interesting and security-wise have happened. This particular one is a security issue was discovered within Kubernetes that could enable users to send network traffic to locations that they would otherwise not have access to via a confused deputy attack. It's a low severity. It was assigned CBE 2021, 25740. It gets into the details and I really like how CJA has done this and how I'm sure that many of the members of the security security are responsible on this one as well. It says, if a potential attacker can create or edit endpoints or endpoint slices, and that means that you have the ability to define that object within Kubernetes, then they can potentially direct a load balancer or ingress implementation to expose back end IPs the attacker should not have access to. Importantly, if the target's network policy already trusts the load balancer or ingress implementation, network policy cannot be used to prevent exposure from other names fixes, potentially bypassing any security controls such as load balancer source ranges. This issue is a design flaw that cannot be fully mitigated without user-facing changes. With this public announcement, we can begin the conversations about a long-term fix. So if this, I mean, if you're making use of load balancer and ingress, which you probably are in your production clusters, this is probably something to be aware of and be aware of that those things could actually be changed. So there's information on how to mitigate it. It's basically changing the permissions for that particular object or those objects, the endpoint and the endpoint slice object, how you can detect if anything has actually used this. And there was an additional advisory, a similar attack as possible using the ingress implementations to support forwarding to external name services, which are one of the valid services within Kubernetes. This can be used to forward two services in other namespaces or in some cases, sensitive endpoints within the ingress implementation. So really good update from the, from security. And if you're interested in understanding more about this vulnerability, put this here. And one of the things they always add here, it's kind of at the bottom, oh, it was actually Rob Scott sent on behalf of the Kubernetes product security committee. So awesome stuff. So there's a link to the GitHub issue. If you want to know more about it, or if you want to follow along for the resolution of this particular CBE, definitely check out that issue and you'll be able to see those things. So pretty cool stuff. Lots of good information coming out of Cube Weekly and also out of this week in Kubernetes it's coming right up. And then the next thing I wanted to get into were these two pieces here. So this was the Twitter thread that I was describing earlier. It started with Kat Cosgrove putting out a Twitter thread saying, you need to pay attention to the release notes for Kubernetes version 122. I know nobody likes reading release notes, but you gotta, and in this version, multiple beta APIs will be removed, not deprecated. They've already been deprecated or we wouldn't be removing them. And this is a removal of those APIs from serving. And we're going to play with that during this session. That's one of the things we're going to dig into. But one of the comments to this thread that I got a lot out of was actually from Stefan. Stefan is a core contributor to Kubernetes and he makes a few valid points about how to understand this particular state that we're in when we're going to stop serving a particular API inside of Kubernetes. He says that APIs go away in 1.22 but understand that existing objects created with old APIs are not. And what that means is that if you're in version 1.21, you can still create those objects using those same versions. And those objects will then be inside of your cluster's database, inside of SCD, stored as objects. And when you then upgrade that cluster to version 1.22, that means that those objects are still defined within Kubernetes. And we'll talk about how and why that works when we get to the playtime part of this session. How's it going, Dev? Good to see you. Welcome. So there is no object version of V1, beta one in the cluster concept. Objects are converted on the fly. And that means that there is an abstraction layer in the API server that is responsible for defining that mapping between the API objects that you create and the stored object that is stored inside of SCD. And this is gonna be one of the fun things that we play with today. We're gonna look at the difference there and see how that works. Hey, Pop. Good to see you, Pop. But suffice to say, what that means is that as we change API versions that are made available or unavailable within the Kubernetes API server, those API versions are not generally stored as an object within SCD. It's actually just the core object itself that's stored in SCD. And the abstraction that the Kubernetes API server provides is how we surface those objects either to post when you're going to be creating those objects or whether they are gets or reads when you're actually trying to keep kettle get or update those objects. That's actually how that bit of it's going to work. We're not in a situation necessarily where we store the API version that it was created under. All right. So that's actually pretty interesting stuff. And we're gonna play with this. And I like how it says, this is magic. It's not magic. It's software, you know, it's all right there. When storage versions are removed in a release but Etsy still has objects of the old storage versions, QBPI server fails starting up. That is actually incorrect. You have to touch the objects to convert them for the upgrade. And the Kubo storage version migrator can help. Read the release notes. And there's going to be documentation on how to do that. And then the last thing that Stefan calls out is this piece here, which is where I'm gonna spend the most of the time in playing with this stuff. Because I feel like this article by Evan Cordell, who's somebody I worked with at CoreOS, great guy, really gets into, like really deserves more time, right? Because I feel like this article really gets into some of the internals about how Kubernetes API server works and how those API versions actually are surfaced and manipulated. When you do a kubectl get or a kubectl apply or kubectl, those sorts of things. When you're working with the API server directly. And so my plan today is to kind of get into this. This is going to be the play time. We're going to walk through some of these scenarios. There's actually, I think this blog post does an incredible job. And so if you want to skip out and just read the blog post, that's the thing that you can do. But I really feel like a lot of the points that are made in this blog post are worth kind of exploring live and we're going to play with them. All right. But there was one more thing I had, which is something that I learned. And I wanted to play with you and I wanted to share this with you. It was one of the things on my side that I thought was actually kind of interesting. So I'm working with ISOVALENT, which you already are aware, I'm sure. And I'm working as their field CTO there. And so I'm having a great time at ISOVALENT. And my work computer is a Linux laptop. And this is generally true whenever I'm not working at a place that doesn't allow that. But right now I have a Linux laptop. And one of my goals was to get to a place where I could do better work with like BPF trace and those sorts of things. And the way that BPF does that bit of tooling is that it needs header, not header information, it needs a BTF symbol information so that it can understand how to map events that happen in the Linux kernel with context around what's related to that particular event. The way that it does that is that it actually holds, you could consider it kind of like a dictionary of symbol data. So if you've ever built a debug package for a package in Linux, right? You could create a debug package and that debug package will hold the symbol data for the package that you're going to debug. And that way when you're doing a trace on that application to understand where it's spending its time or where a particular bug has happened or you're digging into a crash, that symbol data will actually allow you to kind of like really kind of in a more human readable way understand what's happening with that particular data. So BTF is very similar. It holds that type data for events so that we can use BPF to map what's happening at the type with at an event layer with what's actually happening inside of the kernel when a particular syscall or a particular event happens as a kernel layer. That was a lot of information and I hope it made sense. But if it didn't, feel free to reach out to me and like let me know that the thing I said doesn't make any sense. Like you can hit me up on Twitter or you can hit me up on the Kubernetes Slack. There's lots of ways to get all to me. But I'm Maui Lyon everywhere. Suffice to say what I was working on, I wanted to get to a place where I could actually do BPF trace and those sorts of things from my own Linux kernel that I have running on my laptop. Now what I noticed was that the Linux kernel that I was operating didn't have access to BTF information or BPF type information. And there's a couple of different ways that I can solve this problem. One of the ways is to build a debug kernel and then to grab, yes, the link to the Evan Cordell is definitely in the blog post. But you know what? I can actually just put the link in the chat, I think. Let's try it. Hopefully you can all see that. Did that work? Also, hi, Diane. Good to see you. So the way this works is that you can build like a debug kernel and that debug kernel will basically build a version of the kernel that gives you access to all of that type data. And then what I found a new config option in the Linux kernel, well, new to me anyway, I'm sure it's not like super new, that gives you the ability to define that you want the kernel to make that BTF data available inside of SysFS, which is really cool. So let's look at that real quick, because this is actually the thing that I kind of learned this week. So begin here. So we move into, so I've got my Linux kernel. I'm using Arch as a Linux as my operating system for my laptop and it seems a little contentious. So I think a lot of people kind of chuckle about the whole like I use Arch kind of thing, but that's not really where I'm coming from. I like that it's actually very flexible I like the packaging system and I like the AUR stuff. But let's dig into how this would work inside of Arch because that was actually the problem that I had to solve. So this is the actual build package for the Linux kernel that is shipped that works and it is most compatible with the version of laptop that I'm using. I'm using a T14S. So it's got an AMD chip, it's a Ryzen chip and I'm using the Zen version two chip, right? So it's the, I can't really like a Cezanne family or something like that. Regardless, this kernel has a lot of the optimizations for the AMD chip that I'm using on my laptop. And what I had to figure out was how to make it so that I could build that BTF information. So let me first, let's just take a look at where that data is actually being made available. So it's in this SysFS underneath kernel, BTF. The setup here, we can see these are actually all of the BTF data that are actually stored for not only my Linux kernel, which is right here, but also for all of the modules that have been compiled for the kernel. So anything I compiled as a module, BTF symbols are also generated and made available here. And when these things are available and when these things are made available inside of the SysFS, that means that BPF trace and other tools that leverage libbpf can just make use of those symbols directly without having to go look anywhere else. They can immediately find the symbols they're looking for and then give you really good contextual information about what's happening on an event level for BPF trace at the kernel layer, which I thought was mind blowing, really, really cool stuff. So if we do it that way, then we don't have to worry about like actually making that BTF symbol information available to some other application. Say I wanted to run BPF trace inside of a container or if I wanted to like run some other application that would need BTF symbol data within a container or something else that was going to share the same Linux kernel as my laptop. So if I were to do Docker, excuse me, Docker run bash ls sys, we can see that we have that same data available inside the internal. And so what this means to me, this is actually the thing that I was playing with on that very beginning was I needed some way, I wanted to play with some tracing information in a Kubernetes cluster, but I wanted to be able to leverage kind to be able to do that. And so to be able to get across that finish line, what I needed, what I was missing was making that BTF symbol data available to my Linux kernel directly and then any containers that are also associated with that same Linux kernel. So anything running inside of a kind cluster or anything, any of the Docker container that I run locally on my laptop will not have access to that same data. And I don't have to make a file available to that to BTF trace. I could actually just build it right into the kernels sys fs, which is so cool. Anyway, that's what I wanted to show you. So the differences that I made between my, the upstream kernel that is available to my laptop and my own are pretty, are not very substantial. The difference was it diff or the master configured. All I had to do was add these three lines and actually only two of them are applying right now. So it's config debug info and config debug info BTF. And this makes my, that makes that sys fs piece work and that makes it so that BTF symbols are available to anything that would actually try and leverage BTF inside of my laptop at any layer as long as it's not a virtualized layer. So really neat stuff. Anyway, that's what I wanted to share with you on that piece of it. And now we can kind of get back into that incredible blog post. So six things, I love the call out here too. This made me chuckle as soon as I saw it. 16 things you didn't know about cube APIs and CRDs. Number 15 will shock you. That's so great. So this is actually, I mean, I think I had known this but I hadn't really thought about it before. But what this says is not all Kubernetes APIs have controllers. So the thing that might be modifying state for a cluster may not be a controller, right? The thing that updates that object may not be an actual controller necessarily. It could be some shared context that is updating status. So let's read this a little bit and see what it says. The QBVI server is largely responsible for the storage of API data into a backing store. And typically that's at CD, as we know. The Q controller runs a series of reconciliation loops. I consider these to be controllers because it's a bunch of controllers that are just really kind of using a shared cache. And so it's a more efficient set of controllers than you might think of for like controllers for CRDs, for example. But in this particular case, because they're using a shared cache, they can kind of get back and forth to that data very quickly. An example of that set of controllers that happen at the controller manager, for example, when I create a Kubernetes, when I create a deployment, if I do a kubectl apply or kubectl create deployment, then that creates a deployment object. And then there's a series of controllers that are responsible, that are being managed by the controller manager that progress that deployment object to a set of running pods, right? And so for example, there is a deployment controller that is responsible for breaking that deployment into a replica set. And there's a replica controller that is responsible for breaking that replica set into a set of pods that have been defined. And then we pass the control or lifecycle of those pods down to the kubelet, or actually no. Once those pod objects are created, then the kubescheduler kicks in, kubescheduler associates those pod objects that have not already been attributed to a node with a node. And then once you have a complete pod object, the pod object exists, it's been scheduled to a node, all of those things are checked off the list, then the kubelet sees that that pod object has been created and is associated with itself, and then goes about managing the pod. Again, in a controller kind of way, right? There's a reconciliation loop that happens on the kubelet that is responsible for ensuring, A, that we know about all of the pods that are associated with this node, B, that we understand the lifecycle, where we are in the lifecycle of each of those pods, and that we're reporting where we are in that status to the Kubernetes API server so that those objects can be updated. And this is, I think, where Evan is going with this, right? Is that when you go ahead and create those things, there are different pieces that are responsible from updating, perhaps, some parts of that object over time, right? So the deployment controller may not be the thing that marks itself as progressed, and that might be the replica set controller. That pods are active or unactive is actually something that's going, or active or not active, or something that's going to be kind of bubbled up from the kubelet to the pods status object, and then to the status object as a controller, right? So what information is reported at the API server when you do kubectl get deployment, whether you have the, whether you've reached the current running state of that object or not, can be updated by different sets of controllers, not by one particular one. So let's take a look at how this works. Clear. I go ahead and create a cluster here. So let's take a look at what he did. So he had, I think it was just doing the single node. Let's bring up our cube cluster. Thank you, Dan, for updating that. I thought I'd put the link in there, but if I didn't, thank you for putting me in there for me. I appreciate it. And thank you, Dan, for the great comment. Makes me happy. Glad to, I'm glad you're here. I'm glad everybody's here. I'm gonna do a kubectl create cluster or create deployment dash in. Let's call it quicken image equals ngenix, port equals 80, like it was equal to. I just created a deployment object. And if I look at that object, events, there's a couple of different things that happen. You know, some of which are, oh, here we go. So we see the deployment object get created and then we see the replica set object get created and then we see the pods get created. So here's the deployment object. Here's the replica set. Replicaset controller, basically creating two pods. Then we have the pods themselves being pulled and created, starting the container ngenix. And this is being reported usually by the node that is reporting it. Pulling the image and creating it. And so what we see there is that like those controllers are responsible for each of those controllers are responsible for that. The command I just ran was kubectl get events. And I was just targeting the namespace. In fact, like deployment, I think you can do, I always forget if there's a way to actually get kubectl events for a specific object. I think there are. I'm used to just looking at, here we go. You know what it is? Describe deployment. So these are the events. This is the event data happening at the deployment object itself. And so this is an event data that is actually represented inside of the object itself. And then if you look at the kubectl events for the namespace, then you'll be able to see even more event data for all of the things associated with that deployment, right? And so what we saw was a deployment be created two minutes and 30 seconds ago. We then saw a replica set telling us about things that have changed, right? So we've got a successful create for the pod. And then we saw the pods being started up. And we're kind of getting a little bit out of order, right? Because we're seeing like two minutes and 29 seconds ago we're seeing the pole image. And then two minutes and 23 seconds ago we saw the creation of the pod. So we're seeing, actually we're not seeing out of order. We're seeing the order of events. So a creation of a pod object is different than showing up on the kubelet and kubelet reporting. Okay, I'm pulling the image so that I can actually start that thing up. All right, so I'm gonna do kubectl describe deployment. Click in. And then we're gonna look through some of the data that is being presented by the deployment object. Right now I'm only looking at the deployment object. I'm not looking at pods. I'm not looking at replica sets. I'm just looking at the thing that I created which is the deployment object. And it's interesting stuff because I have a lot of information that is made available to me here at the deployment object that might have been set or made available to that deployment object through some other controller, right? Or has been bubbled up by some other controller. Replicas for example, right? I did say when I created the deployment that I wanted to replicas but the deployment object is not responsible for creating replicas. It doesn't understand that, it doesn't understand necessarily that those replicas have been created but it can surface the answer to that question, right? So it's surfacing that I requested two replicas that two have been updated, that there are two total and that there are two available. And this information is made available to me via the deployment object when I'm doing kubectl describe deployment I can see that data but this information does not come from the deployment controller. It's bubbled up to the deployment controller from or to that object from the replica set controller which I think is really kind of the important point that Evan was getting to in his blog post. So let's move on to the next one here. So that was the point of it that was pretty interesting. And then he has some other examples, right? So controller manager has a deployment object subject access review is one that actually exists but it's actually the abstraction for that is entirely within the API server. So this would actually just be a function call within the API server when you do a subject access review although we create an object to hold state for that object. The thing that is actually manipulating that object is actually going to be the API server itself because the API server handles the authorization piece of it. So there's no controller necessarily inside the controller manager even or on the kubelet or any of those things that is responsible for managing that subject access review object. When you do a, when you, so when you create one, you're actually only seeing the, we're only capturing the state of that leveraging a function within the API server itself. Kind of interesting stuff. So we return the controller manager off we would be able to see the object get created. So here he's actually pointing out that like that the subject access review gets created. And so there is an object inside of the API server that is known but then there's actually a status, right? Whether it's allowed or not allowed and that status is set by the API server itself. So instead of actually just creating the object and letting some controller handle it, the API server itself is handling that object that status update. That's pretty neat. Now here's another one I think I personally had internalized but I had never really thought about it in these terms. And I think Evan does a great job of making this very approachable and understandable. And that if you've worked with the Kubernetes API at all, one of the things that you learn about is this idea of GVK group version kind, right? And this is gonna be really relevant to those removed or deprecated APIs because group and version are part of what gets deprecated or removed from the API server in these circumstances. But the kind persists, right? We still have an ingress object that you can define in version 1.22. But what's different is the group and version. The path to that object that gets created is different. So we're gonna dig into like how this works in this article. And I think he does a great job of describing it. But the other piece that I think highlights is that you also have group version resource. And these things, these three coordinates identify APIs, right? And so when we think about custom resource definitions, we're defining group version resource and we're defining whether that resource is scoped to a namespace or to a cluster. But this is a resource, it's an API that is extended, right? And then the objects, the representation of objects that are consumed by that API, that's GVK, group version kind. So if I defined a new CRD for pieces, and I might say my group and version are, I don't know, trying to think of some really famous New York, Italian pizzeria, right? As the group, right? So another practical example is it might be like isovalent.com or yourcompany.com or cluster.com, any of those particular groups that you define. And then the version is where in the API versioning scheme am I for this particular API that I've extended, right? So is it an alpha, is it an alpha API? Is it a beta API? And what version or where in the life cycle of that versioning scheme am I? So the very first one, I might call it V1 alpha one. Rouse, there you go. Thank you, dad. I mean, thank you, pop. So the group could be Rouse and the version could be V1. Actually, that's probably V1, right? Cause like these guys are very mature and making pizza. And then the resource object might be pizzas. But when I create a pizza, like say I want a, just an amazing pizza that has like basil and tomato and just like really just like, I want to go for like the OG pizza. I want to go for like, you know, really good, like the really good just like base upon which all things are built pizza, right? Then I might create that object and I have to create an object of kind pizza. And then I can say what the ingredients would be, basil, tomato, the really amazing dough and that's going to be my, that's, I mean, it's some cheese and that's going to be my pizza. And so this is where the GVK part comes in, right? When I define a pizza, leveraging that API made available to me by a resource, then we can actually, let me create the GVK. We create that object and that object is represented by the group version kind. All right, so let's take a look a little bit more into how he defines it here. And the example here saying, let's take a look at a single object, right? And so in this particular case, we're looking at the, an RBAC object that's been defined kind of by default within your cluster. Oh, nope, this is actually one that he created because it's inside the namespace default. So this is a, this is a roll. Interesting, why is that namespace there? Because he created the roll within the namespace. So this is a namespaced roll that has been generated that he generated directly. And you can see that the group is RBAC authorization kx.io and the version is V1, right? So this is the actual object that's been created. And this is how this was created, right? So if I do kubectl create roll and default, we're going to give it a name like pod reader. And then we're going to verb equals get list resource equals. This is actually just using the kubectl command line to go ahead and create that same object that we see in the blog post. And if I do kubectl get roll pod reader, that's OEML, we should be able to see that basically that same output that we found on the left and it gives us the ability to play with this stuff, right? We can actually just create that object directly. Now, this is an object, like I said before, and this is actually one of the things I thought was really distinct about the blog post, right? This is the group, this is the version, and this is the kind. And this is defined here because this is an object that has been created. If you're extending kubernetes to understand a particular CRD, then the group inversion might be something else. But this particular group inversion is made available via the core API. There's nothing in the objects that tells you what the resource is. The kube API server maintains a mapping of resource types to kinds. As do kube clients like kubectl or client go. This is called the rest mapping. In these client projects and related docs, once you know the resource, there are simple rules for building URLs. APIs, group, version, resource, name. APIs, group, version, namespace is a name. If it's a namespace object, if it's a cluster scope object, you'd be able to see it here. If it's a namespace object, you'd be able to see it there. Now, if you want to explore that stuff live, or if you wanna kind of dig into playing with this pathing information, you can actually look at this information directly if you just turn on kubectl proxy, right? So let's just say kubectl proxy. And now I'm serving the entire API server in a kind of a read-only mode at 1-2-7-0-0-1 call it 8001. It's not read-only. You can do all kinds of stuff there, but let's go ahead and pull this up. So in the, let me make this a little bit bigger. So in the blog post, you know what, I'm gonna clean up my browser tabs here. I know people don't generally do that, but I'm a wild person that way. Let's do some stuff. So we see APIs, group, version, resource, name, right? So let's see if we can find the resource associated with RBAC. So I know that this is gonna be under authenticationk8s.io because that's where the RBACs, actually it's gonna be an RBAC. Well, where would it be? That's a great interesting question. So our object is a role. And actually we can see from the header information, it's gonna be RBAC authorization. So if we look inside of here, we can see RBAC authorization. So this gives us that same path that we talked about in the blog post, right? If I wanted to understand for the role, what the path would be, it would be under slash APIs, group, version, resource, name. So my group and version are RBAC authorizationk8s.io and my version is v1. So let's go check out that path and see what we see. So I'm gonna go ahead and grab this URL, copy. And I'm gonna paste it against the RBAC proxy that I see here. Inside of here, this is that rest mapping that Evan is referring to, right? So these are the things that can be created. So this is the resources that have been defined. You can define a cluster role binding and the kind or objects that can be defined for this particular resource is a cluster role binding. The resource for cluster roles and the object or kind that can be defined is cluster role. And these are the things that can be done to that particular object for this particular rest mapping. So when we create this object, these are the things that can be done with it. If we cruise down a little bit further, I think we can see roles, right? And this is an interesting one because roles are defined not at a cluster scope, but at a namesake scope. So in this rest mapping, we can see that this is a role, is represented by a kind or an object called a role, and it is a namespaced object, meaning that it can only be created at a namespace level. I can't create a cluster wide role with this object. There's a different object for that. But since this one's namespaced, it can only be created within a namespace. And that's why when I did my kubectl command to create the object, which I think I made go away, but when I did the command to create the object, I had to specify a namespace in which to put it. And then the verbs associated with this one, it can be, you can do a create, a delete, you can delete a whole bunch of them, you can get them all, you list them, you can patch them or update them, and you can also watch for them. Now, in this rest mapping, we also have storage version. And this is actually kind of an interesting one. Let's go ahead and decode this because I think it's kind of interesting that it's in base 64. I thought that this would actually not be, but go. Whenever I see base 64, I kind of almost always want to just decode it to see what it says. A helpful rest. I wonder if I messed something up here. Oh, storage version hash. I'm silly. Up here at the top. So for the group in the v1, we can see what the preferred version is that we're going to store it as or make it available. Let's go back to our blog post here. So this is actually how we can see it. And there's actually another great doc in the docs that says API concepts document that gives you information about how this stuff works. Thank you for coming by, Diane. See you next time. All right. The next one up is objects are accessible and all API versions, which I think is another really important one. And I think we're probably going to have to hop through these a little quicker because I think we're taking our sweet time about it, but I feel like there are some really succinct points that we should make first, that we should get out of this, right? So objects are accessible at all API versions. And this is actually a really useful one for understanding how to convert objects and how that conversion process works as you handle it upgrade, right? Let's take a look here. So we create an ingress at this newly to be deprecated version, networking.kh.io if you want beta one. And then we do a get or a create of that. We'll see that object get created. And we can see that instead of, and then it's made available to us on the get at this new version. And that means in this particular case, it's calling out the preferred version as what he's referring to here, right? So when we looked at the group version, right? We can see that the preferred version, so anything that when somebody just does a random get of roles, then I'm going to provide them the result as mapped in this way. But if I had a different preferred version, then even though I might have compatible versions, like in this particular case, I support creating, it's supported to be able to create a role using V1, beta one or V1. You can create either one, but when you ask me for the results of that object that was created, that I'm only going to provide that data unless otherwise specified as that particular version. You can find out what the preferred version is by looking at the group. And if we're looking at it here, but he also calls out another really cool command, which is this one, control C, you can get raw APIs. And so in our example, let's just do networking. Okay, it's to IO. So that's the group. We can see the version that is being made up, that is preferred as V1, just like they were talking about before, right? The version of the object stored in SED may differ from the submitted version or and the preferred version. For every group and resource, the Kubernetes API server knows the served versions, the decodable versions, the storage or encodable version and the preferred version. If you submit an object to the API server and it doesn't match the storage version, the API server will convert it to the storage version before storing it in SED to be more precise. At first it converts it to an internal version before converting it back to the storage version, but this is an implementation detail. Now this is where like some deep magic happens at the API server. And I think it's worth understanding that this happens because it's actually pretty neat. Trying to think if we can actually have an example here to walk through it, but maybe he's actually answering that for us. So for example, in cube 120, the preferred version of ingress is V1, but the storage version is V1 beta one. This can be seen by looking at the data the API server stores in SED directly. But he's created the ingress object and he's going ahead and grabbing the content, the data from SED directly, leveraging Auger. And we can see that the API version stored inside of SED is a networking k8s.io slash V1 beta one, or V1 beta one. Now that's a little bit contrary to something we said before, when like the object that is stored, the content that's stored in SED doesn't know about group version and kind. And I want to clear that up a little bit. So I think where the disconnect is happening is that we say that the object doesn't know anything about it. Now, while that's true, it means that if you look at the object that's stored in SED, there is some content that's stored about it, right? So we can actually have a preferred storage version or a storage version that is represented by that storage version. But that storage version isn't necessarily the entire API that is exposed by that extended API itself, right? It's just whatever is necessary to store the object in SED and these can differ. And this is I think where Evan's point is coming in, right? So in 120, for example, we stored only that data that was made available inside of that beta API. But in the newer version, the stable version, the content might have changed. It might have required new fields. And so there's some conversion that has to happen. Now, if we cared, if we get caught up in it here, what that means is that we can easily get into a position where the data that's being presented when you do a kubectl get of an object might be materially different than the way that that object is stored. So let's say I created that pizza that we talked about before. And before I didn't particularly care about what the contents of the dough were or a particular topping. Like I didn't even make it available that you could put pepperoni on a pizza. And so I didn't really have like the topping section built out. And so to make the topping section, to build out the topping section, I needed to change the API. So if I created a version of pizza at V1 beta one that didn't have toppings, and I made that my storage version, then the thing I'm storing in SED wouldn't actually have information about toppings at all. And so even if I were to make a new extended API that had that topping information, I would need to have some way of defaulting or making available the toppings option so that that object could be stored with toppings as we move forward, right? And so the storage version that is represented inside of SED might be materially different than the version of the API that I made available via the CRD. I know that wasn't super easy to follow. Was a little bit confusing, but suffice to say the storage version is usually the preferred version, but it can be overridden. So in 120, for example, the storage version was different than the preferred version. Changing the object stored version requires writing a newer API server or a newer CRD depending on what you're working with here, right? So if we wanted to change the object storage version for something that was built into the API server, like that role object that we talked about before, we would need to change the API server because that's actually where we're defining the API for a particular version of Kubernetes. In this example, we create an object at V1 in Q120 which stores it at V1, beta one. We update the API server to version 1.21 and check in SED that it's still stored as V1, beta one and then we overwrite the object with a new API server to see it stored at V1. So check this out. This is how you might migrate or how you, earlier we talked about, I'm gonna pause this for a moment. Earlier we talked about the very salient point that Stefan pointed out, right? Where in if you create an object at a version that will be removed from API serving, how do we actually convert that object on the backend to make it available so that we don't just like abandon those objects that don't work anymore, right? And this is actually practically how it works. And this is a great example, right? So we have a cluster, its version is serving V120.7 and then they go ahead and create a new ingress object using V1. And we can see because we're still running at Q120 that when we go through the QAPI server version at 120 the object is stored in SED as V1, beta one. So it's that same SED CTL command and we can see that it's stored as V1, beta one. So now with that API server operating at V1.20 this is the only way we're gonna store these objects. Now he's migrating his API server to version 1.21 and we can see that the change has been made. We reconfigure SED using the same storage backend. Now, even though you couldn't, you wouldn't be able to create this object at V1, beta one anymore. You would still be able to see that it's stored that way because this object came from before the cluster was upgraded before the API server was moved to version V1.21. And what this is doing is it's just doing a straight replace of the object. And we can see now that the version that we do a Q, if we do a SED get, we can see that the stored version gets updated. And this kind of happens somewhat organically within Kubernetes, which is actually pretty neat. So it means that anytime you modify that object and the storage version has changed, anytime you modify that object, the object will get stored as the new preferred storage version. So if I created a whole bunch of resources like deployments, for example, at V1, beta one, and I haven't modified that deployment, but my cluster has moved forward in versions, right? So I'm no longer a 116, I moved to 117, I moved to 118, I moved to 119, what have you? The, those objects need to be touched or modified in the cluster before you move to the next version so that we can store those in the new storage version. And this is where that, those release notes that Stefan was talking about before. You have a couple of different options. So you can actually run a keep, got a command that just goes through and does a replace on every object in the SED data store. And it's an intensive thing, but you can even like, you know, stash it, right? You can go buy API paths. Let's do all of RBAC, let's do all of ingress, let's do all of networking, let's do all of, you know, deployments and replicasets and all of those different CRDs that we want to modify. And this gives us the ability to go through and touch all of those objects and get them stored to the new version before we move to the next version of Kubernetes. Now you might have heard like in the upgrade guide or read in the upgrade guide inside of Kubernetes that you want, that you need to ensure that you don't skip any patch versions when upgrading your Kubernetes cluster. So if you're gonna go from 116 to 117 or if your target is from 116 to say like 120, you can't just go from 116 to 120. The steps that are necessary to do that are that you bring up a 117 cluster that might have changed the storage version for those objects stored in your data store. And then you touch all of those objects to make sure that they're now at that new storage version and then you can upgrade to 117, right? So you go 116, 117, you upgrade all of things to the new storage version, 118, upgrade all of things to the new storage version, 119, upgrade to all of things to the new storage version because that's the guarantee, the API guarantee that we have is that we'll get you from old version to new version but not necessarily old version to some forward version. And now you kind of understand materially why that's important. The storage version, the thing that's actually stored in NSE needs to be stored in a way that is compatible both with this version of the API server and potentially the next one. And that's actually how we manage that. So these are the pieces of this blog post that I think really, really apply to API removals and deprecations, right? What we talked about so far is if we're going to remove an API from serving, then that means that the objects that you created with that old API server are still around. They're still hosted inside of the SED cluster. And they may be hosted in SED at some older version of storage version, right? But the thing that's important is that those objects don't just get dropped on the floor. They don't stop existing inside of your Kubernetes cluster. They're still there. They just need to be migrated to a new storage version because you've changed the version that is available via the API server. And that may have actually already taken place, in fact. So like the preferred version is likely forward for all of those objects that we talked about in the blog post, right? So whenever you're creating even the old versions right now, they're being stored as a new version. The conversion is already happening for you. But suffice to say that as you move from 121, the scenario of I had a cluster, it was operating at version 120. I created a bunch of objects against old APIs that were available to me in version 120. And then I upgraded my cluster to 1.21. Are those objects still there? Yes, they're still there. And they're still available. And you can still modify them and touch them and make use of them. But you can't create new objects at that API version anymore. If you tried, you would get an error back saying that that API version is not available. But all of those things that you did create are still present within the API server. If you want quick ways to understand what versions are available, right? So this is actually one of the tricks I wanted to share with you here. One of my favorite commands, I give so many shout outs to this command. It's like seriously a lifesaver, I think. Keep kettle, explain, deployment. But even if your CRD hasn't implemented the explain piece completely, this bit up here will already be made available inside of kubectl explain. So if you wanna know what the current preferred version of an object is, you can actually look in kubectl explain and it will tell you if you're gonna create a deployment, go ahead and create that as group apps version V1. And if you have CRDs defined inside of your cluster, you'll be able to see that as well, right? Because CRDs also extend the API server and they also have to define what their preferred version is. So if you do kubectl explain of some CRD that has been defined, you'll be able to see what the preferred version of that object is there as well. I'm not gonna get into ratcheting validation. There is in 120 plus, there's actually a storage version API. This is something new that I was unaware of. Apparently it's stuck behind a feature gate, which is pretty neat, but apparently it would perhaps show us information. Oh, that's cool. Okay, so with the storage version API, this is a relatively new thing. I don't think it's actually there by default. Internal API server kxio, so let's see. I'm gonna check to see if my cluster has this resource defined. So let's take a look and see if it is, right? So because I'm looking for a resource, the way that I'm gonna determine that resource is available or not, is I'm gonna do kubectl get, actually kubectl API resources. And I'm gonna look for, I'm just gonna grub for internal. And it looks like this particular cluster does not have that feature gate enabled. And the way that he enables that, I kinda wanna play with this one. So let's go ahead and turn this on. I think that looks really cool. Unfortunately, Evan gave us an example of what to do from the kind config to go ahead and enable this, right? So let's go ahead and do that. The kind delete cluster. And if you wanted to play with this stuff, you could play with this stuff in this way as well. I'm gonna make a directory that we're gonna work on here. We're gonna call this twikin5cdtwikin005 and then we're gonna do kind.yaml. And so I know that this is a kind.yaml because of the API version in this, right? So it says kind.jsio v1 alpha four. And then these are parameters I can pass to my kind cluster that configure the cluster in a specific way. And these parameters get passed to QMADM inside of the cluster on startup. So these feature gates, if you wanna enable a feature gate inside of kind, you can just use the feature gate stanza and then enable particular features. And then inside of the runtime config is also enabling this piece of the API server. It's the API piece. So this enables the API on the backend and you can think of this as enabling the API on the front end. Whether we surface that API or make it available, it's our kind of two different pieces here. But with this configuration, we can do kind create cluster. Dash-config kind.yaml. This will create our cluster. And as we saw before, we didn't see any internal version when I did a grep for that inside of API resources. But let's see what happens after this new cluster starts up. We'll see if we actually see it there now. I'd love to play with this API and see kind of like what it exposes. That's actually pretty neat. Again, huge shout out. You'll always hear me say this consistently every single time. Huge shout out to the kind community and the project for making stuff like this, not only possible, but relatively simple, right? Like, I can't imagine being able to do this kind of work without having some way of locally creating a Kubernetes cluster and playing with all the knobs and dials and learning about these environments in a place that is like safe to break and not in your production clusters and also something that's just very quick to start up. Now I do see that API. Okay. So I'm gonna go ahead and do, come back over here real quick and break my cube proxy and start it up again. We're gonna go explore that API just like we did before. We'll do a refresh and internal API server kh.io. So I would think that this would be our path. So our group is, there it is. Found it. I was like, why don't I see that in here? But I see it now. All right. So there's our group and our version, right? So our group is API's internal. Let's go ahead and go to the group first and just take a look at the content on that page. Slash. Here is the content that is made available on the group. So we see the group version is kind of meta. So I can get the group version of group version or storage version. Then we can see that the only version that is available right now is v1, alpha one and the preferred version is also v1, alpha one. No shocker there. It's only the only one available. And if we go and look at the API that is exposed there and we're gonna look at the kind of, oh, I messed up something there. All right, about that. v1, alpha one. These are the resources that have been defined. One of the resources is the kind of storage version. That looks like it can be, it looks like it's the only object that's been, oh, there's a status object and the resource object itself. Both of them are exposed. Likely these are differentiated by the permissions that are associated or the way that you can update them, right? And so the object itself can be created, deleted, delete, collection, get, list, patch, update, watch. And then there's a different permission if you want to apply get, patch or update to the storage version object status. If you think about our back, that makes a little bit more sense. But so if I go one step further and actually see the API that is extended, right? Like what objects, what's the actual framework for that API looks like? So this is the spec, status, the storage versions, things that I can learn about it, right? So this is the encoding version. This is the decodeable version for the admission registration object. Really neat stuff. The relatively new API, it's still behind a feature gate, but it really gives us like some interesting view into how storage version is being managed, right? So in this case, let's see. Apps V1, there are three decodeable versions. Apps V1, Apps V1 beta two, and Apps V1 beta one in my cluster. The encoding version is Apps V1. So if I were to look into SCD for anything stored at the group, Apps V1, then I would understand that it would be stored in this particular version and this particular group. And that if I were to do a get for the one of these things, I could get at any one of these other decodeable versions, right? So for example, let's just play with that real quick and I could show you exactly what that means. So if I do kubectl get deployments, well, kubectl create deployment twicken image equals nginx replicas to port equals 8080 deployment twicken dash OEML, this is the object that was created for me, right? It said deployment is stored in sd1 beta one, but can I get the older versions of that, right? So if I do to understand what versions might be available, one of the other commands you can do is API versions. These are all the versions that are available. So that might not work, but let's try it anyway. So kubectl get deployment. Yeah, I want to create, let's see, kubectl API resources. There we go. So ingress exists in both. Kind of like the example that Evan pointed out before, right? Ingress exists in extensions V1 beta one and exists in networking V1. So if I do this, do kubectl create ingress tests. I actually tried creating an ingress this way. Expose deployment test twicken, and then I can do kubectl ingress twicken.com default equals in suite. All right. So kubectl get ingress twicken. OEML. We can see that right now it's giving me the result as this new version, networking.ksio v1, but this is the fun part. So I'm using kubectl bash completion, right? So if I do kubectl get, I can see the things that I can get. I want to look at ingress, the ingress object. And if I look here, there's a few different versions that are available to me, which is really pretty neat. So what this means is these are the decodable versions available within the ingress object. I can actually get the old version. This is .extensions dash OEML. And it will on the fly convert this object according to the API that was presented when this API was available, right? So I can actually get, I can convert this object in this way. I can do kubectl get ingress.extensions, which tells it that I want to see extensions v1, beta one for that particular group, this particular ingress object. And I can also look at the other one, right? So I had ingresses and I also had, oh, there it is. So there's only the two storage versions right now. Ingresses.networking and that's the v1 and ingresses.extensions and that's the older one. And then in a relatively recent change, I can't remember exactly when this was introduced, but I think it was in like 1.20 or maybe 119. If you start interacting with an API that has been deprecated, even if it's just a get, you're going to get a warning. It tells you, hey, heads up. This is a thing you probably want to know about. You created this object using extensions v1, beta one, that's a deprecated API. So we're doing as much of you can in the community to make this possible that you won't get caught out by this. All right, I think we've covered quite a lot and I want to go back to our notes here and see where we are. Well, that's a pretty new API. The cube storage version migrator can handle migrating changes. So we can just take a look at how this works. So like I said, if you're going to do like an upgrade from version 116 and you're aiming for 120, how do you get there? One way to do it is by leveraging tools like this, right? Where we see the old version is still stored at the old version, but we want to change that. So we have this tool that gives us the ability to modify the stored version. And it goes through basically and touches everything that needs to be migrated. Actually, it touches everything. It doesn't care whether it needs to be migrated. We can see what the migrator did. It touched a bunch of different stuff. If you think about it, this is like database migration. This is almost effectively what it's doing. Once that's all done, we should see the ingress object be changed. And there it is. All right, CRDs define new APIs, new resources, just like we talked about. All APIs are cluster-scoped, even-scoped namespace CRDs, meaning that the API itself is expressed as a cluster operation. Now, this is an interesting point. I'm not gonna harp on it too long, but when we think about multi-tenancy, this becomes a very important point because when you define a new API, you make that available, you make that API, you define that API at the cluster level, not at the namespace level. And so it means that you have to think about the permissions piece of it a little bit. A CRD has a stored version, just like any other API. This field indicates every version that has been a stored version, all of them. You can't remove a version from a CRD until it's been removed from status. Oh, that's neat. So this is more details about how CRDs themselves are managed. Anyway, so that's what I wanted to cover today. I do recommend reading this blog post. There's some really great stuff, and it really gets into the detail. And I think it really highlights some ways of understanding and playing with how these things could work and how you can explore this stuff, leveraging kind and explore this stuff in a non-prod environment where you can learn all about the way that storage versions and all of these pieces are actually managed over time. And you can take away, you can work in a really safe environment without having to worry about learning this stuff on the fly in production environments, which I also really highly recommend. I hope you enjoyed this episode. I enjoyed doing it. I'll be back in two weeks, and I will see you then. And if you have any questions or if you ever want to chat about anything or anything else like that, you can reach out to me on Twitter or out on Kubernetes Slack. I am at MauiLion everywhere. Thank you so much for tuning in. And I'll see you next time. Thank you. And then graphics.