 My name is Eddie Zaneski. I'm joined by Macha Shulik and Katrina Viri. And Katrina just ran here from her previous talk. So check out that recording. It was sold out. So we all work at a bunch of different companies, but we all work on Kubernetes together, which is kind of the beauty of open source. OK. And what we do, so basically we are responsible for development and standardization and maintenance of everything CLI related. And that also includes making sure that we maintain stability of kubectl, especially if you have kubectl running in your CI-CD pipelines or anywhere in the production, you care about the stability most importantly. So we put a lot of effort not to break you and at any point in time. So the projects that we maintain, that will be kubectl. So I'm pretty sure that you all already heard about kubectl. If you have not, please check it out. It's pretty cool. There are a couple more projects like Customize. Katrina is doing a ton of great work. If you haven't tried Customize, please try it. It is amazingly easy to be used and it's super customizable. Katrina is such an amazing person so much you can learn from her. I'm doing that every single day. And I'm very green in Customize. Crew and crew index, if you work with kubectl and you've noticed that it's missing something, we open kubectl to write plugins. But Ahmed was so brave enough and figure out that we also need something like a plug-in manager. So he wrote crew. We also have libraries. I did mention that to write good CLIs, kubectl, compatible CLIs, we expose libraries. So CLI runtime, CLI experimental. Although the other one is we try a lot of different experiments. So be brave and have lots of time because we break everything in experimental. KRM functions, that related with Customize, lots of funny client-side functions. Please check it out if you have a little bit of time. Yeah, you can find us on the CIG CLI Slack in Kubernetes Slack. And we have a mailing list. And we have some other contact information in our GitHub repo there. I should have started. Are people familiar how the Kubernetes project is organized in terms of CIGs? No. So CIGs are special interest groups. So Kubernetes has a very large code base that we've carved out into different groups that own different parts of the code base. And so we work on the CLI tooling for that. So the projects match they went through. And it's all open source. And you can join us at our meetings, which are right here. So if you'd like to get involved with any of the stuff we're working on, you have questions, want to talk through things, join us for our meetings. There was online, on Zoom, open to the public. And we also do a bug scrub every other week, one for Kube Control and then one for Customize. So that's a great chance to come, get involved, kind of hear how we talk through issues. We say no to a knot a lot. And it's not because we're jerks. It's just that Kube Control is a very old code base with lots of people depending on it. So making breaking changes is very difficult for us to do with. Everyone's, go ahead. And if you have time next Wednesday, that's the bug scrub Wednesday this time. Yes. Anyone seen the XKCD comic? I should have put it in here, where it's like, I fixed a bug where holding down space power would cause the CPU to overheat and turn off. And then there's a user complaint that says, you broke my workflow. The control key is hard to reach for me. So I've programmed an Emacs macro to count a increased temperature as the controlled key. Please, please add a flag back. So that's the kind of stuff we deal with. We have some updates to go through from our recent releases. OK, I hope I won't break anything. There's something that I broke in a minute. So some time ago, we figured out that people add more and more containers to their pots. And whenever you're trying to reach the locks, it's super irritating that you have to specify the container name every single time. Especially if you have multiple of them, you have to remember what's the name of the containers. Of course, KubeCuttle will tell you nicely that, oh, there are this many containers. These are the names. But you still have to remember that. If you're trying to debug in production, you're under stress, there's no time to, oh, which one was it? And especially if they are similar, that is problematic. There's an annotation that you can put on a pot that allows you to pick the default container that works for KubeCuttle locks, port forward, exec, and anything else that picks a specific container within a pot. And if it's not working, let us know. We have to fix it because that's an important thing. KubeCuttle events. So get is cool because it allows you to read anything in a cluster. The problem with that is get is generic. It does not allow you to be specific about certain resources. So we figured out that events are rather, ah, they allow you to get a lot of information. So it will be handy if we allow a little bit more information filtering sorting around the events, especially if you know the KubeCuttle. Under the KubeCuttle describe, we will usually show you the events related with a particular resource. So we would like to be able for a user to have a little bit more control around that. Initially, we played with the idea of expanding get just for events, but then it was like, yeah, but that means that we will turn get into a fronkin monster that will behave differently for a different command. So we eventually settled on the events command. And one of the fun things about events is that when people were doing KubeCuttle get events watch, they would complain that they would see duplicated events. And what was actually happening is that you would see the deletion event for the record, and then they would see another deletion event like 10 minutes later. And that was actually the event getting deleted, which creates its own deletion event, right? So there's a lot of things that are involved with the events, which is why we built a whole sub command for it. I'm not gonna go through the highlights from 126, but big shout outs to Mark. He's doing an amazing job for the past. Good couple releases with everything related with completions for KubeCuttle. He's put a ton and ton of hours, so big shout out to him. Big shout out to Brian. Brian has been a core contributor to KubeCuttle for numerous releases. He's leading a lot of efforts within KubeCuttle. Big shout out to Shyam for his work. If you wanna know more details, the slides will be available in the schedule afterwards so you can go read about those details a little bit more. A couple more, that was too fast. A couple more shout outs to Marley, to Alex for server side changes, for hiding everything that is sensitive because we were purposely showing a little bit too much. And we are also adding new commands and we're slowly building the scope of what KubeCuttle has to offer. Yeah, and that auth to MI command is really cool if you haven't used it yet. The plan to expand it is really good but just knowing who you're making a request as when you're doing a lot with OIDC and other identity, really cool command. 127 just got cut a few weeks ago so we had a couple commands here. Sub-resources have landed in beta now so if you haven't used the dash dash sub-resource flag, forget you can now get a sub-resource on a pod, right? So Kubernetes API has the different resources that you can walk down. So pods also have a sub-resource of, what's a good example, of status. So you can get pod slash status and so now you can just do KubeCuttle get pods dash dash sub-resource status to get more about that. I think scale is a sub-resource or resources so you can kind of do a lot more with scripting and filtering there. Aggregated discovery is a big one for us. Discovery is that phase where your client will have to go figure out what APIs are available on your cluster and so it has to walk basically the entire open API tree so it makes hundreds of requests. If you ever do KubeCuttle any command like get pods or something dash v10 you'll see just all these requests go by where it would cache these and so what we ran into especially with our friends at crossplane for I had a bunch of CRDs for all their resources is we would hit the QPS limit and exhaust it and then it would get rate limited and throttled. So there's been a lot of work done by some great folks to make discovery a lot better so instead of caching the entire discovery we can now cache individual resources so it should clear up discovery and speed up that first KubeCuttle get command for a while. So that's really cool. Open API v3 was shipped for, when did that actually hit like 124 or five? Roughly around that. Yeah so now we have support for it and explain. It's beta, open API v3 has a lot more metadata and other cool stuff you can do there. Apply set is a new command which we're gonna talk about in a little bit with Katrina and then we want to create sub plugins so for QubeCuttle create those work with generators and so we have to kind of like have a hard coded generator that says when you wanna create a cluster role binding or something it has to know about that and so we want users to be able to create plugins for create so you can create your own resources for maybe your controllers or CRDs. So check those out. And then change log highlights. We're doing a lot of work with QubeCuttle debug, some debugging profiles has been added. Customize, so customize is actually bundled in QubeCTL so you can do QubeCTL apply dash K. Is QubeCTL customize the command too? Yeah so you can access customize directly from QubeCTL a lot of work goes into upgrading that and especially the initial one which we talked about a few years ago. There's a lot of internal libraries and state there. We promoted the who am I plug in so it's no longer alpha for 127. I didn't know that, that's great actually. And then different prune. Did you wanna say anything about different prune? There's a lot there. Yeah there was just a particular bug that we fixed in the. Okay yeah there was a bug so. It was a pretty bad one. 128, 128 hasn't started yet. I checked the dates, I'm pretty sure about that one. The release cycle has it's open. The release cycle, yes. So master is open but we're talking about Hensman's mode. Sorry to fuss. So we already know and are aware that these are the stuff that we want to work on 128 if you have any particular requests, demands or you wanna fix something. Let us know ahead of time. The cycle hasn't started. The sooner you get your caps discussed, reviewed and approved, the better for you. It's very hard to get cap approvals in the last week where we have to go through multiple of them and it's not fun. We would like to push the kubrc so that's a configuration file which will allow you to tweak, mix and match and whatever you wanna, magical things you wanna do with kubectl and we have a couple proof of concepts. Marley had one, I had one, Brian had one. We have different ideas. Some of them are differing opinions about how things should be implemented if you wanna help with that. Let us know, there's an issue. Have a look at the POCs if you wanna work on it. That's even better. Yeah and we'll have lots of time for Q and A where we can talk through some of these if you wanna learn more. And with that we'll hand over to Katrina to talk about apply sets. Thanks, yeah so apply set is a new feature that we introduced in alpha four in 127 so we wanted to take the time to go over it. Not very tall. So the history of this feature is that it's basically new iteration on a feature that's existed since 1.5 so really, really old. It's qctl apply dash dash prune. And that was added to allow people to declaratively manage the deletion of the resources as well so they could have a set of resources under management and say in git and apply those and when they delete something from their configuration system and apply again, the resources automatically be cleaned up and have to go and manually go and find it otherwise have leaked stuff that they did not actually want in their cluster. So that is a common need and the basic approach that was taken to implement this was an allow list. So you provide a list of types that you want to sort of manage via a flag. The actual selection of which resources to delete though was a bit more complicated than that. This selection would be based not only on the allow list or its defaults that would be used if you didn't provide that flag but also on the namespace flag and visited namespaces. What visited means is you have your configuration set. Those resources, a bunch of them probably belong to namespaces so we'd keep track of which namespaces we had seen in your set and we would also consider resources and those namespaces for deletion. Optionally you could also add label selectors as a flag to constrain the set that we would consider and the annotation that apply, client side apply, important actually client side apply uses this annotation and we would use that as a filter to sort of judge whether or not the resource belongs to apply so we would never delete anything that was missing this. Unfortunately this approach had a number of pretty bad problems and that's the reason why it's still in alpha even though we're at version 127 at this point. The biggest one is correctness. It would do the wrong thing sometimes and when we're talking about deletion, that's very bad. So there are a couple different ways to trigger this problem. One is by having a mismatch in the group version kind allow list that you're providing. So that is a hard-coded list whether you're using the default or whether you're having it in your command line invocation and it is not found in any way to what you're actually deploying either in the previous set or in the current set. It could be a completely different list of types. So if there's a mismatch there you're not gonna clean up what you expect. And then it can also happen with the namespace because as I was saying we add in the namespaces that we've seen but say you have a resource set that has two namespaces in it at the first time and then it has one namespace in it at the second time. Well, what you wanted was all the stuff from the second namespace to be cleaned up but now this invocation doesn't even see that namespace. It's not in the set anymore so it just forgets about it which is not what you wanted. Another problem is with scalability because of this approach of sort of querying a large number of GVKs some of which might not even be relevant. It is very expensive to do and if you try to mitigate the correctness issues by composing a list of every type under the sun under your cluster and making Q pedal visit all of them just to be sure it makes the performance issue even worse. And the invocation even worse. Oh yeah, and the invocation becomes a very huge line at that point. Yeah, another problem with this approach was as I mentioned it is relying on a detail of client side apply for its selection which means that if you switch over to server side apply any new objects that you create using that strategy won't be considered because they're not, don't bear that invitation. And there's also a bunch of user experience issues where strictly speaking it's behaving as intended but it's not really intuitive or not really what the user wanted. So a lot of these come from unintended changes related to the flags that you have to provide all those flags that I went through at the beginning. So if you change the flag set from one invocation to another you're re-scoping your command and chances are something strange might happen because you had a different scope in the one command and the second scope in the other and they're not actually related because what we're operating on here is really trying to deduce the previous set. You can also inadvertently over select resources in a couple of ways. One is that you could accidentally use overlapping label selectors from two different deploy pipelines or just overlap the scope that you specify in any of the ways and say some operator comes in and manually creates a resource with Q pedal apply. Well, that's gonna have the apply annotation. It might be in scope of one of your commands. Did you want that cleaned up or not? It's not clear because it's not actually there's no explicit set involved. We're always trying to sort of guess so you can get unexpected results. It's also difficult to use with custom resources. The built-in list will never include any but you can configure just like any other resource but then you have to manage that list, right? You have to add all of your custom resources to every invocation of the command line which is a pain and easy to get wrong. So the history of this was that we saw this problem. We get issues about it all the time that we couldn't fix because they're pretty fundamental to how it was designed. And the first step to try to do something about this was an experiment in the CLI Utils repo that we mentioned back in 2019. And that was successful. It was a good proof of concept of what an alternative design could look like but it couldn't be brought directly back into Qt Control for technical reasons. So it was used by other projects in the ecosystem so that was great. But we still had like the stagnation of the built-in approach. So at the KubeCon North America 2022 Contributor Summit Sean Sullivan and I led a session on how do we move forward with this? This is a big problem for our users and we were happily able to reach consensus that yes, we do need to do something about this. We can't leave this permanent alpha in here for 20 plus releases and just have it be continuing to cause problems for folks. So we're gonna go with a new independent implementation instead of trying to somehow fix all those problems that were very fundamental. And we're gonna do it based on a low-level standard that other tools can adopt. So we're also doing sort of an ecosystem play here. We can help the tools that need the sort of functionality work better together. And after that, myself and Justin, who is in the room today, hi Justin, worked on a KEP for version 127 and we did get the MVP features in to that release so you can go try it. So how does the new feature work? We went over the details of the old one. What are we doing different now? Now we have a concept called apply set. That's really the central key concept that we're working from here to make this better. Apply set is a group of resources that are applied to a cluster by tool. So it's what you're passing to Qt control in our case, that set of resources that you're trying to manage. Group membership is tracked both in and by reference to what we're calling a parent object and that parent object is persisted to the cluster. That state is managed by Qt control itself. You don't have to deal with it. The parent object can be a secret or it can figment by default but you can also configure your cluster in a way that I'll show in a moment to allow certain custom resource types who used as parent objects as well if you have an existing aggregate object, for example. A given object can only be part of one apply set which is a constraint that is new to this implementation and we did do that in some cases where we felt that although we wanna accommodate as much of the original use case as possible we also really wanna make sure that this time is a lot more intuitive and safe to use. So this is an example of one of the constraints that we added. So the specification is really based on metadata and you can go to the Kepp for full details on this but this is just gonna be kind of like a whirlwind tour of what it is. There's a bunch of labels and annotations that go either on the parent object or on the member object. So the parent and the most important one is that gets labeled with an ID that we generate and that is what makes it a parent object. It also accepts a series of annotations that help it interrupt better with other tooling and have appropriate scoping and the ones at the bottom there are basically hints to help it know where to look for its children because we aren't actually inventorying the full children as the source of truth here. We are, okay, yeah. We are annotating the child with an annotational show in a moment and that is the source of truth of its set membership. Like it says I'm part of this set. So we need to go find it and make sure that it is in fact part of that one set before we consider it. Yeah, all part of that. So we still need to know where to go look for it and the last two annotations are hints to keep control to go find it and we can reconstruct them so that we can have a way to correct the set if ever something got messed up in your cluster or manual changes perhaps intervened with the set that you were trying to manage. Those last two are under discussion and might change in 128, particularly the one I have grayed out a section because we're considering between resource and kind based on our experience in the alpha. So as I mentioned, the actual source of truth from set membership is a label on the object. So we can do a label query to find all the children and you can also configure CRD types to be allowed as parents by just labeling, sorry, CR instances. You label the parent CRD that defines the type and that allows the instances of the type to be used as apply sets. So the important part, how do you use it? Since this is an alpha you're going to have to set an environment variable to use it to just keep control of price that equals true. And then you use the prune flag as usual but you add this new flag instead of all those other ones. You just put apply set and you choose a name for your apply set. There are docs on this that you can find at the link. Let's go into more detail. And one thing to mention though, of course, this is an alpha. Please give it a try. Give us your feedback, especially if you're using the old implementation let us know if this works for you. But it is an alpha so be careful about using in production as with any alpha. And that's all. So we have some time for Q and A, I believe. Yeah, and a quick shout out. There's a SIG meet and greet happening immediately after this. It's 12.30 to 2.30. It's downstairs, I believe, in the Euro four or one. All the Kubernetes SIGs will be there. So you can talk to different folks at different parts of the code base and find out how you can get involved too. So definitely make sure to stop by. Grab some lunch and come see us, so yeah. Any questions, things you wanna complain about, talk through, feature requests? Yes. Are you planning to add any ways so I will be able to control the caching better except for aggregated, for example, how long do I want the object that the discovery have seen to stay in the cache and stuff like that? I'd be curious to hear you use cases for the caching side of things. Theoretically, we would prefer users do not manually manage the caches because it usually causes problems. You can always invalidate the cache by invoking API resources or API versions. That invocations removes the file from the disk and retrieves the entire discovery at once. The problem that we were going with with the majority of the discovery was to ensure that especially if you're trying to use kubectl in a pod, the initial discussion, the initial discussion, sorry. Yeah, well, it's a discussion with the cluster. The initial request, as I mentioned, the more CRDs you will have in a cluster, the bigger the initial query it is and it is very frequently causing issues for a lot of the users. So that was the main reason why we wanted to squash the amount of stuff that you have to read initially to speed up the first request, especially when you're running or creating a pod to do something simple. And especially if those are short-lived pods which only do one, two, three tasks and they are done. And then again, we will bring new one and you're wasting, I don't know, up to 30 seconds to a minute, I have customers that complain that they're really waiting to two minutes, to three minutes because the entire discovery was taking this long. So we intentionally, because there was a time where the cache was valid for roughly 10 or 15 minutes. So we intentionally pushed that limit over to six hours for the majority of use cases that we identified, that will be perfect and it should not cost issues for the majority of the users. That's why I would prefer to hear use cases, why we would jump to do that. You can always do, like I said, manually by invoking API or manually just deleting cube.cube.cache directory. You can also set the cache directory to DevNol and that will cache every single request or fetch every single request. So I had one question which calls back to your previous talk, Katrina, about the KRM functions. I've seen that you have a bunch of Kips for composition and catalog and stuff, which all look amazing. I'm just curious if you have some sort of status update on that whole thing. There isn't mostly because we don't have enough folks to work on them. If you want to use that feature, if you think it's great, then we could use your help with those. Those Kips are all implementable, I believe. It's just that the folks who were originally working on it moved on to other opportunities and we just don't have anyone to implement them right now. The catalog one in particular is the one that's most likely to get attention anytime soon. That one is very important to the sort of long-term plugin strategy for Customize. So we might work on that one somewhat this year, but composition itself, we're more focused on, if anything, some of the features that are required to use that in Customize we had an implementation, but we don't want to merge it until we're sure that folks need it and that all the pieces are in place for it. So yeah, I would suggest if you're interested in any one of those, catalog is the one that has the most traction at the moment, but yeah, let us know if you're interested in working on it because we need more folks. We got like seven, eight minutes left. Any other questions? Things you want to talk through, features? Hi, you mentioned that Kip CTL code base is relatively old and thus it's, you don't want to make breaking changes. Are there any CLIs code base that would be a bit more approachable for beginners and folks who have less experience writing both go and CLIs? Customize. Customize? Okay. I mean, Customize is one thing, but if you think about Kip CTL as a whole, yes, it's a big code base. But the important benefit of getting involved with 6CLI and Kip Cuddle itself is that you can literally limit yourself to a particular command. Please do not use get. It seems simple, it is not. But there are multiple other commands where in the worst case, you'll end up in a single directory with two or three files that you have to work. In the best case, it could be literally just one file. There are some commands that are missing tests, like simple unit tests. And that is a perfect opportunity for someone to start. There's at least a couple of ongoing rewrites that we are trying to slowly push because there was a bunch of commands and a bunch of approaches that we took over the years, how Kip Cuddle commands should look like. Eventually we settle with one that is easy to maintain, easy to read, and easy to test, most importantly. And we're slowly going through the commands and rewriting a lot of them. There has been multiple presentations that I did in the past, Eddie did, Katrina did, about how the flow of developing with any Kip Cuddle commands works. So if you have some time, just check out one of the previous recordings from 6CLI from KubeCons. We've done that a couple of times. Nothing has changed since then, and it's still up to date, or ping us during the box crop, or during one of the meetings, or just on Slack mailing list. And we can point you to the issues that are describing because we literally, what we did is in the issue we described, the weight command is literally the best example of how a perfect KubeCuddle command should look like. Slowly, all the new commands that we are adding are required to follow the same patterns, and events literally follows the same pattern. What was the other new addition? Who am I in the auth as well? And those are rather simple commands, and we will be promoting events to stable because I realized when preparing this presentation that we overslept. And it should have been promoted to GA in 127, but we'll catch up in 128. Yeah, but yeah, the code base is old and there are lots of dragons, but there's also lots of approachable pieces. A good chunk of the code, so I don't know if a lot of people know this, but Kubernetes internally doesn't talk YAML. Everyone complains about YAML with Kubernetes, but Kubernetes only understands JSON internally. And so a lot of the code that we traverse in our journey is going from JSON to YAML to JSON to YAML, and then back to your client, and then in reverse when you give us YAML. So that's where a lot of the dragons lie, and my favorite one is the cube control config command, where you can edit parts and get parts of your cube config. That was all done through insane go reflection, where it looks up the original struct definition for go and interpolates the JSON metadata tags that are there to validate what data you're giving it. So it was written a long time ago when people were learning go, and then it's a use of heavy reflection in the code base that we'd like to replace one day. And Marley did great work there too. Yeah, and we still have unstructured, don't forget about that one, which is a map of maps, of maps which represents a JSON. If you're talking with a dynamic client, you're getting unstructured. We basically have to take that JSON and that unstructured struct that comes in and then figure out what go struct we can marshal it into. And that is several layers of abstraction and fun to understand. You don't have to touch that as a beginner. Yeah, but Eddie, don't forget about them. Don't forget about them. I think they are freaked out enough. Any other questions or thoughts? Nope. Cool. Well, thank you all for joining us. We really appreciate people coming to the maintainer session. If you could please review it so they let us keep doing these. It's awesome. And be sure to come by the SIG meet in Greek to meet some of the other SIGs and enjoy the rest of the week in beautiful Amsterdam. So, thanks everyone.