 Okay, thank you everyone for joining us today. Welcome to today's CNCF Live webinar, Extending Cube CTL with Plugins and Crew. I'm Libby Schultz and I'll be moderating today's webinar. I'm gonna read our code of conduct and then hand over to James Sturdemont, Principal SDE at Microsoft. A few housekeeping items before we get started. During the webinar, you are not able to speak as an attendee, but you are welcome to pop any questions into the chat box on the right-hand side of your screen. Please feel free to post everything there and we will get to as many questions as we can at the end. This is an official webinar of the CNCF and as such is subject to the CNCF code of conduct. Please do not add anything to the chat or questions that would be in violation of that code of conduct and please basically be respectful of all of your fellow participants and presenters. Please also note that recordings and slides will be posted later today to the CNCF online programs page at community.cncf.io under online programs. They are also available via your registration link that you use today and also on our online programs YouTube playlist on the CNCF channel. With that, I will hand things over to James to kick off today's presentation. Take it away, James. Thank you. Hi, welcome everyone. I'm James Sturdemont and we're gonna be talking about extending Qubectl. We'll talk a little bit about the plugins and crew. Oh, James, I think you muted yourself. Oops, sorry. No problem, there we go. And when I was trying to move the slides. All right, you're all good. Cool. So the agenda today is to go through Qubectl plugins, just an introduction to what they are. And then we'll talk a little bit about crew, which is kind of the way to distribute your plugins. And then we're gonna dive into how do we develop one and publishing them. So we'll go through all that and ask questions as we go along and we'll either be able to answer them as we go or at the end of the session. So I'm an engineer on the Azure open source team. We do a lot of work in Kubernetes and all of the related six. I'm also a lead for SickWindows. And I guess the reason I'm here today is I created a Windows debug crew plugin recently and showed it to some folks and they thought it was cool and wanted to know how to develop their own plugins. And so hopefully can lay that groundwork for you. And then related to my Twitter handle down in the bottom there is I can create fire six different ways with using only sticks and stones. So if you're interested in that maybe you can reach out to me on Twitter and we can chat about a little bit about what that means. So Cube CTL, I'm gonna take a little bit of an assumption here that folks are pretty familiar with Cube CTL. It's the command line tool for interacting with your cluster, your Kubernetes cluster. It allows you to look at all the resources. It allows you to inspect logs. You can do cool interesting things like port forwarding. And here I'm just showing what the command looks like. So Cube CTL get pods and we're just listing all the pods out. This is probably a fairly familiar tool for a lot of folks. And I think this is something that you probably use quite a bit. The thing that you may not know is that you can do a lot more with it that doesn't come right out of the box. And so here I'm showing another tool. There's a link to it down at the bottom here but this actually lists all of the RBAC rules for a given type of resource and prints it out in this nice fancy list shows you exactly who has access and what kind of access they do have. And this isn't built into Cube CTL out of the box but you can add it as a plugin. And another cool example of this is a plugin called SNF and you can point it at a particular pod in your cluster and it will take care of wiring up all of the different components. It will run the tools to be able to look at all the network traffic and then it'll even launch Wireshark for you and stream the live traffic into Wireshark. So this is a really cool extension for using Cube CTL. And when I saw this, I was like, this is awesome. I need to be able to create something that's as cool as this. So hopefully by the end of this you'll have an idea of what that looks like. So what is a Cube CTL plugin? This is from the Cube CTL documentation. If you think of Cube CTL as the command tool for like the essential building blocks of interacting with the Kubernetes cluster, the plugins are ways of interacting with those building blocks in the Kubernetes cluster in a much more complex way. As you just saw, we were able to connect to a pod, stream the live traffic running on that pod to Wireshark. That's a much more complex scenario than Cube CTL is set out to solve but we can integrate directly into it and make that part of our development workflow. And so plugins would essentially extend the Cube CTL with new sub commands which allow for new and custom features that aren't part of the initial Cube CTL distribution. And so I pulled that from the docs. There's quite a bit of a really interesting well written docs out there that you should go check out if you are interested in this type of thing. Is anybody out there, maybe you can drop it in the chat here, but is anybody out there using any interesting plugins that they'd like to share with the team with everybody here? Cube CTL is free. Okay, I don't know about that one but I'll have to look it up afterwards, it'd be interesting. So what makes up a Cube CTL plugin? It is literally very simple. It's a standalone binary that begins with Cube CTL. And so here's a couple examples. The one that I developed, I named it Cube CTL Windows Debug. There's another tool called Neat. And so it's just Cube CTL Neat. Once you have the binary named that, you move it onto your path and it can be anywhere on your path that you want it to be as long as it's accessible. And then it can be written in any language. So it doesn't have to be written in Go, it could be written in Bash, it could be written in Rust. It just depends on what you feel comfortable writing that plugin. Once it's on your path, you can start to use it. So it's pretty simple mechanism here to be able to extend Cube CTL, but very powerful as you saw. So you can install a plugin by just taking the binary and copying it to anywhere on your path. It doesn't have to be this particular location here. But it does have to, as I mentioned, have to start with that Cube CTL component there. Once it's there, you can list all of the various plugins that are on your system that Cube CTL is aware of. It will iterate through your path and list anything that starts with Cube CTL. And so here, let's say this is compatible plugins. And then once it's on your path, you can use it. So this one is a little extension I wrote for cluster API for Azure. It's a little bash script that lets me connect to any node over SSH uses the SSH key that I have on my tool. And when I was developing, I was having a lot of trouble connecting to the nodes, we do a lot of the infrastructure. So the nodes wouldn't necessarily come on line because I messed something up when I was developing. And so, but SSH was usually available. And so knowing what the IP address of the node was and all of those types of things was just a lot of work. And so now I can just say Cube CTL Capsi SH and the name of the Capsi CRD and I was connected to the node. So I kind of simplified my workflow by writing a little bash script and then dropping it into the binary. And so this is a great way and hopefully can inspire you to kind of simplify your workflow in some way. And so I just saw a question from somebody in the chat there. So does there a central place, platform or marketplace for third party plugins? That's actually my next topic here. So obviously if you have it in your repository it's maybe available to your developers but being able to share it out to the wider world that's what crew is trying to solve. So you can think of it as the app get or NPM for Cube CTL. And there's currently over 200 plus plugins on there. You can find the Windows debug one if you're a Windows developer out there. And, but there's just so many out there. The ones that I demoed earlier in the slides are also out there. I would assume Cube CTL tree is available through crew. If it's not, we should look at trying to have that there. It helps the entire life cycle of the plugin. So it manages installing, removing, updating. You can search for all the plugins right through the command line. And the plugins don't necessarily need to start with Cube CTL in this case. So if you develop some binary, I think actually access matrix is not, doesn't start with Cube CTL. It's a different binary name. And crew will manage that for you. So they copy it to a local folder with that Cube CTL specification that's needed to become a plugin. And then it also helps with the distribution. So you can push it up to this chart here and it helps through the command line tools to be able to pull all that down. So it looks a little slightly different when you're interacting with crew. Before we saw our, you just kind of to install it, you would copy the binary into your, onto a path in, sorry, into a folder on your path. And with Cube CTL, we can search for anything we want by any kind of command. It does a fuzzy search and comes back. So here I searched for Windows. You'll see the Windows debug. It gives you node access, tells you a little bit about it and then tells you whether or not it's installed. If I want to install it, I then run Cube CTL crew install Windows debug. And it will manage pulling it down, selecting the proper platform. So whether I'm on Linux or Windows or Mac or whatever it is, it'll get those all installed in the right place and then tells you how to use it. And so really nice interface for distribution and consumption of these, all right. So what I want to do is just quickly go through kind of using some of these tools and then once we're done that just so you can get a feel for them, we'll move over and start talking about how to develop them. So cool. So the first thing I wanted to show here is that first tool that I'd sent. So I haven't already installed the plugin. You can see I've got it, I'm connected to a cluster. This is my cluster that I'm connected to up here. And you can see what I'm running is just access matrix resource config map. And so this binary QCTL and folks it and it goes out and does a bunch of mapping across all of the RBAC rules and CRDs out there. And then comes back and says, here's who has access and what they can do with those tools. So really, really cool way to get more information about your system. The next thing is it interacts just like any other QCTL tool. So there's, when I do dash H, I get the usage. I get various examples. You can see I can specify a different context that I want to use. And so a lot of these plugins are built to just use the same tooling that you're familiar with through QCTL. Excuse me, I'm gonna go back here. There we go. So the other thing I wanted to show is that QCTL sniff. So I have a pod out there and I'm just gonna run this on there. What this does is it will wire itself up to the node and then it launches fire fire. Sorry, it launches wire shock for me. And you can see that I'm actually streaming the live requests from that pod right now. So 10224.30 is my HTTP bin binary. So getting that wired up and getting the traffic streamed to me would have been a fairly challenging situation but with the plugin it becomes very trivial. So I'm gonna stop that there. And so if I look back at the pods here you'll see that this is indeed it was coming from that pod. It's not from just my local traffic or anything. It was being streamed over the top. So we can look at the plugins that I have. So here we can see I've got some crew plugins that we were just looking at. Here's an example of one that I've installed locally. I don't have this. This is something that you can install from the repository of cluster API for Azure but it's not something that I've deployed into crew. So it's just local. These other components here are they're just kind of some additional things because I have some extra things wired up because I'm running on WSL. So to install a tool I can, I'm in my windows debug tool right now. And so I can copy this component here to my system. Now if I do the plugin list I should see that added there. And so now I can do queue CPL windows debug and this will pick a random node to run it on. I can also specify the node I want and I get dropped on to the actual node here. This is powered through host process containers and we're in beta right now. And so there's this long queue here but this will eventually go away. We're planning on adding support for host process containers in 126 and it's gonna go stable and you're gonna see similar features here but there's cool things that we're doing with that type of stuff. So when I'm using crew I can also use crew to install these plugins and I can kind of mix and match here but if I use crew you see I get a little bit nicer UI. It tells me what plugin is, what version I'm on and then I also have the ability to do things like I'm searching. So here I haven't installed windows debug as a crew plugin and so it's not showing it all this install but I could then go ahead and install it as a plugin and it would work. So now it's installed and I can use it just the way I just showed you there. So that's kind of just a quick demo of kind of how these work and how you get started with them and next we'll move on to developing plugins again. Let me just switch back here. All right, I think you can see my slides again. Cool, so developing plugins. I think the first thing you need to kind of address is how do you want to develop it? Do you want to develop the plugin directly as a Qubectl plugin or do you want to move towards crew? And I think there's a little bit of trade-off here. I think developing a direct Qubectl plugin can be really quick. It can be a simple bash script that you prefix the name of the binary with Qubectl. That's what I did with the cluster API for Azure script. It's something that I had a little script on my side and just wanted to integrate it into Qubectl. It kind of just eases me, my developer workflow. You can keep it right in your repo so you don't have to go through publishing and maintaining versions and all those types of things and it's super simple to install. So those are kind of like the pros of that. I guess if you look at crew, crew provides that distribution platform. It also manages multiple platforms. My bash script doesn't necessarily work or no, like I have to tell different commands to install it on different systems. With crew, it's just the single crew install plugin. It also makes the upgrades easy. So I can just say upgrade and it'll go through and upgrade all the plugins. It has a wider audience because it's out there as a marketplace. And then it does have some install metrics. And then one other thing I didn't list here that I would call out is that crew also has the ability to host your own marketplace. So the one that I was showing and demonstrating from is from the public marketplace. But if you were in an enterprise and you wanted to give access to some of your plugins to the developers, you could host your own. So that way you knew where those plugins were coming from and you could manage them from that perspective. The next thing when you start to talk about developing these is what language would you start in? The bash is just one of those really easy ones to get started with. There's no build process. And so you can kind of put together a script really quickly. You can write them in any other language but the other really well supported language is Go. It comes with the cross-platform. So it'll work on all the platforms that your users might be on. You get access to the CLI runtime. You can also get access to the Go library for Kubernetes. It helps, gives you the printing logic out of the box. It can set up configuration. And one other big one that it provides is some clouds require some additional support for authorization. And that comes out of the box with the Go plugins. And so when you're choosing, it probably depends on, are you going really simple or you wanna make this distributed out to the wider ecosystem? I chose to go with crew and I think the rest of the presentation is gonna be kind of focusing in on crew. I think developing a plugin in your local repository is pretty straightforward. Developing for crew requires a few extra components. And some of the things that need to be in that crew package at the very least is a license file. A read me on how it works. And then any of the binaries that you're going to be copying. And so you can put this as a tar file or a zip file and host it somewhere and that becomes your crew package. That crew will go and grab and then drop onto the nodes. So at the very least, those are the three things that you need to have. The next part is the crew manifest. This doesn't go inside the package. It goes actually in part of the crew index, which is where everything is queried from. And so this manifest is just a mammal. You add information like the metadata for the name, the version, the homepage, and then what platforms. So this is how crew manages the platforms that it unpacks for. So you can specify, you match this particular platform in this architecture. And this is where you go and get the package for that particular architecture and the rest of it crew will handle and deploy. The crew does provide kind of a templated language, which you can see here. I'm using add Yuri and Shah. So it will go out and drop all those binaries in there. It can also drop in the tag name. And so this is kind of a template that you use when you're actually going and publishing your manifest. There are some starter templates out there. There's an unofficial one called crew plugin template and it will give you that package, it sets up go for you. It has all of the components installed. It has the pipelines for you to enable to get going. So you can essentially fork this and then add your own logic and you can be published and running as a crew plugin pretty quickly. The other one that's really useful is the Kubernetes sample CLI plugin. So this is a example go lang plugin that you can use to get started for the plugins. This doesn't have all of the templating around the crew side of things. So you can kind of mix and match here to learn how to get started. So you've developed your plugin, you've got your package and you've got your crew manifest template. Now you want to automate this so you don't have to worry about it. And so here's a snippet from my Windows debug. Get help pipeline and here I'm showing the dead simple steps to be able to publish one of these things. There's more complicated ways to do this but essentially all I need to have is the tar file with the license, the read me and the binary that I'm interested in. And so I've run some tests and then I tar that up and then I can release the package as a particular file. And then I call this crew release box. So this is developed by the crew maintainers. And since 99% of changes to your plugin are gonna be just version bumps, they've automated the entire like deployment and updating. And so this release box will look at that template file that we just reviewed the crew template file, fill it in with all the relevant information and then open up a PR into the crew index. And this is an example of what that PR looks like. It is, so somebody ran the crew release box it opened up a PR in the crew index. We're bumping the version to 1.7 and the maintainer has another bot who's verifies that it was just the version bump and then merges that PR. And then you can now get the update through crew update, whatever plugin you're working with. So automation is pretty straightforward and simple. Highly recommend it, makes it really easy to get out new versions without doing too much work. I mentioned that you can get a little bit more complicated than just tearing up some files. And so the crew plugin and the template that I mentioned earlier uses Go Releaser, which is another tool which helps automate building and packaging the projects. And so again, you can provide a Go Releaser template file and then during that build step, you reference it and it will pump out the packages for all the different OSS arches and label them all appropriately. And then you can use that queue Releaser to publish the entire package out. There's a few things that I learned when I was going through this process. I showed you the Kube CTL Windows debug and it's three separate commands. The way that Kube CTL plugins work is if you put a dash between the commands, they become separate commands in your system. And with crew, it ends up, they convert those dashes to underscores. And so when I install Windows debug without crew, I can say Kube CTL Windows debug and then the node name. Here, if I install it with crew, it's Kube CTL Windows dash debug. I didn't find this out until after I'd already published it and had some folks already using it. And so I didn't wanna have to change the UX here. So it's just something to keep into mind when you're developing these that they're gonna do that to underscore conversion. The crew index bot is, or the crew release bot was really super helpful. I spent a lot of time getting that working initially and it submitted the first PR and then my PR got immediately closed. It didn't get merged like we saw in the previous slide. And what I found out was that you need to actually generate that manifest by hand the first time. The crew release bot has an extra component that or an extra flag that will print it to standard out. So you can pipe that into a file and then just submit that to the plugin manually, the plugin index manually. The reason this is done is the crew maintainers want to make sure that your description's good. They went through a whole process with me and said they don't like, this should be shorter, change this wording here, that type of thing that they can't do through those bots since you don't own the PR itself. And so another just useful thing to be aware of. And then finally, you can test these locally. Using the crew bot release, you can generate that manifest locally. And then Qubectl has a download package in use archive function. So I can point it at a particular archive URL somewhere and it'll unpack that for me. It won't go out to the crew index and so I can overload that and do some testing of the crew release of the binary that I was working on. And so yeah, just a few items there to be aware of. So I think that was most of the content that I had prepared. And so I just wanted to call out the docs for Qubectl and plugins in general, the crew IO, the crew developer guide. And then if you were looking for a really simple example to get started with, my Windows debug crew solution there is it's a one-liner or 10 lines of bash scripts and a bunch of pipelines to automate deploying it to crew. And so I hope this gives you a quick overview of how to get started with plugins and develop them and I hope you are able to get started and share your awesome plugins. Okay, thanks so much, James. Anyone else have questions? Let's go ahead and pop them into the chat and we can work through any more of those. Give everybody a minute to see if they type it in. Sure. Any feedback to you if you had any questions about anything else to talk about? Sure, or any links that anyone needs to any of the pieces of the presentation. Nobody has a question. All right, well, James, was there anything else you wanted to cover? I think that was it. I hope for a few more questions, but we can... Well, if no one else has any questions, I think you know where to find James if you think of one post-event or are curious in any other Cube CTL information. But other than that, if no one else has a question, we can wrap things up and let everybody let everybody go. Going once, going twice, okay. All right, well, thank you, James, so much for your presentation and for answering everyone's questions. Thank you, everyone, for attending another CNCF Live webinar. Be sure to catch us for our live stream show tomorrow at 9 a.m. Pacific. You can also register through cncf.io. And thank you, everyone, for joining us. So we'll see you all again soon. Thank you, James. Yeah, thank you. We'll talk to you soon.