 OK, and then we'll do filters, and that looks all right, too. Let's go down. Oh, it's just on there yet. All right, about to do a stream. I got to go pee before I start going to start in a minute and a half. Hey, hey. Oh, dang it. All right, I am muted, see, because you told me it was a hot mic, so I was trying to address that problem. All right, well, welcome to episode number seven of This Week in Cloud Native, where I'm going to explore some security stuff. And it looks like I'm getting lost in the fog a little bit, so it might be something I just have to tune real quick. Give me just one moment to get that sorted out. It's always something, you know? I use OBS for this streaming stuff, and sometimes it works really well, and sometimes it's kind of a pain to get knocked in just right. But generally speaking, it works pretty well. All right, there we go. So this week, I'm going to be digging into some security stuff. It should be a fun episode. I'll be digging into a few different pieces of it. Made it just in time. Yeah, it's been a crazy day. It's been a super busy day. One of the reasons it's been a super busy day is because last week, or this week, is going to be the EBPF Summit. And so that should be, oh, I just realized, see? That's what I'm talking about. Bunk, there we go. So this week is the EBPF Summit. So Wednesday and Thursday are going to be the EBPF Summit. Let me go ahead and just share with you a little bit of information about that. So here is the wrong screen. So let's see if we can figure that out. Just bear with me for just a moment while I figure this out. For some reason, my screen is showing the wrong screen. ZZScreen01. All right, there we go. Transition. And I'm back, and there we go. Now it's all working again. Like it's actually supposed to work. All right, here we go. So EBPF Summit, this is what I wanted to show you. If you have not been there yet, go here and come be a part of it. It's a free registration. It's two days of incredible things. That will be an EBPF-based CTF, which ought to be super fun for folks. There'll be lots of really great stuff. So yeah, come check it out. It's a free event, and it'll be streaming live on YouTube. And you can interact with us on Slack. The EBPF Summit should be amazing. That's kind of what I've been focused on. And clearly, there's a lot going on there. So it's been, like I said, kind of a hectic week. But that's what that's all about this week in the news. Let's take a look. So I always start off with a reminder that this is a COC official livestream of the CNCF. So be nice to each other in the chat, and be nice to everybody, really just generally as a rule if you can do that. Registration for KubeCon Cloud Native North America is 2021. It's now open for in-person and virtual to explore the registration oppens. You can follow this link. One of the exciting things is that the schedule is live. And so if you go to that link, you can actually follow the program piece of it. And you can actually see those events that have been put in here, including this event, which I am really looking forward to doing. We were actually able to get a talk in. The hacker crew of Sig Honk were able to get a talk in for exploring a volume vulnerability that is related to CVE 2021 30465. I'll probably do an episode of this show on that at some point, but I have not done that yet. So it's going to be the four of us. Ian, Brad, myself, and Roy, we're going to be doing a walkthrough of what that's about and how it works and how we went about the research for it and that kind of thing. So that should be pretty fun. But all of the other schedule is up as well. So definitely go check that out. Right. So I think I've mentioned this before, but if I haven't, then if you want to catch up with episodes that have already happened, this is a great way to do that. If you go to that link, you'll be able to see the playlist for each channel or for each thing that we've got managed here, including one that just was updated earlier today, Search Magic by SIAM. And we have Solid State. We have Kudal doing ed.u. We have Maddie doing CNCF face-off. And then there are a number of other shows as well. All really great stuff. Cognitive LatinX. Fields Tested sounded like it was a complete kick. And so if you haven't had a chance to check out Fields Tested, this most recent episode, probably the one to jump in on. Because it was a Caslin going through the SecureKubernetes.com CTF challenge that Tabatha Sable and Brad and a bunch of other amazing folks worked on as the most recent in-person KubeCon, which was San Diego, gosh, forever ago. We got this week in cloud native. That's the channel that you're watching now. So as soon as this episode is over, it will be posted on that playlist 100 days with a nice cognitive classroom. And all of these things are just up and ready for you. So if you miss an episode or if you aren't able to catch a thing that you wanted to see, you can always just go to YouTube. And they're all archived there. There's new content every day. And there's always something kind of fun happening. So definitely check it out. A quick reminder again that in version 1.22, which is actually going to be the version of Kubernetes that we're going to be playing with live today, there have been API removals, which means that some things that worked before are not working anymore. So if you have already read this blog post, please go do it. It's a very important one. It describes just those things that will be removed from version 1.22. So for example, I think likely the stuff that's really going to catch people out, I think we've already kind of been aware that Ingress has been updated for some time. So that shouldn't be messing people up too much. But I imagine this one might be a surprise. So custom resource definition changing and having the old API from the beginning of time go away, that might catch people out. One of the big benefits is that in the 1.20 time frame, we also inserted a warning. So now whenever you're registering a web hook or leveraging any of these APIs that are about to become removed, you should get a deprecation warning telling you, hey, that is deprecated. Now I spent the last two episodes kind of covering API deprecation and removal. If you want to know more about that, definitely go check out those videos. But a reminder again, if you're playing with Kubernetes and you are unaware of that, you should become aware of that before you move to 1.22. And then back on the blog side of things, I just wanted to share with you a few other things. You can always just go to the blog by blog.kh.io. Easy to remember, it takes you there. There's been lots of really great posts just this month, and I wanted to cover a few of them. And mainly it's part of the release cadence of a release like this one that actually talks where you will see a number of blog posts kind of referring to different features or capabilities that are changing. And so the 1.22 release came out, and you can definitely check this one out. There's lots of really big changes in 1.22, which I think are going to be really great. And we're going to look at a couple of them that I'm particularly interested in. We have server-side apply moves to GA, CSI Windows support, which means the custom storage interface of being able to actually create new, oh, cool, it deeps in there, to be able to create new volumes and things as a plug-in, or it will be in GA in 1.22. And then we have new in version 1.22, the alpha support for using swap memory. I know Alana has done a ton of work on this, trying to get this in. But basically, this means that you will no longer see the Kubelet crash when you're bringing up on a system with swap and that Kubelet will be smart enough to use that swap reasonably, which I think will be great. This is a pretty significant change in 1.22. I think since the beginning of the project or something very close to that, we've always told people to remove swap capability, like to turn off that swap volume. But now we don't have to do that anymore. I think a good thing. Nope. Well, that was fun. Live. All right, here we go. So that is changing. We also have the Kubernetes memory manager moving to beta. Talks about the way that memory is managed in the subcomponent. I'm actually curious about this one a little bit, and then we'll dig into the fun work. So some Kubernetes workload run on nodes with non-uniform memory access. Now, if you're not aware of what Numa is, this is a pretty fascinating thing. The idea, they probably get into it. Do they? The idea is that basically the path between the CPU core where your process is being executed and the memory that that CPU core has access to will be direct from one CPU core to a particular bank of memory, but not direct to the other bank of memory. And so depending on what you're trying to do, you can actually incur a cost when you're trying to actually access memory that's in a non-uniform location. And so what this does is it tries, I mean, this is going to be for some really specific work, like trying to make sure that you're operating with as little latency as possible, but with Numa you can actually, you can kind of fence that stuff off so that your CPU and memory are the quickest and that you can ensure that those workloads that you're deploying are going to stay aligned on the same Numa node or locality as possible. So you can really drive down that latency, the amount of latency, the amount of latency between your application execution time and the memory that it's accessing. So I know that was a lot to cover, but like it is a pretty interesting thing and that's actually moving along pretty well. So that's exciting. Moving to beta means that it's already been through an alpha period. All right. What else do we have up for today? So I did not see anything in the security announce group, but it never hurts to check again. Nope, looks like the latest, most recent update was in July. So still looking pretty good there. I like to cover those things as they surface. CNCF things, another big announcement from us available this week actually is the EVPF Foundation. That's going to be a foundation focused on the EVPF and like getting the word out, helping people kind of adopt it, figuring out what we can do together as a group. And now that with a foundation, we can actually do more of that outreach and work. I think it'll be really great. So Facebook, Google, Isovalent, the company I work for, Microsoft, Netflix are all in it together and they are announcing, kind of co-announcing the EVPF Foundation. It's a great blog written by Thomas Graff. And one of the things I really liked about this blog was this comic series, kind of like XKCD style, describes like why EVPF, like why it's an interesting thing, like why folks actually want to see it adopted and what it can be used for. And I think the story is very compelling, basically like, you have some new kernel feature that you want, or you're doing application tracing and you want to be able to see what's actually happening with your application. There are a couple of different ways to solve that problem. There might be a kernel feature that does it, right? And so you might like write that kernel feature to enable some particular form of tracing, or you might write a kernel module, but both of these two instances take a bunch of time to actually get something merged into the kernel and you also might take a bunch of time to get something merged into the distribution of Linux that you're using. And if we like, expose an EVPF like interface into the Linux kernel, then you could write that software as long as it's within the constraints of the EVPF ecosystem, you could write that and deliver that much faster. So pretty exciting stuff, kind of enabling us to extend the kernel more dynamically and with a much shorter timeframe than what it would normally take. So I just realized that you may not have been able to see anything that was just presenting. Yeah, but I'm back now. So here's the EVPF Summit page. How about a rough start today? All right, so, oh, you didn't see it. Just not me. Okay, well, that's good. All right, well, okay, I mean, good enough. I'm excited that that happened. All right, so let's move on. You probably saw me catch my green screen. Oh, no, you didn't because you didn't see me. Ha ha, my green screen fell over and I had to go save it. Now, the stuff that I wanted to dig in, oh, CNCF things, let's do that first. So we talked about EVPF foundation. One of the other places I look for weekly news is at Cube Weekly. This is actually a great newsletter put together by my fellow CNCF ambassadors. And so if you're looking for a good source of news, this is probably the best one that you'll find or at least a pretty consistent one that you'll find. There's also the Kubernetes podcast that does a pretty good job of capturing the news. There's also this weekend, or LWKD.info, another one of my favorites. So LWKD is focused more kind of like on the developer side of things. So like what commits are interesting, sometimes Josh will pick a particular commit and talk about it. But yeah, I mean, like you see stuff like this, so I introduce event clocks based on Util's clock. The API server has a new clock in town. The new event clock API provides more testable approach to delaying calls. Also be sure to check out the follow-up PR which improves some interface and struct names. So that's, I guess, the commit of the week. Stuff that's been deprecated, stuff that's been removed, all kinds of interesting stuff in here. So this is where I go for like development news and then where I'm going for like community-based news. It's more like Cube Weekly, lots of interesting stuff happening there. So it's always some kind of fun article or some interesting technical thing happening inside of Cube Weekly. So definitely check it out. The ones I called out today in the news section were the ones that were coming up for the CNCF. So they were managed thousands of K8s application and with minimal efforts using Cube Carrier. I haven't explored this one. It looks pretty interesting to me. It's hosted by Cooper Matic, and if you wanted to check that out, that's happening on the 19th, which is coming right up. And then also on the 19th, Meshry will be talking about, or Lee Calcote from layer five will be talking about the service mesh manager. And then Sean McCord and Andrew Reinhart from Talus Systems will be talking about hybrid Kubernetes clusters with WireGuard. And all of those things are happening on the 19th. So if any of those are interesting to you, definitely check it out. Quality has improved again. Oh, interesting. I don't know why that would happen. So let me take a look at the chat here. Yeah, we missed the real action. It's true. Oh, it looks okay to me. Weird, I wonder if it has to be like refreshed or something. All right, so let's play with this stuff. So there were the two things that I wanted to kind of play with today. I'm not sure we'll have time to get to both of them, but we're going to give it a try. One thing I know, one thing I wanted to play with was this new change in 122, which is behind a feature gate. We're going to use kind to kind of explore that. There it is. I like how it says feet. Your Kubernetes server must be running. Here we go. Okay, that's a bug. I think I'm going to, this is confusing to me and I'll tell you what I'm seeing and then you can tell me if it's confusing to you. But what this says in the docs is restricted containers, sys calls with set comp, feature state, 119 stable. I'm like, yeah. And then I go down here and it says, before you begin, you must be running version 1.22. And I'm like, wait. And then down below a little further to enable the use of runtime default. And this is the thing I wanted to test as a default set comp profile for all workloads. It's behind a feature gate called set comp default, which is configured at the KubeLit. And you can tell it which particular thing to use by default. I know that I'm using the word default a lot, but the way out of the box, we'll put it that way. What ends up happening is that the KubeLit uses unconfined as the default set comp profile. And I'm gonna show you why that's interesting and why that's like brought not super secure. And then we're also gonna play with this new model where we can actually define at the KubeLit level what the actual default should be. And we'll talk about that and we'll talk about Docker runtime defaults and we'll talk about container runtime defaults in general, those sorts of things. So that's what we're gonna dig into in this episode. And I think that'll probably take us right up to the hour, but I wanted to kind of play with that. I wanna let you know what we're gonna dig into and then we're gonna play with it. And then the other one, which I might save for another episode, but I'd like to cover it. There's an incredible cap that's actually moving forward. It's in alpha now. So it's already got an alpha, it's already got a feature gate. And this is a replacement for pod security policies. And so this one is, I blame DNS, nice. This is my motto really. So thank you for that Russ. So this is a replacement for pod security policies. And it greatly simplifies the model that pod security policies follow to interact with things. And I wanted to play with this because I wanted to play with like how it works and like what it looks like and all of that good stuff. I haven't actually started this one up locally yet. And so I think I might come back and do this another time. But it's already in the 122 code base as a behind the feature flag. And I thought we might explore that probably in the next episode, just kind of play with it. Because I have spent a big amount of time like helping people understand pod security policies. Yeah, save yourself there a little bit. But I'm gonna be digging into this one probably in the next episode or perhaps the episode after. But yeah, I think that'll be probably the right way to go. But we're gonna talk about why pod security policies are good too. So let's get started. Let's go ahead and build our kind cluster with this like nothing special going on. And then I want to show you kind of like what you can detect and what you can determine from like those system calls that are available and that sort of stuff. So let's dig into it. There's one more thing I wanted to grab here. We'll do it in a minute. Kind, create cluster, config, change that definition just a little bit. Next I'm gonna go to the kind project which is kind.sigs. And the way kind, one of the nice features of kind is that has the ability to, you can define inside of your configuration manifest feature gates, which make it really simple to turn stuff like this on, right? So I'm gonna say feature gates and then back over here again to the sky. And this guy says the feature gate is, so we're gonna have to do two things there. It appears. So this will be kind of a fun exercise because so we're gonna do up default. Oh, interesting. Okay. Well, we may only have to do the one. Well, let's see. So the way I'm reading this is that saying that it's a feature flag that you determine, you can set the feature flags that come to fault and this enables the use of runtime default as the default second profile for all workloads. And the second profile specified in the security context of a pod or a container. Now, if we go back into the docs on the other side of things a little bit here, the runtime default is going to be whatever the default second profile built into your runtime is. So container D has a runtime default. Docker has a runtime default. Both of these runtime defaults actually limit the system calls that a given process that has been containerized can run. That's so weird. I don't know why it's blurry at all. I mean, what's weird is also like in the read, in the readback to me, I'm not seeing the blurriness. I only see y'all telling me it's blurry. Weird. Okay. Anyway. So I do like high dev doff. I get the highest dev doff. I'm kidding. Yeah. I mean, like I'm streaming at the full resolution, 1080p 30, we'll figure it out. Anyway, back to this because it's a complex topic. So I want to speak slowly and I want to make sure that you understand what I'm talking about as we get into it. Actually, I'm not even streaming from OBS. I'm streaming basically, I mean, I am streaming from OBS but it's coming out of virtual cam interface that is going directly as a camera into restream. We're using restream for this. So maybe restream is having a problem but I guess we'll figure it out. So yeah, let's go back to our hack and B. So this is what we're going to talk about. I am going to, you know, for some now, I'm going to go ahead and delete this one for now. I'm going to take you through this set of questions real quick. So these are the, this, this is basically what we're going to dig into here first. And then we're going to play with the new feature gate sets runtime default. But to give you a little background, Set Comp is a way of providing a kind of a rule set for processes that have been generated inside of a container that can limit or otherwise constrain those processes to specific system calls that are reasonable for those processes so that, so that you can't do things like, I don't know, reboot the node or, you know, delete things or do you think you can't see anything? I'm thinking this is a Twitch thing, very weird. I'm thinking maybe there's like a Twitch thing or something. So yeah, so let's get into it. Let's keep moving here. So I'm going to show you a little bit about Set Comp and what Set Comp's about and then we're going to play with this stuff. So first let's just go ahead and run like using Docker. You can do, you can do this on your own system as well. What is the command for that? You know, always forget it, run, unconfined. This page, by the way, is amazing. So if you're ever, if you want to learn more about Set Comp, this is a great way to do it. Reading the Docker docs for Set Comp are amazing. So there is an option where you can tell it security ops Set Comp equals unconfined. That's the profile. That's what I'm looking for. So let's grab that over here and then we'll go over here and we'll do Docker, run, actually just paste it in. What this command is going to do is it's going to start up a little bash shell, bash, bash, and inside of that bash shell we can look at the capabilities that we have running as root inside of this container. And then we're going to run the same command without the security ops fags and see what the runtime default capabilities are. And then we can kind of compare the two of them, right? So if I do APK at lib cap, I do caps print. I'm going to go ahead and capture this and we'll head back over here to our HackMD. Now remember, you can actually go to this HackMD yourself directly and help me edit it, put notes in, that kind of stuff. That is available to you right here at the bottom where it says hackmd.io slash at TWICN. If you go to that URL, you're going to find a path to 007, which should take you directly to the notes that I'm using today. So if you want to see these notes yourself, you can hit them up right there. So that would be hackmd.io slash at TWICN. And then the seventh episode is the one that we are on. So you should be able to click right in here and see that episode. Yeah, I'm not sure what the problem is and I'm not sure how to address it because it seems like things are okay on my side. Oh, it says network connection 810. Well, I could drop and come right back. I suppose, well, but let me try that. Let me try just doing, let me just try doing this first. All right, I'm back. And according to the connection, it says that it's high quality now. So hopefully we will no longer see any of this blurriness. I'm hoping that takes care of it. We'll see how it goes. Let me know if you see it some more. Maybe I'll try that again after. All right, so these are the capabilities that we have in an unconfined mode, which are pretty significant. And it's mainly because there's not very many limiting calls here, right? So basically you can do pretty much anything. CapNet RAW is there, quite a lot of capabilities in an unconfined mode. So now let's look at what that looks like in a confined mode and then we can compare the two and see what it looks like, okay? Let's jump in over here. Let's exit out of this. Now all I'm doing is getting rid of this setcomp here. And this up again. And then we'll do apk add libcap, part of the Linux capabilities package. That's what the cap here in this case is capabilities. And then we can do caps print. You can even put a spelling right. And then we can grab that outcome and we'll drop that over here on the HackMD. And we can see that our bounding set. Oh, interesting. Oh, you know what? I'm doing the wrong command here. Maybe that's what's happening. Curl, okay, add curl, okay, it's not work. Am I contained? Those are the capabilities that have been granted me, but what I am not seeing in that outcome or in that output is what capabilities are being filtered from me. And the neat thing is, that's what I'm looking for. All right, so this is the runtime defaults. And I'm gonna put this back over here where we can see it in the HackMD. What I wasn't doing is negative testing, right? And that's basically the problem. What am I contained does is it actually goes through, like for each system call that it could make, it tries to make the call to see if those things are filtered or not filtered. And in a filtering state, which is what we see here for container runtime in Docker, there are 60 block system calls, right? And that is a good thing. And we're gonna talk about, if you wanna know more about why that's a good thing, definitely check out the SecComp page inside of the Docker documentation because they really get into like exactly why this is the way it's done this way. And I think it's a really good article on like what SecComp is and why it's important, okay? So these are filtered messages, right? So I can't do things like mess with NFS exports. And I can't do things that would otherwise, that would otherwise mess me up. Things like set domain name, it's a little more subtle than why, but like there's a bunch here that are actually filtered out. Now, if I go back and I pull up our unconfined one, and I do apk add curl, and I grab curl minus LO, just like we did before, h.work slash, am I contained? h.work is my domain for this stuff. And then I could do change mode plus x. Now we can see the list of block system calls is significantly smaller. So in this case, in this state, we can see SecComp has been disabled, but there are still some system calls that are blocked. Swap on, swap off or blocked, reboot, set host name. Some things have been blocked, but not nearly as much as what's been blocked here in the lower end. And what these are, these are basically system calls that have been picked that are likely to affect the overall performance of the system rather than, and be outside of what a normal process might do, right? And so in this baseline SecComp profiling, likely you've probably never even noticed that it was there because this set of system calls that has been limited here really are just there to keep you within the bounds of a regular process, right? It's trying to be as restrictive as it can without being so restrictive that you ever notice that SecComp is actually in place. And at the same time, like, get rid of stuff, like, you know, giving you the ability to insert new modules or to remove modules or all, you know, things like this that you probably shouldn't necessarily be able to do as just a running application inside of a container on a Kubernetes node. They want to limit that stuff. And that's why there are 60 block system calls here when you're using the Docker defaults. And there's only 20 block system calls if you have disables.com. I have never noticed. See, isn't that weird? Wow, it is weird. It is blurry again. Well, I've tried. What is this? Net-er-cert-common-na-me-valid? Name invalid? Are you throwing TLS error? Is that me, Dan? Dave? Oh man, I did that, Russ. You should watch the episode where I was the attacker where I was actually sick honk. Got the credit for the attack, but I wrote a thing where it was so brutal. I like, I did a bunch of the work on the environment and what I had done was I had written it so that when the SSH didn't, he would actually drop into a scenario where he was in a container that was in a read-only file system and he had to realize that SSH was listening on two different ports and SSH over to the other port so that he could escape that kind of trip. All right, let's move on. So yeah, so these are the kind of things that are filtered and why they're filtered. And if you wanna know more about that, like I said, definitely check out the docker.com documents. In fact, I should just put a link here inside of the HackMD so it doesn't get lost. So the next question we had was like, what capabilities do I have in an unconfined mode versus like what are Docker's defaults? But we haven't looked yet at what containers these defaults are. So let's look at that next. And for that, I'm actually gonna do this in easy way for myself. I'm just gonna go ahead and start up a kind cluster because I know that kind runs container D under the covers. I'm create cluster config before. We'll get this guy started up here. Yeah, kind is pretty awesome. I agree completely. Yeah, plus 1000. Yeah, you could even, I mean, and we might try this, but like anything you can do with QBADM, you can tune with kind. In fact, one of the interesting side projects of kind, while this is booting up, why don't we just go look at that real quick. So one of the interesting side projects of kind is that you can do a thing where it's called kinder. And kinder is kinder kinder kinder, what, how you want to say it kinder is the, is tooling that is used by QBADM to test the QBADM software. And if you don't know about QBADM, QBADM is tooling that is used to turn a bunch of nodes into a Kubernetes cluster. It's like a bootstrapping tool. You know what's funny is I always think of kinder as like this particular elf-like creature in old D&D books. Like that was my exposure with kinder. It was like an elf-like creature. So kinder does a bunch of things that kind does not do. And so if you're really into kind, you might be interested in this because it does a bunch of different things. Like you can use kinder to bring up a cluster in different, with different run times. You can bring, you can use, you can stop the process where you want it and re-instantiate particular parts of the process, the QBADM process where you want it. And so there's a ton of stuff that kinder does that are kind of out of the scope of the usual kind project, but it is pretty amazing. The way less painful since 110. Yes, for sure. Also it's still kind of like the premise behind things like cluster API. So QBADM is continuing to move forward. So if you're interested, Russ, if you're interested in kind, definitely check out kinder. And it's under the Kubernetes, I guess it's actually in the main repo surprisingly enough. Cue, main repo QBADM project underneath TreeMasterKinder. And it's pretty neat. I'd be kind of a fun episode to do on that, actually. So, probably booted up by now. Yeah, there we go. Oh, that's not what I wanted. CubeKettle, is this me? Y'all are making me wonder if my internet connection is trashed suddenly. Downloads at 400 meg, uploads at 300 meg. So it's probably not me, because I should definitely be able to eke out a broadcast stream for that, like my goodness. Is there something consuming my CPU? That's the connection we're on. There's OBS, that's the super obvious. Although there is a lot of usage going on. So this is sort of the same bash image that we saw before from Docker. Do apk add curl. And then we'll do a capsh, actually add capsh. And here we have our bounding set, which looks very, very similar in my mind to the unconfined set that we saw before. Let's go ahead and grab those and we'll put them over here. Let's do our filtered test just like we did before and we can look at the output of that. You freeze for a few seconds before it drops in quality. So maybe something is dropping quality automatically if frames are dropped for a bit. If that's the case, boy, that's irritating. That might be the case. Because if I'm not moving, then it's like, yeah, that's all I'm gonna send. I mean, I guess my alternative would be to stream, would be to stream to restream using OBS and bypass the camera in thing. But the problem with that is that they can't do stuff like this. That's actually why I'm using it. So I can highlight very valuable comments. We can see that the defaults, we can see two things from this output. We can see that the defaults that we're in are not limiting much at all. The same like 20 system calls that are limited. And I didn't have to tell it to go to an unconfined mode. By default, it's already in an unconfined mode. So inside of Kubernetes, if you don't use something like security context to specify a secump profile, then you don't get one. It's unconfined. No, no, don't do it, Dave. It is valuable. I hope you know I was just kidding. So we've done three things and we've seen different outputs, right? The first thing we did was run Docker in an unconfined mode. And we saw that we were able to make everything but 20 system calls. And then we ran Docker in a default mode, which meant that we were using the Docker runtime default. And we saw that they were like 80, or is it 90? It might be 90. Let's take a look. So filtered. That's the unlimited one. 60, sorry. There were 60 system calls that Docker limits by default that are there that we can see the output of. Now what I'm curious about and what I wanted to play with in this episode was if we reconfigure our kind cluster with that feature gate, then it should make it so that when I just do a Docker run of a container, then what that means is that I would be running in the default runtime mode which should limit the number of system calls that I can make as just every container in the system. Should we take a look at that? I think that'll be the interesting part. Next out of here, I'm gonna do kind, create, wait, hold on. Then after. Was runtime default, right? I think it was. Nope. It is a second default. Is an optional cubelet feature gate as well as corresponding second default command line flag. Oh, both have to be enabled to use the feature. Oh, well, we'll see. We'll see how it goes. The theory of this should not work, but I wanna see it anyway because that's how I roll. Kind, create, cluster, config, after. Name equals after. Does I already have a kind cluster brought up with before that I can't, I don't wanna create, I don't wanna mess with it. So I'm creating a new one with a new name called after and we should be able to see that. I was gonna ask how Cades tells the runtime to allow that, but how, but they both have to be active. Yeah, that seems weird to me. I thought that like effectively as a feature gate then it should be kind of configuring that stuff, right? Like science, it is science. While that's coming up, I am going to look up two things. This will be a fun one. So according to this, let's go look at the reference for, you would think that, okay, maybe I would think that the second profile, I wouldn't have to apply this on everything, but it looks like I do. So split my thing horizontally here so we can see this happening. We're gonna make a patch. And for this, couple different things we're gonna have to do, but we're gonna, oh, you know what? This isn't gonna work. Anybody know why this isn't gonna work? I'll tell you why. We don't know. I just saw this myself. I'll give you a hint. The answer is on this line. Anybody? Anyone? This is a feature flag in version 1.22. Ha-ha! Yeah, exactly. You got it. And I believe that mister, that we just pushed, I think we just pushed a new image, which was the, oh, kindest, node. I think we just pushed a new image that was 1.22 when it got released. One way to go see that is to look at the tags on the kindest node project in Docker Hub. 1.22.0. Pushed by Mr. Ben Elder himself. Copy. All right, so that should take a minute while it downloads. I'm gonna go to github.com. I'm gonna swipe some work I did before to understand how patching works. Actually, you know what? We should be able to get the same output from here. Extra mounts, extra port mappings. So this guy is part of the init configuration. And then we also have, so we have init configuration, cluster configuration, kuproxy configuration and kubelet configuration. What we're gonna be doing is we're gonna be hacking that extra command line option in. The difference between join and init is really interesting, but you can conceptualize it like join takes care of things when you're adding a node that is not a control plane or the very first node that you're creating. An init handles things that are just on the way that maybe kubelet would be configured on that node that is part of the control plane or one of the very first nodes that you're creating. So if you have init and join both defined, then you're gonna be covered. But if you only have one covered, then you're only going to be manipulating the things that are going to be like the first thing into the cluster. So I think that should have us there on the configuration. So let's take a look at this here. kubectl get or run it bash image equals bash. Same bash program we had before. Add curl, curl minus LO. kx.work slash mi-contained x and we see no difference. Still 20 syscalls and syscomp is still disabled. So it's definitely not, it's definitely working as advertised. You must at least have both created. So I'm gonna go ahead and delete, kind delete cluster, name after. I'm gonna write this and then we're gonna do kind create cluster after v1220. And let's see if we get it. Hopefully this actually works the way we expect and we're able to see it. So that would be tremendous. Doot, doot, doot. Let's take a look. It's like things are kind of taking a minute to start up here. I don't know why the control plane's taking so much longer. This used to be much faster. And so I'm, I need to dig into like what's happening here. But one thing we can do though, because the nodes have been created, but they haven't been joined yet. So I'm not very happy. So somewhere, I probably in the way that I've specified the command line flag, things are not working very well. Let's see here. I see runtime default in valid syntax. Here's what I'm gonna do. I'm gonna do a cubelet, help. Oh, I'm gonna get kicked out. All right. So it looks like the argument passed, but what the content was, the content was incorrect. And what I saw before looked like that was right. So what if I get rid of this? Oh, you believe it empty. It's this weird curse, kind of a blessing and a curse, right? The blessing is, hey, if you mess up the configuration, you can go look at the configuration and see why it's messed up. The curse is if you mess it up, then you have to go look at the configuration and see why you've messed it up. Yeah, I don't know what's happening. I mean, oh, nope. Nope, still unhappy. Okay, here's what I really wanted to know. Cubelet. Oh, greater than one. Secump default. Nable the use of runtime default as a decump. Yeah, feature game must be enabled to allow this flag, which is disabled by a per default. And the secump profile root string, which is fine. So secump default, oh, yeah. The secump default feature flag is enabled. The secump default argument is being specified, runtime default is the only argument. I want it for running into a bug. That'd be shocking, wouldn't it? Kind create cluster name equals after those kindest. No, we want 22.0. Oh, that's why, and fig equals after. It's all good, Dave. What do you mean you can't read any of the errors? Like is it my text fuzzy again? Or is it just that it's going by so fast? If it's the second thing, we're in good shape. While we're waiting for this to boot up, I think I can actually see that this is my case real quick. I just want to prove it. Oh, I did it right. Arg. I'm gonna have to figure this out for next time. I don't know why this is happening. I know it's not my connection, life of a streamer. Yep, that's true. That's a good question actually. I don't know the answer to that one. If so, I can always just redo this episode if it's all messed up. It should be all right. We should get our cluster here in a second and then we can see if this works. Oh, you know what? It's the same problem probably. I think we are kind of running into the same problem. Well, I think we're gonna go a little off the reservation here and see if we can figure this out manually. So I'm gonna go ahead and bring up. I'm in the middle of a stream, just so you know. And I'm gonna do which actually I'm gonna do pgrep-cubelet-a, could all minus up all u-cubelet. That's still broken. So now what I'm gonna do is I'm gonna go mess with the cubelet itself to figure out if I can figure out what the parameters should be. So let's see. Actually, it should be under var live, right? So let's see. So there's the param, wait, what is that? Okay, just extra args. So I'm gonna go ahead and edit that. Let's see if we can get that working. Var live, cubelet, cubetium, flags environment. Update. It is rather like clustered. Yeah, my whole life is kind of like clustered really. Which is why I think partially why I was so excited to see that show start up. And I think David's doing such an amazing job with it. Like, we have the tools. Let's get this figured out. Well, they have cubelet, cubetium. So this is what's being passed as extra args at the moment. Well, what if I wrap this as a quotes? Same error. It's basically telling me that the argument that I've passed is not, the text isn't being parsed. What if I get rid of it? What if it's just like that? I just saw an interesting output. So seccomp default feature gate must be enabled in order to use a seccomp default flag. Maybe, ah, maybe our whole problem the whole time has been the feature gates not working. After you know, it looks right to me. So there's two ways we can enable this feature gate. So there's a feature gate enablement that goes this way, right, into the configuration piece of it. And then we could also just patch the cubelet with that particular feature gate. Oh, this is a cluster wide thing, which I would have thought would have actually worked out Kubernetes feature gates can be enabled cluster wide across all Kubernetes components with the following config. Maybe I need to go back and look at the instructions a little bit more. No quotes in their example, but that's how they're turning it on. So that really should work. Provides the minimum required Kubernetes and seccomp default feature is in the kind configuration. So here it is. Right, yeah, I got that. And I did it, but I think I was quoting it. Maybe the quotes are messing me up. I've been burned by quotes before. Oh, good question. That's a very good question. Be kind of wild. Knowing how kind works, that would kind of blow my mind, Dave, but I'm ready for that. Yeah, and that's another good point. It is documented only in the 122 side, so I can see that, yeah. What it's complaining about is that it says that the feature gate's not enabled. So I think what I'm gonna do, I think we're having a bug where the feature gate's not getting populated down. R, lib, cubelet. No, that's not it. The feature gate is being populated. Seccomp default is being set to true and pgrep-a-cubelet, or I guess we, seccomp default is set to runtime default, just like it says in the docs, but it's not doing the thing. You're right, Dave, this stuff does happen, but I just checked and we are on 122 on the cubelet. Also, we're reading the docs, and the docs will only be embedded in the cubelet that was running, that had that feature gate enabled. And so, what I mean is I was reading the output of help. So if I was reading the output of cubelet help, and I'm seeing jazz about seccomp profiles, then it should sign me off on that one. Russ, I can see that the feature gate is enabled in the cubelet configuration, and I can see that I'm passing in the argument. Right? That'd be wild, too. The seccomp default feature gate must be enabled in order to use the seccomp default flag, but it is. So I think we're hitting a bug. It's either a bug or there's a typo somewhere super not obvious. Pretty sure we got a bug, because for all the world, fails to load cubelet config file, violabcubelet config.yaml, error failed to decode. Feature gates read Boolean, expect t or f found slash. So I can't quote the true, that's for sure. I was able to make a change in the config file that changed the feature gates that were enabled, and that shows me that that is set correctly. It works out pretty well. So this is seccomp default true, and that seems to be applying. And then when I go back to the cubelet ms, they're just making it not obvious that you need to know what the default is and specify the right thing there. Then the error would be different, right? Cause the error is a moment, unknown flag. Is that my bad? Oh, that is my bad. Invalid argument, parse bool. What is actually supposed to be true? Failed to validate cubelet flags. The seccomp feature gate must be enabled. So however they're doing the validation of the seccomp feature gate, appears to be incorrect, because this looks correct. Yeah, that's what we get for playing with the sharp stuff, you know, seccomp default. Okay, well, yeah, not a bad idea, that's true. That'd be super weird. If you get this, if you get it right, I'm gonna send you like stickers or something. There is however, this option, which means I think, you know, they're not taking into account the dynamic. So what this does is it basically, it's a command line flag that tells which feature gates are enabled. Let me just go over here real quick. We'll look at cubelet feature gates. I think that's what's happening. Dash dash feature, which I think you pointed out earlier, or one of you two did. So it is Boolean. Maybe if I put it on the command line itself, that'll be enough to get me there. Oh, now it's trying to actually load the, the config was definitely correct when it was capitalized. We now have a way forward. Delete cluster name equals after. Then after YAML, feature default equals true. Create cluster config after name after. Image, kindest, node v1.22. So I'm real please, I'm invested now. I like that. Okay, so basically what I saw was, if I pass it, if I pass on the CLI, the command line for cubelet, it was actually, it was actually taking it. And there were two things that were happening. So like in the cubelet security file, I had tried out lower casing, the second default flag. And as soon as I did that, the error in the logs changed, telling me that it can't be lower case, that was my bad. So I put it back to the correct case. But then the problem was that the, even though it's the correct case, the, it appears that it only checks the command line flags. So that feature gate isn't on the command line of the cubelet binary, it's not showing up. And so the testing for whether that feature gate is enabled or disabled, that's the bug is that it's only validating that against the command line and not against the actual configuration of the cubelet. Wow, we're already an hour and some change in. I'm almost done though. I just wanted to see this work. And then I wanted to show it and then we're going to close it out. So we're almost there. Thank you for hanging out. It was fun to kind of deep dive into cube ADM and troubleshoot and all that stuff. These alpha features, you know, super sharp edge. 104298, correct, Russ. That was the problem. Cubelet fails to start. If dynamic cubelet config feature gate is enabled. No, this is not the same thing. Since 122, enabling the feature gate on cubelet causes the cubelet to fail to start. Nope, in my case, that was working okay. Also, I wasn't trying to do dynamic cubelet config. The problem was more that I needed to get it. The way I read this, what he's asking about is, so the cubelet can hold. So a dynamic configuration means one that you can actually reconfigure on the fly. And I'm not using that functionality. Oh, interesting. Okay, maybe you're right. It is related. Well, if we had oodles of time, I would grab this patch and prove that it works, but it still doesn't. So we're still seeing the same fault. Dang it, this is my last shot at it too. What if I get rid of this? You know what? That's why, because this is supposed to be true. That was the other thing I figured out. Okay, so then thank you, Dave, for pointing that out. You're totally right. I might try building that later and seeing, building with that patch later and seeing if it actually solves this problem for me. But I think that was the problem that I think you've highlighted the actual issue. I don't know. I think it's a hard tie between the two of you. Probably still got a problem though, because the control point is still not coming up. Same problem. You know what? I wonder if it's like, you have to all have all three. Then after, that would not be surprising to me. Let's give this one more try and then I'm gonna call it one or the other because I'm out of time. And it's bedtime for you. Are you in the UK or in the world are you that it's bedtime? Or is it always bedtime? Germany, nice. Oh yeah. All right, same problem. That's all the time I have for this today. We may come back and visit this another time and get it working. But at the moment, I'm gonna look at the error real quick. Control plane, journal kettle, I'm gonna set value to Qubelet, crap. Well, don't go enabling this flag yet. I'm just saying, probably not the way to go. Anyway, thank you both. And thank you everybody else because maybe we're on May or May not be listening. I appreciate you. I'm glad you were here. I had a bunch of fun. I'm gonna try the next thing I'm gonna try is actually grabbing that patch from ExMudry and building it with that patch and seeing if that actually solves my problem also. But I do suspect that this is a bug that we're running into. So thank you all so much. And we'll see you next time. Have a great, great week week.