 Good morning. Good afternoon. Good evening. Wherever you're handling from. Welcome to the latest Cube by example show. I am Chris short joined by the one and only Mina camera. I can't say your name I'll just hi Mina. Sorry It's been a long past three days for me. I've gone without power cell service or internet for some almost two days and Getting back into the groove of things when going from like glamping to Internetting is really really like Different young he books, right? It's like when a president leaves office, you know, and they kind of like lose so much of the gray You know, like like you look lovely. It's like it's like you're reborn. I know my wife said this weekend was like two weeks and three days. So I Don't disagree, but thank you So and I'm laying in white. So Also the host of another show called the level of our and this is a show that we do to try to give you some So we recently launched this cube by example website to try to connect people into You know understanding Kubernetes and so, you know, hopefully from a pretty introductory perspective But really to try to take you through quite a long way and there's and and what we're trying to do with the site is actually Support multiple learning styles, right? So there's written stuff. There's video stuff There's and then we also are trying to give you what I think is one of things really important about a software project Which is to understand where it's going or like what it's those is And the reason is is because it's it's much easier to make a decision about like should you commit to this thing? If you can say, oh, this seems like it's aligned to where I want to be and where the project wants to be in Five years, right? So it's much simpler to make choices that if you can kind of understand like, you know, I refer to it Probably incorrectly is like the philosophy behind the project and so we're doing that a few ways one Mina Provides us with regular kind of news updates on the site So that you can kind of get a smattering of what's going on as long as as well as some recommendations about what to kind of follow To try to get that philosophy ethos part of it, but we also do this show It's called the KB insider and here's where we try to interview people on the show who we think are having a pretty strong impact on the Kubernetes project And so what we want to hear from them is what is their vision for the future of Kubernetes so that we can say? Okay, if I take, you know, if I take this person's vision and that person's vision and that person's vision I can come to an amalgam of where I'm I can align my work or my projects or or my IT department or whatever with where I Understand Kubernetes to be going because the people who really know where Kubernetes is going and people who are typing in the code Not the analysts, you know, not the pundits and all that stuff. So that that's what we're trying to do this show and today we invite Alana Hushman and We will ask her to introduce herself in a few minutes But before we do that, we have a little new segment where Mina gives us an update First on how to pronounce her last name But then also how what's going on in the community since our last episode our last episode we run the show The last Tuesday of every month So we've done one real episode so far We also did a behind the scenes episode where we kind of get into more detail of what I just explained So we'd love you to watch those two and obviously come back next Tuesday or sorry a month, you know I'm the last Tuesday of the month And you know and as we don't say often enough click like and subscribe That's how we can show that we're performing exactly. Thank you a lot of It's show how we're performing on the show and you know So we can kind of continue to have the time to be able to dedicate to this. So Mina Why don't you take it away? And we'll continue from here. Yeah, absolutely. Good morning, everyone If you've been with us before you've probably seen my face before. I'm Mina Karamajan It actually means black coral and Turkish fun fact my last name. It's a little it's a little hard to pronounce I know but I'm and to be honest my southern public school Background in education. I will completely blame that so I didn't start traveling the world until after my first day of school Yeah, I get a lot of problems as well But yeah, I'm here to give you a highlight of what we've what we've put up on the KBE New section of the Q by example website. So we're gonna start off with an article we put about three to four weeks ago It said Kubernetes complexity slowing adoption of containers. So what this means? There was a survey of a thousand cloud developers and it found that the steep learning curve and About 57% of those thousand cloud developers said this new terminology as well as new concepts and commands are the most Significance check significant challenges. They faced what's more many were frustrated by how long it takes to spin up a work in Kubernetes cluster. So this is a pretty pretty important finding a thousand people Definitely go check this article out if you are interested in hearing more about it We also highlighted the top 10 hottest Kubernetes startups of 2021 2021 I'm not gonna list all of those right now But I will put all of the links to these articles in the chat after I'm done. So definitely go check them out We also talked about how Kubernetes and containers are powering tomorrow's applications digital businesses today are leveraging cloud native technologies to power their applications Their application stacks and Kubernetes and containers are essential in enabling them to achieve this The author highlights a couple points in this article importance of data centricity when it comes to business is staying ahead today Container adoption speaks for itself according to a 451 research 95% of new apps are developed in containers today microservices and containers are a perfect match as Containers are the perfect building block for microservices They present a lightweight and consistent environment for microservices that can follow the application from the developers desktop to testing to final deployment Packaging applications with their dependencies are the trend today. That's what people are doing and not all container services are equal Kubernetes is a black hole of unpredictable spend according to a new CNCF report in association with the FinOps foundation shows that costs of Kubernetes are rising and companies struggle to predict them accurately spending on Kubernetes was up according to the survey 67% reporting an increase of 20% or more over the last 12 months and about 10% of those of those companies spend more than one million dollars per month on their deployments, which is Pretty crazy if you ask me And then we looked at the future of cloud computing and what trends will impact your organization One manage Kubernetes in the cloud will be a really really big focus to the idea of multi cloud will become a lot clearer Three server serverless computing will quicken and simplify cloud operations For managed open-source services are going to be in the forefront of things five service mesh networking will become the norm six automation and policies will ease cloud management and Seven better integration between cloud and data centers is something that we're going to see We've also seen some Kubernetes security issues through different articles nearly all survey respondents in a recent state of Kubernetes Security reports cited at least one incident stemming from Kubernetes security issues Again, I will drop the links into to these articles, but we also have seen some Kubernetes clusters attacked Via misconfigured Argo workflow instances security researchers are warning Argo workflows is an open source container native workflow engine for orchestrating parallel jobs on Kubernetes To speed up processing time for compute intensive jobs like machine learning and big data processing. So these are pretty important I definitely encourage you guys to check these out. There was a lot of noise going around this on threat post and social media as well We've seen that IBM plans box boat acquisition to continue their hybrid cloud business expansion strategy In a push to further explain further expand their hybrid cloud services business We've also seen two pretty important Announcements coming from Kubernetes. So I really want to highlight these as they were announced in the last couple of weeks on April 23rd 2021 the release team merged a Kubernetes enhancement proposal changing the Kubernetes release cycle from four releases a year to three Releases a year. So Kubernetes 1.23 will be the final release of the 2021 calendar year And the major change end users will experience is a slower release cadence and a slower rate of enhancement graduation Kubernetes release artifacts release notes and all other aspects of any given release will stay the same another Announcement coming from Kubernetes is as the Kubernetes API evolves APIs are periodically reorganized or upgraded when APIs evolve the old API The old APIs that your place are deprecated and eventually removed Kubernetes 1.22 do for release in August 2021 will remove a number of those deprecated APIs These are beta APIs that you can use in current supported Kubernetes versions and they're already deprecated the reason for all these removals is that they have been superseded by By a newer stable GA API Like I said, I will drop the links in for all of these articles But we are updating our KB new section every Tuesday. So definitely go and check it out today as well and See and be in the loop on what's going on in the Kubernetes world Kristen Langdon back to you. Thank you Wonderful. I just I also have to comment because you didn't cover this part of it But the the best part about the Argo CD hack was that of course, it's Bitcoin mining Which I thought was impressive Way back in the day open shift calm when it was open shift to One of the biggest challenges basically it offered, you know, kind of free, you know, things like containers, but they used to be called gears To you know, the general public And basically they had to have a constant watching for bitcoded mining and that was a big chunk of the effort involved So again, we'd like to welcome Alana. Thank you so much for appearing on the show. Thank you, Mina. And if you could give us maybe a brief introduction to yourself, you know, You we know you work for red hats. That means we have no idea what your title is or who you work for because it changes to regularly to keep track of You know, so if you could just kind of just a little bit of that and then we can talk a little bit about Kubernetes Yeah, absolutely. Alana Hashman. I currently work for the OCP node team as a principal software engineer And previously I was on the Azure Red Hat open shift team as a principal SRE and one of the tech leads So that was only recent change I only moved over to the open shift engineering side of the house in January of this year So they got tired of you complaining and said now you have to come fix it. No, not at all. It's a long story. We can catch up over I'm really happy that I'm not on call anymore. I think that's probably like as you know died in the little boss that is my biggest Right, right. I one of my first kind of job offers out of college was working on IBM's level two global backbone for their internet And yeah, two weeks out of the month. You weren't allowed to be more than two hours away from from the office Wow, because they would get calls like oh, San Francisco is down. Please fix Yeah, I was like 15 minutes or something like that. It's only gotten worse. That was that was many years ago because I but you know So we we kind of have like some questions we like to ask kind of all of our guests And so why don't we start there? And so kind of you know, what brought you to kind of contributing to open source in the first place? That's a long story. Oh When I was like, I don't know a young glass I discovered IRC at some point and like the mid aughts and a friend was like, oh, have you like seen this Linux thing? And I'm like No, what's that? And they're like here. I'll let you access one of my machines. I was like, what? And so I don't have like some Windows machine and putty Assessations this Linux box. It's like, well, this is cool. But like what am I supposed to do with But you know like it being I don't know like probably 2006 or something like that. I was I was installing I think it was a bunch of hardy Heron six dot oh six And so like I was like, okay great like I think the sound worked But the Wi-Fi did not and so I got to fight with Endis wrapper and it was a mess and it was my introduction to open source software But like prior to that I had been using Firefox, which I thought was super cool And I was mostly just kind of a user for the longest time And then in university I like went to some conference and there was like an open source themed lunch table So I sat down at the table and I'm like checking my email or something like that and the guy next to me He like looks at Marshall. He says is that alpine? Yeah, he's like hi, my name is a sheesh. I'm the alpine maintainer in Debian and I was like, oh my god That's awesome And so there was an open source day at the conference that year and he's like you should come and Contribute to this cool project called open hatch. I'm like, well, you know, like I've been using open source for so long But like I don't really know like like I matched to like contribute to WordPress or something But I don't like no php is like no Come contribute to open hatch And so I did and I started fighting with this bug which he's like, oh, yeah pretty easy bug episode I started dating and no it's not an easy bug But I am Like oh, yeah, I'll just work around this real quick. You know, you can fix it today. No did not happen But I ended up learning python and I became open hatches gsoc students. So I completed google summer code and Like fast forward then I started working for rack space and I got more involved in like other pythonic things I was working with paul carer So I wasn't like directly contributing to cryptography, but I was encouraged to do things that might help cryptography And that got me like sucked into the world of like, well, why don't you try out this many Linux thing and like help us build wheels better? And then, you know, how that goes like you stare into the void and you become an expert on the void and suddenly I'm like the only person Who understands how this symbol versioning logic goes for like the abi compatibility and audit wheel And yeah, no, I maintained that for a while I still participate with the the python packaging authority At the same time we used a lot of closure at that job So I was working on closurey things and someone was like, hey You really need like this thing which is the build tool for closure called line again Like someone should really package that for deviant. I'm like that sounds like a terrible idea like what a bunch of Who would be so dumb as to like go and waste all this time working on that? Yeah, I was nerdy tonight So, uh, I guess like however many years later I became a deviant developer. I'm now serving on the deviant technical committee Uh, and yeah, I do and are maintaining closure at this point, right? Yeah, I'm the maintainer of closure line again and like, I don't know a bunch of random, uh, like closure libraries that Are used to build either and the exciting thing is that uh, we have a gsoc student now working on packaging things I don't have time for so it's like it's come full circle Right, right. See, I mean that that's the the key right is, uh, you know, bring them in young, right? And you know, get them get them into the mix and then you know, you can you can finally hand some things off Um, you know the dream, right, right, uh, you know I still maintain though because of your maintenance of closure that you are a closet lisp Lover and therefore oh, it's I'm not closeted I am like way out of the closet All right, all right So flashing my red hat Oh, yeah, I'm sorry. I was actually using the more general term of Just, uh, you know secret in lisp in you know user, uh No, no, no, nobody nobody uses lisp in secret. It's like it's like, you know It's like you know because they've told you they use arch right, right, it's uh Oh, I'm blanking on the joke, but you know, it's the uh, you know, how do you know? Well, yeah, how do you know if someone's vegan or does the uh, what's the the exercise keto? This is the keto diet now too. I was thinking extra crossfit. Yeah Just you know talk to them and uh, and they'll tell you they'll tell you right now, so, uh, I uh, Yeah, it was very funny. I I totally agree with you. Actually, I would say scheme is is more like that than than lisp even Um, but uh, you know, but they do have a whole book called the little schemer, which is amazing Um, all right, so bringing it back to the focus of the actual show, uh, all right So, okay, you know, we see how you kind of got into open source, right more than just software per se But so what brought you to kind of kubernetes? So, you know, where where did that? Uh, so when I was at rack space, I was working in managed security I've worn many hats in my software career And so I was working in managed security and I was on what we call our defensive infrastructure team And we had a bunch of containers and at that time we were running with like core os nodes And like using fleet to manage them and I was a little bit of the opinion. This has been miserable Uh, and we also had like VMs that we were managing and I thought like Yeah, I was learning about this new hotness called kubernetes Um, and uh, you know, there were various flavors of kubernetes. I heard about open shift at the time But I thought like, you know as uh Reading like white papers and things like that. I was like, wow Like, you know, we can like save like 30 compute overhead if we don't virtualize and like, uh I would love to not have to deal with like all of the overhead of dealing with VMs, uh, if you look on like same at devian bug tracker You will find that my first contribution directly to devian was complaining that libvert was broken. So Yeah, I think that was fun. Yeah, so like, you know, it's like this is like this new thing And like maybe this is going to be kind of like the future of linux Like we've got these like all these cool abstractions to like sit on top of the kernel abstractions that are containers I have been using lxc for years at that point Uh, and like wasn't really necessarily as much like we were building docker stuff But I wasn't necessarily like a docker enthusiast Uh, I was like, okay, well docker is like a means to the end But like I don't really like fighting with docker compose very much. So I thought ah kubernetes This is going to be the next thing. This isn't like, I don't know 2017 or something like that Um, but I'll ask my team at uh rec space wasn't really doing cube stuff So I started interviewing to like try to find a job where I could do kubernetes stuff And I landed at a Financial company where they were like, so we've got like these gigantic kubernetes clusters, which by the way, we're fine as companies So for us, they're like tiny, but we've got these gigantic on-premise kubernetes clusters And I was working with like, you know, 10,000 core clusters and those were toy clusters Uh, and they're like we need somebody to like help our ops team Administer these things because we have a bunch of software engineer if you don't know a lot of operational background So I was like, well, let's do it. Uh, and so I spent a year there starting with like I think when I got there, they had just upgraded from cube one five to one eight and everything was custom patched And we had like kerberos all over the place and uh, I got there and they were like so we'd like some metrics for our clusters And you know, like there's this heapster thing that like we really need better implement instrumentation So I was like, okay And I like looked at various solutions and I was looking at various vendor products And I was like, huh these all just kind of like sit on top of like the open source stack Like there's nothing there's no particular secret sauce other than you know The like from what I could see playing around with these trial products at that time in like 2018 I was pretty early This is like well, we weren't really using a unified pane of glass anyway So there wasn't a lot of benefit to having these things going into a vendor dashboard So, uh, they were like, can you make some metrics work? And so I started playing around with all these open source tools We had Prometheus, uh, I spent a much time working on cube state metrics performance testing cube state metrics because it turns out our clusters were big Uh, and uh, I ended up getting involved in sig and stir rotation as a result and for the longest time I was like, you know, kind of there as like sort of an oxy Focused user. I wasn't doing very much in terms of code But I did submit a two line patch that like broke the entire world trying to like Trying to just like Label normalization because I'm like, well, I'm doing this like horrendous join In order to like make these metrics match these metrics and they don't conform to the kubernetes like Instaritation guidelines. So like can we fix these? They're like, sure. It's a meaningful request Uh, I was more like make a cap and then submit a full request. It took a very long time But uh, yeah, so now I chair the instrumentation sig as of uh march of last year Uh, and uh, recently since I joined the node team at red hat. I've also been spending a bunch more time on sig node That's cool. So Let's talk about let's continue talking maybe about instrumentation, right? I think, um, you know, there's a lot there, right? I mean because Trying to understand the environment particularly when you talk about anything that's cloud native is often one of the hardest parts, right? You know, the second you introduce these micro service things, right? The problem is is trying to understand what's actually happening at any given moment. Um, and uh, so What do you see like or what what do you feel like is the kind of biggest? You know change that you've seen while you've been involved in that sig and what do you think is the next kind of one You think is coming So I don't profess to be an instrumentation of thought leader, but I would say there are like Two big things that are in flight right now that I am very excited about that. I think will Significantly improve instrumentation. So one is that api server tracing is happening Uh, so this means if like a given request goes into the api server You can generate a span which shows like I spent time here. I spent time here. I spent time here Oh, look, this one's very wide. Maybe we should see why we're having performance issues here Uh, and so on and so forth. Uh, I dealt with a lot of api server perfect issues In a past life and it was really really difficult to tell where like things were blocking and why we were having performance issues and Uh, when you're dealing with like just black box monitoring tools like from the outside where I'm just like Running io stat trying to see what's going on with ncd Like it's it's it's nice to have a sort of internal like white box Sort of like representation of instrumentation. We're like, aha. It is stuck here. Why is it stuck here? Oh because it's blocking on system d because it's trying to write all of these logs into the database But like why is it logging 100 megabyte thing? I don't know. Uh And that is why my command is blocking. Uh, so that's the sort of thing that hopefully uh tracing will be great And we just finally um, I'm so happy uh had that land in 122 alpha api server tracing David ashbowl has been working on this since I joined like the instrumentation sing in like 2018 So that's super super exciting The other thing in flight, uh, that I think will be super helpful Uh, right now, you know Kubernetes generates a lot of logs and as a systems administrator like It's all sys log and sys log is kind of a pain to deal with and it makes it really hard because you have to do like Post-processing with reg X's and like all sorts of other stuff to try to like do correlations It's not like here's a pod and now I can trace that pod or here's a request ID and now I can trace that request ID uh, and so like that's Always been like an issue in upstream cube and like way back in 2018. I was like Could we have structured logs, please? Like I thought At the very least like Like five years ago. I thought this was a thing but apparently not. Um, and so Uh, yeah, actually, there's uh, we just formed and I uh helped sort of sponsor the creation of it Since sing instrumentation, uh, was the sig that kicked off the effort A structured logging working group. So like it's much larger an effort versus like a single cat Uh, and uh, yeah, it's it's been quite a wild ride So like in 121 I spent a bunch of time working on trying to migrate the cubic to structured logs And then ensuring that we had like automated tooling that it wouldn't regress and so that's been awesome Like cuba it's on structured logging. Uh, but structured logging is an effort of stolen alpha We still have every other component to migrate. So, uh, that's been a super cool thing that happened That will be great in terms of sort of future instrumentation stuff Like I'm you know, old schools to submit. I just want tools that like can help me do my job Uh, and so it's like really like, you know back to basics Yeah, yeah, I want my metrics to be accurate. I want my logs to be agreeable And I want like traces and events that are useful So I yeah, totally area. Um, they have a long time ago now like, you know, 2000 or so Maybe before that uh from mine and I had built a system that did uh visual basic over hgp Uh, and one of the things that was most important about it was actually doing a visualized Logger to show where you know where it's stuck, right? Um, you know, because it's it's just so necessary Um, one thing I did want to quickly cut a sidebar here a little bit. Uh, you can use the term cap twice Um, and could you give a little bit of uh to our audience? You know, it's a lot of new people. Uh, what does that mean? Uh, sorry cap Yeah, I thought you said um theorem. Oh Uh, I don't think I said anything about cap theorem Oh, I thought uh, so maybe I misheard you. I mean, I love the cap theorem, but Yeah, but Right. That's a that's a whole another uh discussion. Anyway, um, so uh, yeah If you do want to know more about cap theorem, uh, go the rickety page is actually remarkably good. Uh, I would definitely go Search that all right. Well, maybe it'll come up again. Uh, or maybe I misheard you So why don't we move on from there? Uh, and talk about a little bit So, you know, we talked a lot about instrumentation, which seems like, you know, you're very passionate about And now you're working with, um, the node sake, um, which If you could tell us a little bit about what you're working with there and what you see there And actually if you explain what the node sake is that would be a good start as well I would be to like that one is I for me at least the the naming and what it is like Have a very hard time sticking in my brain Um, I will try to break it down in a way that will stick maybe. Yeah, hopefully so so kubernetes Do I have anything with a kubernetes logo on it? It's like handy if I do Oh, I got that. I don't think I do. I like pointing at it. Oh wait. No, I do Thank you, josh bergus Oh, very nice. Very nice. Okay. So you see kubernetes. This is a kubernetes cluster the logo, right? So the hub that's the api server and then all of these other things are controllers talking to the api server Everything goes to the api server. So a node is a special controller And why do we need a node because we want to run workloads? So the the node the cubelet is the thing that turns something into a kubernetes node It is a special controller and all controllers and kubernetes they tend to operate on a certain object in this case the The cubelet cares about pods So the cubelet is the thing that natively understands pods and what to do with them and all of that jazz So sig node is responsible for the node and when we say the node we mean like the cubelet really And other things that interact with like node life cycle. So Sig node is a really interesting and large sig to be in because you know The most minimal api or the most minimal kubernetes cluster doesn't have any nodes You can just have an api server and an ecd. You don't have to have any nodes like that works You could talk to it won't do anything you could put as many pods into that thing as you want It won't do anything But like that's that's a perfectly legitimate kubernetes cluster Uh, but like nodes are really interesting stuff happens. I think uh, and so that was kind of what like interested me in sig node like if I care about the sort of like kubernetes as like nice abstractions to like parallelize linux on a distributed scale Nodes where it's at so Yeah, I'm working on sig node. Uh, I think that was a very good experience. Yeah. Oh great. I'm glad that that was helpful No, I like that was solid Yeah, so what are you working on? I am working on many things, uh, but probably like my top three things are uh, so I'll start with the least exciting maybe and go to the most exciting least exciting making sure it's not broken so like working on Deflaking tests and paying down tech debt and long-term tech strategy and that kind of thing That has been uh, really interesting working on the last two releases node, uh, was like a little bit stagnant uh for a while And so like one of the things I've been looking at is like, how do we pay down that tech debt? How do we make node go faster? How do we ensure that like features are happening and that kind of thing? Uh, and so like one of the most important things you can get from that is like our tests working Our our test suites running the way that we expect them to uh, and so on and like are they giving us good signal Because we have a bunch of test suites right now for example that are failing all the time So it's like is it failing because something I did or because it was already failing like, I don't know Uh, we can't rely on them to be passing uh as is and so, uh, you know, we've been spending a bunch of work on that Uh, uh, so I wanted to interject there. So so least exciting for you, perhaps I don't know very exciting usually for users of Kubernetes that the uh, If the test the more automated testing the better the testing the less problems that consumers have right? Well, I would hope so. I mean in a lot of these cases It's not even like like I think it's it you know, maybe borderline boring not because like I love testing I have a huge testing proponent one of my uh Challenges with upstream kubernetes is that like the test covered like if you go go look at like kubernetes issue one kubelet like unit test coverage is lousy like that is like the history of kubernetes right there Uh, you know and coded issue one um and so like This is the only that I'm definitely passionate about but a lot of like for this stuff. It's not like exciting It's not like adding new tests. It's just like are the tests running are the tests passing Is there a silly reason why they're failing? A lot of the flakes are there's a silly reason why the test is failing as well as like that exciting to typos, yeah Like time out periods, uh, you know that kind of thing. We are also definitely uncovering real bugs So those are good to fix. Well, another thing that's kind of related to another question We like to ask is from a getting involved perspective. I'm sure it's doing Like amazing things for your understanding of the code base that you're now working with right You know, like that's one of the greatest things about kind of fixing tests when you're new to a project Is like you you really have good tactical things to kind of help learn it Which is hard to do in kind of the abstract You know when you're when you're kind of coming into a process at least that's been my experience Yeah, so I've been working on uh, for example, I spent all day working on a failing test that uh, was like Failing in part because the node was running out of disk space And so I did like possibly the most horrible thing I have ever submitted to cube Ci which was like I sent in a work in progress pr With like at the beginning of the cubelet I modified it to like shell out to df and dump everything and because the cubelet in uh, Certainly test suites it like constantly is restarting the cubelet because of the dynamic cubelet config um And so, uh, like you'll see the cubelet restarting and therefore every time it restarts it'll dump like df again Uh, and so yeah, uh, that was we were able to therefore figure out that uh, a test that was filling up the drive Was failing and not cleaning up And that was what was causing all the rest of the test to fail due to like being out of disk space So, uh, like you will learn how to do I guess horrendous things to kubernetes that shouldn't be done If you go down the path I will say checking storage, you know running out of storage is one of my go-to defaults for when I can't figure out why this thing is And You know no space left on device But like I was I was like really puzzled by this because I was like how like this isn't how pretty is that was it possible? Uh, and so I was my suspicion was um, maybe it's just that i nodes Uh, so I was like, well, we don't dump any i node information anywhere So like let me dump that like no, it was actually like out of space. Uh, my i node full hypothesis totally wrong But you know, that's that's why you search Right, right. Uh, all right. So what are the other two? Other two, uh, so second thing is node velocity Uh, so like one of the things that I did when I joined sig node is I was like, oh my god We have so many prs We have so many issues How are we possibly gonna get to all these things? Uh, like sig node has like 200 active prs at any given time That's a lot of prs. Yeah, uh, and so like I can't review a lot of rebasing Yeah And so like one of the first things that I did is I was like, I can't crack this I'm gonna like make a little board so I can like visualize like the things I need to pester people to approve The things that are like ready for me to put an lgtm on the things I should not look at and like the things that somebody needs to look at Uh, and so I put together this node triage board When I joined the sig and then I like showed it to some people they're like, this is super useful Make this like accessible the whole thing that I did Uh, and so now we've been using that for node triage, uh, and that has been making it a lot Easier to deal with like our glut of uh prs that we deal with on a given basis We previously had a board for just test and see I think only But because it was just test and see I think only it meant most of our prs weren't actually Like getting looked at in the same way. So I made sure everything else went on a separate board. Uh, and then I also, um When I joined I was like, oh my god, no, it has 500 bugs Uh, like Oh, what am I gonna do with that? Uh, so, uh, this past release I ran a node bug scrub Just like in any other project, you know, devian bug scrub Open hash bug scrub my open hash gsoc project was a bug scrub tool So, uh, I have many years of experience running bug scrubs and I was like, let's just let's just get him done Look at everything. Uh, and uh, we'll see where things go from there And then at least, you know, things will be all updated even if we're still not necessarily like triaging incoming stuff regularly Um Did that closed like 160 bugs? so That's a bunch. Yeah Yeah, like I feel really bad because I want to be able to look at every single incoming bug I don't have the time Especially because a lot of them are somebody asking a kubernetes question And like not it's not a real bug and therefore I can't really like spend like three hours of support every day Looking at the questions and trying to answer like, it's just not enough hours a day. I wish Yeah, yeah, we should have like a we should have like a high school class that is like filing bugs, you know Like it'd be really nice because I don't blame them But like I mean the the number of times I say like this is a support request. Please go to our forums. It's like There's a button. There's a button that says please go to the forums when you try to file support requests But they're like, no, it's a bug because it's not working. I'm like, no, it's not working because pep cat right One of the one of the most underused phrases in technology Yes, uh problem exists between keyboard and chair viewers unfamiliar But yeah, I mean, I'm often the problem to be clear Suffer from pep cat myself. Yes. Yes. Um, but that doesn't mean that uh things that I have worked on Someone else is like this would be a really good place for people who are looking to get involved to get involved Because like this is something that I'm like, I don't have the time to spend on this But like if somebody could flag me the real things and be really nice to the people who like are submitting questions Maybe try to help them like I always try to at least point them in the right direction when I close them But yeah, it's which is I mean to be frank, right? Like that's that is the right answer. That's how you build a community That's how you you know, this is how you, you know, be kind of a good person in software But at the same time to your point that's a reduction in the amount of time that you can spend on on other things right and so we you know, we always need help I think in um kind of Doing the things that actually require human components more than they require technical components You know, it's like if you have a little bit of kubernetes expertise Go try to find the places where you can say oh, I know I recognize that this isn't actually a bug and I recognize that Oh, I've run into this problem before Let me go help that person so that somebody who is working in engineering Qe or whatever can From my ops background I gotta say like kubernetes upstream full of developers very few qe people very few operational people Like people think but like there's not really anywhere for me to contribute code. Trust me Does not matter We can find you a place trust me You will have a superpower if you come at this from an operational perspective Or like a qe perspective in upstream because we just don't have a lot of those sorts of folks working on things and like I can tell you If we had someone who was like sort of watching say like the bug backlog and like helping folks And like flagging real bugs so that like other people like devs don't have to spend their time like verifying that things are actually bugs And then not actually fixing things like if we had more folks who like I can't fix the thing, but it's like don't worry I could fix the thing, but I don't have time to figure out how it's broken so Yeah, like it's it's a superpower And I am like a little bit weird in that when I became a chair of sig instrumentation Like I had like four lines of changes in all of kubernetes kubernetes Like I I was an ops chair for a sig that's like kind of owned code Instrumentation kind of owns code. We have a small amount of code, but it's horizontal So like each each component owns their own metrics and whatnot We don't know but yeah, it was like I was weird as like the ops person doing like Cube stuff upstream and it could be you and it could be great And now I guess I'm on the production readiness review team. So there you go. No rest for the wicked Can you can you give us a little bit of outline of what that is? Yeah, so every feature that's proposed to kubernetes that is like substantial So like might touch multiple components or might need to be life cycles from like alpha through the ga has to go through what's called a kubernetes enhancement proposal or cap and In the last couple of cycles, we've added this new additional review and some questions to these caps called production readiness and the idea of these is to sort of like ensure that every single Feature owner is thinking about how this feature will work in a running cluster And how to deal with like upgrades and downgrades And how to deal with like my thing has gone wrong. What metrics or like things should I look at to figure out that it's like broken and so um That team when it was initially envisioned was very small was three approvers for all enhancement proposals and we get something like 60 Each cycle Workloads so uh, I I volunteer to distribute I am now the fourth approver on this team who will look through this and be like Yes, you have a good upgrade or downgrade strategy or like have you thought about this as it applies to version skew Or like yes, this is not eligible or no No, no, no, this is applicable and you don't have to add a metric just like we need a metric because like I don't know how this thing is working What should I look at? What are you the author looking at to see if this thing is working? No, oh, I hadn't thought about that. It's like right now you're thinking about it Well, and it's almost like uh, kind of supporting like the user experience of kuhedis, right? You know, that seems like a a very good thing. Um, you know, we desperately need more people on this team So yes another place that sort of opsie people, uh, like me in a former life and get involved Well, or to some extent or even just consumers in general, right? You don't need to be necessarily running in production as much as kind of looking at it from a holistic perspective Um, you know, so I mean not to open the floodgates, but like the enhancement proposals are public anybody can comment on them So like uh, I mean this is this is maybe like another superpower when it comes to open source Like the people who do the things in open source like nobody gave them permission Like right showed up, right? Like when I got involved in things nobody did who I was Like my my reputation did not precede me in kubernetes. I built one by doing the things so That's that's all you gotta do do the things One of the things uh, it's actually kind of an internal red hat freeze that's been kind of going on I don't know from the quote it quite right, but it's like, you know The person who wins is the one who writes the code. Um, you know, and and to some extent it's a looser version of that Which is like, you know, you can kind of just show up and start doing stuff. Um, you know, it's amazing I wrote a blog by saying exactly that You have a much more defensible proposal if you have code that actually runs versus the person's saying Well, I don't know if you should do that or a or a test or uh, whatever Code is euphemistic for you know a contribution And uh, you know, and I think that's really important The the key is right is that is to be polite about it And that way and you will you know, almost always get a positive response. Um, and I think that's uh, you know, that's really important Just a segue into the third thing that I work on. Oh, yeah, please. Yes. Go ahead. I'm working on swap support Okay, uh, well, so the swap support is special because it didn't exist before And now it exists and you may ask why did it not exist and the answer is Nobody sat down to do the work I mean, there was like definitely some concerns in terms of like, how are we going to do this in a way that's not going to blow up kubernetes clusters? Please give us a proposal that convinces us and then we will be happy to implement it and that just kind of sat there And I was like, you know what? I'm passionate about this, uh, because like the First of all, it's one of those things where like you've got to Take into account the needs of like So so many users and so like the first thing I just spend a whole release just talking to people being like how you swap Like is this document abuse cases like exhaustive for you? Like is this going to work for you? Here's like a sort of a straw man proposal. Like what do you think about that? Uh, and be like no, I hate this. I mean, what if I change this thing then like do you still hate it? No that's pretty good, uh And and so on and so forth. Uh, I like doing that Which was like, you know, kind of like very like thankless behind the scenes And then I show up with like, you know, what was effective like an 80 line implementation. Everybody's like clean crisp good merged Uh, yep, yeah A lot of it's the like getting there like after, you know, I think the first issue from a rock swap was like in 2014 it was filed, uh, and it like took until like 2021 for us to like make some movement on that So like I think, you know, it's really just Rank the proposal do the thing and like be open to feedback. Yeah Don't afraid to like, you know what? I don't like that feedback I'm not going to do that thing because then people aren't going to support you Right and just because I think I know what you're talking about, but uh, you mean like swap space like memory swap Yes, that's right. Uh, so like on a given node, uh, people love to, you know, use virtual memory Which is not real memory and uh, is like, you know, usually some form of disk space And it could be very fast. It could be NVMe. That'd be awesome. It could be opting. Who knows. Yeah Who knows? We don't know. Uh, it could be pretty fast. Uh, and uh, it like swap is often used for Applications that for example have like big memory startup costs that then they're never going to touch again But they can't actually evict it from memory. Uh, so we see this on like virtual machines of very various kinds, right java like, uh, node, uh, you know, actual virtual machines like, you know, what not It could be potentially useful for virtual machine live migration where you have to like very briefly over commit memory Um, like all sorts of applications. This could be helpful, uh, because you know memory is a Tight resource sometimes especially on edge systems, uh, or uh, my favorite example of an edge system My raspberry pi. I mean don't maybe dig too much into the swap pool there or no be sitting blocking for like six years, but Exactly. Exactly. Uh, one thing I wanted to kind of catch back up to so I think I heard you saying cap But you were actually saying cap And you were talking about the place enhancement proposals. It's the canadian accent Perhaps perhaps. Um, so I didn't pick up on that. Yeah Um, so so to kind of cover that back up All right, so so the kind of I think the last question that we have time for uh, because we're almost out of time Is so what do you think is, you know, kind of in Kubernetes in general What is the thing that you're kind of most watching or what do you think the biggest and both from a positive You know, either from a positive perspective or negative Like what is the, you know, kind of the biggest challenge or the biggest thing or the biggest, you know, Awesomeness that you kind of foresee in the next, I don't know a couple of years So the the question is What am I watching in kubernetes? Yeah, and but like it doesn't necessarily need to be, you know, positive It could be negative or vice versa or both or the api server Okay No, you didn't get the joke. Okay. Oh, yes now I get Hey, sometimes a little slow. I haven't had a lot of coffee. It's 7 a.m. Here. Uh, right, right It's it's it's fine. Uh, what am I looking so maybe I'll go what am I looking forward to in kubernetes? so one thing that I have been spending a lot of time on is like trying to do knowledge sharing and Uplifting new leaders in the community So like there have been a lot of people in kubernetes who've been around like literally day one They've been here since the start and they hold lots of great leadership positions But like I don't know like do you want to be doing this forever? Like I can say in instrumentation my time is going to come. I'm not doing that job forever Uh, and so like it's really important for me to be like, okay You know, I'm thinking like a year or two down the line like Who's going to be doing this in the future? Is it going to be me? Do I want to keep doing this? If not, how am I setting up that person for success? And so, uh, I have done a couple of cohorts One I was working with a bunch of people to try to show them how to become node reviewers So like here's how you review signode prs and that kind of thing and latest cohort that I've been working with I've been training people how to how do you be a sig lead, right? Because like I work in two different sigs, you know, I chair instrumentation, I guess I also There isn't I'm not sure who the sub project lead of production readiness is But like I will often run those meetings And then I also work in sig node and while like I'm not like a chair in that sig I'm often like running the meetings or running like the ci sub project meetings or that kind of thing so Um, it's really really important. I think to like Build and uplift the sort of like next generation That's going to be doing the work in kubernetes. And so uh, that I think like is really the thing I mean not to be like everybody's like cheesy Community but like honestly like it's about people, you know, this is uh, ultimately at the end of the day. This is a job. Uh, and so like People move on from thing to thing people find new interests And yeah, I just want to ensure that like for the time that people do spend here They have the best experience that they possibly can and uh, you know, when they move on the community will still be fine It'll be all the better for it. Right. Right. No one irreplaceable Yeah, like people say that they're like, but like I want to be like, you know, the bdfl I don't want to be a bdfl. Uh, and uh cube explicitly really does not want bdfl. Right. Yeah No one wants you to do that beyond for you Right. Well, there's I mean even if you look at it from the you know, there are bdfl's or whatever Like those people are still finite, right? Because there's only so many hours in a day. Even if it's bad for them It's bad for you, right? Because like as a bdfl like, you know, you First of all, like there's so these ridiculous crushing expectations. I've been that person I was the sole maintainer of a thing that like the entire weight of binary python distribution lay upon I was like, I can't do this anymore. Like it's stressful. I don't have the time and like I'm letting everybody down So I quit I stopped doing it. You can do that Uh, and so like, uh, you know, it's I don't want to see Kubernetes do that. And so Kubernetes was founded with this like explicit Anti-bdfl mission. So now we're gonna see like, you know, when the rubber hits pavement, like can we actually make this a thing? Right. That's what I'm looking forward to. Yeah, I mean, uh, at least for me too. It's like the There's there's so many reasons that you want a community, right? The reason that we want to do this open source thing is because we think there's value in the community and like if you have You know, kind of the bdfl or just even just continued of the same leadership, right? You are limiting the number of perspectives that are contributing ideas, right? You're, you know, like you're not kind of You know, or like the the more Um, kind of correct way of saying, you know, the The the lottery factor, right? You know, somebody wins the lottery and now they you know now they want to work on their pet project You know, which is you know, the You know kubernetes on raspberry pi only, you know, uh, or whatever And so they they get to go off and do that because they won the lottery. Um, you know, there's a there's an older version of that scenario that we try not to use anymore But uh, you know, all like there's all of those reasons, right? It's not it's not even just the simplistic kind of the like Or like sorry, not simplistic more like there's lots of good reasons to encourage a lot of That are Like sometimes it's because of burnout, but sometimes it's because you want to do, you know You really get into one little deep dark corner of node, right? And you want to stay there and finish that But what happens to all the rest of the stuff while you're doing that, you know, so there's just so many reasons Um, it's like honestly the thing is centralized systems like that like they're easier like right It's prior to initially But like, you know, yeah, but you end up, you know, you're bandwidth constrained by that one person It's really hard to hold them accountable because they're the ones calling all of the shots. They have effectively all of the power Uh, and so like that sort of thing, you know, it's just like It's hard to be able to step back and move on It's hard to be able to sort of like hand off the torch but like honestly, I think that you know it Thinking about the tech and whatnot like the tech will move on like, you know Do we need to rewrite the kubelet or something like that? Well It depends like are the right people going to be in the room to make that call? I don't know. Uh, so that's what i'm working on and uh people should judge me on that in five years Right, right. Yeah, we'll uh, maybe there should be a you know part of the instrumentation sake should be those metrics Yeah Well, thank you so much for coming If you wanted to add is there a final thought or anything you wanted to add or we kind of close it down We're just about to the top of the hour And so I don't want to hold it. We do have a hard stop. So yeah, I don't have anything else It's been great speaking with you and watching the fun like glowy things and chris's background Right, right. Uh, we're missing the fire today, but there's no fire today. I'm sorry. Yeah, you know So it's hot and having fire in the background doesn't really work that well, right? Like it gives me a snowstorm or something But like here's the thing anything that's bright like beach snow whatever it just doesn't show up in the background Like it's well, you could do boston snow which is kind of more of a gray color. Um, so yeah, that's a good point Right. Well, again, thank you so much for You know giving us some of your time. We know there's a lot of prs out there to review We know there's a lot of caps to review, you know, then so we really appreciate the time And uh, you know, thanks for coming Cheers. It's been great speaking with you. Awesome. Thank you. See you soon. Bye