 Good morning. Good afternoon. Good evening. Welcome to another edition of the level up hour here on open shift TV I am Chris short executive producer of open shift TV and we're joined by a very special guest today But I will hand it over to the illustrious Langdon White to tell us what's going on Hi everybody and Yeah, so today we as you can see on the screen. We have a very special guest is Chris Wright he is the chief technical officer of Red Hat and And knows some technology stuff and in particular knows a bunch of stuff about both Linux and Kubernetes In fact, I think your Twitter handle is Colonel C. Dubs. So, you know, it's kind of like he's kind of Colonel guy knows a little bit of stuff And so we brought him on the show But we'll talk about kind of what we're like first up We wanted to talk about is that Red Hat summit is coming up and yes, JP Dade I also need more coffee as you can already tell but Coming up Red Hat summit is next week And so we have that and then immediately following that we have KubeCon EU And so we did our little contest for KubeCon EU and I think we've identified some people who we want to give out some free tickets So we're pretty excited about that You know as we have said on the prior shows we have given all of you and everyone else free tickets to Red Hat summit So, you know be sure to attend because it happens to just be free So there's but there's lots of good stuff coming up in the one that is coming up next week Whereas kind of like the way I've been kind of thinking about is like it's focused on like keynotes And so it's kind of all the big, you know Big demos big cool stuff that we're launching is all coming out kind of next week And then the later one is kind of the more deep divey sessions And then we should be doing one more kind of series that may may with any luck be able to be in person Where we're gonna do more like workshoppy lab stuff and we're delaying that so long to try to make it in person Because obviously workshops are a lot better if you can do it in person So Chris I wanted to ask you about or talk to you about is like You know, what are you looking forward to? You know next week in particular, you know, like you're talking So what are you gonna be talking about? Can you give us a sneak peek? Yeah, like how do I keep from stealing my own thunder? Yeah, so Next week as you described it It's a little bit more of the the keynote type of format a little bit more high-level stuff Trying to to give a sense of all the things that are going on that are red hat relevant to red hat and and both in the technology space and Even how we deliver technology So some of the things that I'm talking about Red hats focused on the hybrid cloud. We think a lot about the open hybrid cloud and where open source creates the you know, the sort of Infrastructure support and capabilities for building applications and delivering them all across the the open hybrid cloud What's really interesting to me right now is that notion of hybrid cloud It's not new. It's been around for a while. We've been Thinking about it and and sort of advocating for it and supporting our customers who are living that as a daily reality for quite some time Today and maybe we were kind of the early ones and and the lone voice for a while Today you find similar types of concepts from public cloud providers And and other software vendors across the industry That mostly focuses on your cloud footprint and your on-premises footprint or maybe your clouds footprint because many larger businesses have relationships with multiple clouds the the buzzword I can't say de jure. It's due the last couple years Not my friend. Yeah, French isn't strong enough. Yeah Is is about edge and edge compute and so when you take that notion of hybrid cloud and say well you're gonna have applications that run in different locations and and the consistency of Your operational experience the consistency of your development targets really helps build efficiency for for the business in general That that is a lot of what I'm talking about how we extend hybrid cloud out to the edge and think about the edge as a location for compute you'll see like kind of headlines in the news talking about You know the clouds dead everything's moving to the edge and whatever clickbait type of title, but the Notion that compute is expanding out to this new location and the purpose for Expanding compute to that new location is connecting compute to data either a producer or a consumer of data and You know, we've known for for quite some time the value that data can bring It's you you can mine it for information and use that information to change how you run your business or change how you respond to particular events I I think that's really interesting and then when you when you throw in another Kind of hyped up technology buzzword. I live in a buzzword compliant world. So just case you're wondering Ringo cards out folks. That's right. That's right 5g is also not brand new, but it's gone from people are just talking about it and anticipating it to early stage rollouts of new infrastructure to support The next generation mobile network. Yeah, I got it. I've got a 5g phone here. Yeah questionably 5g with my Quarter of a bar this location. Yeah in my in my house in inner city Boston. It's yeah, it's pretty But that that connection of 5g, which is Just a way to connect devices to networks in in the end I mean, it's it certainly has some specific characteristics that are interesting But when you when you bring together two different trends in the industry You you really unlock a whole new set of opportunities that don't make sense If just if you're just looking at it from one perspective or the other And so a lot of that is is what I'm talking about it at at the Red Hat Summit and it's certainly an area that I'm super interested in Because I I like to think about how that how that impacts our our world our daily lives I mean, you think you go back 10 years and 4g was new and Cloud was basically new and smartphones were new and Mm-hmm and now look at it a day We take those things for granted in ways we couldn't imagine 10 years ago Yeah, so we're at the cusp of building that what we're gonna take for granted in 10 years We're building those infrastructure building blocks today, and it's really fun It's just sort of to try to imagine what what what like what will life be like We're already seeing it here in Detroit, right? So Detroit if you don't know folks is where I'm from and we have what we're calling a connected corridor So that we're building out from downtown Detroit. We're all the major, you know Big three motor companies here in the US are all the way out to the research end in Ann Arbor at the University of Michigan so that's a very long stretch of quote connected highway where Autonomous vehicles are gonna be tested autonomous sidewalks, right? Like that's a new term to me, right? Like allowing traffic flow and it knows who and what is where kind of thing based off sensors in the area That is the kind of stuff of the future that we're seeing popping up now here in Detroit So I can't wait till that is the standard in 10 years to be honest with you Yeah, I want to I want to continue this conversation a little bit But I want to back up a half a second and just be like could you define edge for us Because I think that's one of those it is a very buzzwordy term and The reason I'm asking for the definition is not so much because the definition but because I want to know What do you think of the aspects in that definition are the important bits or what are the hard bits? about what we call edge Well It's not easy to define the edge succinctly and unless you give a very abstract definition But the way the way I usually describe the edge is it's not it's not one thing and The edge it's it's sort of a matter of perspective and I I usually draw an analogy to to an onion at any At any layer you sit within an onion. There's an adjacent edge or a next layer. And so if you think about Maybe that that sort of mental picture of a sphere in the center of the sphere you could you could place something like a data center now oddly enough From a public cloud provider Putting a piece of their equipment in that data center looks like the edge of the cloud to the cloud provider But it's in the data center from a data center perspective. So again, that's why the perspective matters and as you kind of Move out from that core data center There's different hops in in the network essentially that are locations where you could you can do computing And so there's there's core locations in that fans out to a slightly more distributed set of regional locations and then eventually you get to things like cell towers and and an actual Devices and devices could be as as sophisticated as a smartphone Which is you know, it's a powerful computer to something really tiny like a sensor that might have a fixed purpose and So that the edge is across that entire spectrum again, it's really important what the What your perspective is the the the probably the most important part about what is the edge? it's the the the latency relationship to compute to data and Obviously it within a data center You know across a few racks with with fiber connecting all of your servers You've got a low latency connection, and I wouldn't really call that the edge But the you know the places where data is being produced or consumed and how you bring compute to that location as you leave The core data center or leave the public cloud Those are those are different areas that we would define as the edge and if you in in some Industries like the telecommunications industry. There's a more specific and detailed definition Which includes something called a provider edge So the edge of the network that the network is a service provider for so that's the edge of their network and then the Customer premises side Which is the first hop into the customer's Infrastructure and so from again from service provider you might think of like in your home If you have a Wi-Fi router that's terminating a cable or something for your for your internet service provider That box is called a customer premises edge device And so again, it's all it's all perspective but you know and today we have Open source code Coming from red hat running in the international space station So that like that's as far out edge as you're gonna get when you think about the earth I mean we got the we got the Mars rover now, so Helicopters out there. Yeah pushing the boundaries of the edge, right? Yeah, I think actually the data point is particularly you know interesting especially because like Sure, you know 20 years ago the data sizes were were also kind of non-trivial when you talk about the bandwidth We had and stuff like that, but now they're ridiculously large, right? Like I mean, you know when we're talking about doing like machine learning or whatever, you know Dropping a terabyte of data on something for a problem is like whatever that sounds normal And you know, it's still still kind of blows my mind a bit, right? That, you know terabytes of data is getting thrown around I mean I have terabyte hard drives you're crying out loud and I remember buying my my 500 meg, you know big spend hard Yeah, the So I think I think that's a the really good point is like trying to make sure that you have the appropriate compute right near the appropriate data Is a at least for me an interesting way to think about the edge The only thing I want to do too was pause for a second because I remembered that I forgot my super ugly slides and so Let's run through those real quick just because otherwise it wouldn't be a show So I'm going to try to do the share screen button thing while you're doing that I just want to point out it's talking about edge and data and big movie I've read an article last night where an airbus 350 Generates anywhere between 800 and 900 megabytes of data per flight Yeah, right like just sensor data, right like parts failures that kinds of things there that data is being analyzed And that is just fascinating to me that wow one flight can generate almost a gig of data That's that's impressive right like what can you do with that to optimize your fleet of hundreds of air blind? It's actually really really interesting because flights are There's there's a safety consideration. There's a business efficiency consideration and When you get especially to to longer distance flights Very subtle things make big differences. Um, for example Fleets traveling across the Pacific from the us to uh To australia There's there's lots of historic data and different Different planes even if this even if it's the same kind of high-level moniker Could be generated at different times And they're not all identical and just The chemical compound in the paint can change the weight of the Of the plane and it can impact Which routes are accessible because of how much fuel they can fill into the wings and so It's nuts. It's pretty subtle. Of course, they're they're also collecting not just flight data But weather data because you're looking at Where there's likely to be storms and there's seasonal Weather patterns that mean it's more means it's more efficient to fly one route versus the other route And that that's all you know, those are just airplanes cars produce You'll you'll hear anywhere from A 40 plus terabytes a day in in data with in a vehicle. Um, wow and So yes, as langen said There was always data there was You know, I've I've dialed up and wondered if I'd ever get that floppy just downloaded But the the data rates are so phenomenal and when when you think about um The richness of video data and the different ways that you can apply computer vision to video data to to discern something People use video data when it there might have been an easier sensor to give you Pointed information But with a video you can collect a whole setup, you know A whole bunch of information all at once and it's obviously, you know It's high high definition and a lot of a lot of data associated with a video feed. So It's yeah, it's really interesting. We should we should really have the stat on hand of how much video data we're using right now In fact, but you know consider especially seeing was we're re-streaming across multiple I mean, I think part of that tendency too is also interesting because you know, there's 600 megs Okay, all right so the um It's also interesting like Uh somewhere, I don't know five 10 years ago or so, um, you know, somebody got it into their head that like We we have enough hard drive space to basically store all the things that we can discover We don't know what questions we want to ask yet But let's make sure we grab all of the data so that we can ask whatever question we want to later And there's a lot of really cool factor to that. There's also some questionable ethical factors to that But you know for the sake of kind of this show, right the um It really it really just blows out the data problem But also, you know gives us all kinds of really interesting new questions we can ask And techniques we can use right so like, you know machine learning techniques or whatever To try to Extract you know extract knowledge from all that data, you know, yeah Yeah, awesome. All right, let's switch your slides before my my awesome slide So this is the level up hour where we talk about Kind of like why containers are interesting and why you might want to start using containers In your everyday, you know, kind of life You know a little bit focused towards administration or administrators, but also for developers as well Um, you know, what I kind of comment on is like I use containers all the time now Basically anytime I do anything it actually really annoys me if I have to like install something locally So and then kind of eventually what happens is as you start to use containers more and more You start to realize that now you need to orchestrate them And so that's where kubernetes and the open shift kind of come in Is that you want to well when we've even covered on the show doing limited orchestration in like podman And then kind of as you scale up, right you go to like kubernetes, whatever for for more orchestration because they're so They're they're difficult to manage Not because they're difficult to manage, but because you tend to have a plethora of them really really fast And if you've ever managed, you know, 100, you know, like virtual machines, you know how hard that is And so the same thing happens with containers except even faster, right? So that's what we talk about on the show And this is our first slide and this is our always the same second slide I do hear from from our brand team at red hat that we will have Brand new fancy fancy colors and graphics any day now. So As I think I've said once or twice before So I'm Langdon on twitter with a one and chris short is chris short on twitter with two s's And as I mentioned earlier in the show chris is actually colonel c dub Is it c dub or dubs? I can't remember Singular singular It's like my fingers can type it, but I can't necessarily say it, right? And then you can also chat with us on our discord as norenda have asked this morning. Yes, we are having a show today however Just fair warning because of summit and kubecon. We will not be having shows next week or the week after But then we will be back with I want to say container health index. I think the week after that I can't remember now. I should have I mean, I usually look it up beforehand, but I forgot so I do not have unfortunately show notes from last time Just yet. Unfortunately. I ran into some challenges yesterday on the personal life And so just kind of ran out of time But today we are talking about control planes and red hat summit And we're going to kind of get to that right now But i'm not going to show the next slide because that's where we give away the points and Chris when we get to that we will explain the points If you haven't heard about them before so Basically what Part of the reason we invited you on the show today was because red hat summit is next week but then also You did a show with Kelsey hightower You know, I think you recorded a while ago, but then it kind of you know goes through the video editing process and all that just And it's actually being released today on a series that we have that is like short segment quasi deep tech stuff called technically speaking And that episode's coming out today. We had a little teaser for it That we asked for because we decided that seven minutes was Or whatever it is seven eight minutes was too long So we want to just do the highlights But then what we want to do is we're going to show that real quick and then Basically talk about the the concept kind of a little bit more in detail So chris, uh, sorry chris short. Uh, yes, you want to run our little video? Absolutely hang on tight folks The whole point of this is that kubernetes by itself Isn't all that interesting It's only what we do with it and there's a few stories that come out of the box and you can read those but they get boring Pretty fast But then if you look at it as a blank sheet of paper and you're willing to take all your expertise So if you're a network administrator, maybe you go make a c and i plug in To take everything you know about networking and just abstract it away So that a developer who needs an ip address or to live on the right network segment Doesn't really have to think about it If you're in operations and you know how to roll out software safely over multiple regions You can make a controller for that. You can name it the multi region deployment controller and take all of your expertise Serialize it into that thing and share with other people And lastly if you harness the power of open source That you can get a whole community of people to do the same thing and then share that knowledge and propagate it through These systems and we just happen to call it kubernetes I'm seeing a new meme. We had there's an app for that now. It's There's a control plane for that All right, folks. That was the uh sneak peek there. What'd you think let us know on chat Definitely, uh, I will link up where you can find more of that um Yeah, so, uh, you know Obviously, um, we had a little bit of a chance to talk about this before the episode today so I kind of wanted to kind of walk into this is that um One of the things that came up in that discussion is kind of this idea of kubernetes as a control plane and You know as uh as we joke around about on the show sometimes You know, not everybody watches this show has a computer science background So could you give us a definition of what a control plane is? You know and actually that that would really even be from an engineering background You know, we've kind of co-opted the term in uh in the software world But uh, how how would you define it? And I I don't think we really care if it's a dictionary definition as much as to say This is what we mean in this context. You know, yeah It is a little kind of a nebulous term although it's used Really pretty consistently especially in certain certain subject areas I I would say there is When you when you think about applications or or you know computer systems, there's there's two basic worlds One is the world that's doing the actual stuff. Um, sometimes we call that the user plane Sometimes we call that the data plane, you know, whatever. It's the place where you're Your application is running and doing application-y things business logic Manipulating data as it's flowing through the system, etc You also have something that decides When where and how things get done. Um, and that's more the control plane. So um, you know, I I The some of the orchestration Conversation bit around what kubernetes is for containers the podman to kube kind of Bit that you were highlighting earlier That orchestration is is a a form of control. It's and that's what we would call control plane activity. Um, in the uh In in the setup of a of a networking connection You have some oversight That's ensuring that you connect the two ends and then the data flows through the system Um, you know, so there's a control plane element of how do you set the whole thing up? Maybe that's Authentication to the system Maybe setting up a handshake to do encryption and then you let the data flow. So Another example of just how there's a there's a control plane element that's controlling what's happening And then there's a user plane or data plane that's doing the actual low level work and kubernetes combines the two so In the in the show with kelsey. We're kind of deconstructing that back to what what are the differences and Almost always we think about the user plane. Um, it's most of what we spend our time on It's writing applications. The applications do the application logic But it turns out Managing things controlling things is hard. Um, I mean, it's it's why kubernetes exists It's it's a it's a real challenge to build especially when you think about distributed systems. How do you manage orchestrate control? A large-scale distributed system Kubernetes does that and it's also it gets even I think it's even harder because uh control planes at least in my experience Right. They also tend to be like fractally In that, you know, it's like kind of no matter how deep you go. There's another one. Um, you know, and and so because of that you know It's kind of like it's like recursion, but the opposite in the sense that you know, it's like you have You know, you you have to orchestrate or control or manage or whatever at kind of all these different layers All at the same time. Um, and so I find that uh, also particularly challenging about doing Uh about doing kind of the software systems that that manage those things and so I guess kind of what When you were talking about it with uh, kelsey, I think the idea was that um, you know, so we have this control plane, right? Um called kubernetes, but okay like like so And yeah, so we have this thing that can that can run cloud native applications But I think the idea that you were discussing was that maybe it could be more It that was the that's the crux of the conversation and it it's fascinating I mean, it's not meant to be a plug for for the show. Although you should certainly check it out if you're interested But it's fascinating topic. Um, and it was really fun to have that conversation with kelsey Who has been, you know involved with with kubernetes for for quite some time and has a lot of great insights based on his his experience in the way Uh, you know his role at google and the work that he does on a day-to-day basis. Um that Probably the the the simple like so what? Um As I mentioned earlier, we spend most of our time thinking about the application logic. Um building building apps. Um There really isn't an app that doesn't have some kind of control controller associated with it Um Because you have to get you have to decide when to start it and when when and how and why to run it. Um it just to oversimplify it and the the The focal point for kubernetes we we recognize it as a way to deploy applications at scale in containers and Okay, so there's the orchestration of if I spin up a container. It has dependencies of it needs network It needs storage. It needs to talk to these other parts of the application um Let's do the the The control plane work behind the scenes to make all that work and we we want to make sure that it's Got a number of instances You know always so if you want it to have five and one dies you restart one and there's so there's there's um you know kubernetes is watching all of that and paying attention and and just managing the resiliency and and um Making so the declarative statement that you set out with when you said I want this application to be to be deployed this way What's interesting is kubernetes has these primitives that enables all of that and we think of kubernetes as managing containers, um, which it does but Those primitives could be used in very different environments Even environments where where there are no containers. There's no pods. There's no It's like a very different world which gets a little abstract and a little hard to imagine. Um But the what there are efforts in different industries where they spend all of their time focused on the control plane and I've given presentations to uh to teams that work on things like control plane systems for for networks and Made this observation to those teams like hey, you know, you're off there building something. Um It's pretty specific to your world It's going to be a huge amount of work You might want to leverage some tools that we've built in this toolbox that are going to make it so much easier for you to create Uh, what you're trying to do, which is intent based or declarative or you know, I want to just have the make it so button um and all of the control loops and and and sort of Attention to detail that a control plane needs to to pay to the to the user plane or data plane that it's controlling And so we are just exploring if you deconstruct kubernetes to its core primitives It has these notions of control declarative state control loops Or a controller and then a a really important piece which is extensibility So we understand well what it does for containers and pods and and all of the infrastructure associated with applications But we can extend it arbitrarily with the same rigor of how you creating apis and declaring The creating the declarative state for how you would like that system to behave And you can build arbitrary control logic to witness the current state understand your desired state and work in a forever loop Rectifying the reconciling the difference between your Current state and your desired state. You know, basically that is what a controller is doing and it's just It's fascinating because we don't think of it that way When you change your perspective and think of it differently, you suddenly a whole world of opportunity opens up So I think I mean one of the things that that occurred to me too is that um, like do you have um Like a little bit Do you have like a little bit of a challenge with having some of those conversations? Because there's like these classes of software problems or like like things you want to go build That on the face of them look really easy Um, and then you get into it and you're like, oh my god, this is impossible So like my first experience of this was writing a database Basically way back in the day. There were no in-ram databases in java. So I was like, I'll just write my own how hard could it be? Um, and yeah, it turns out it's really hard Yeah, package managers are in a similar vein. Um, and something else that seems like a really easy on the face of it software problem Orchestration I actually think is in that same bucket, right? Like writing a good orchestrator that is resilient and repetitive and always does the right thing um Is Really really difficult. Um, and so I'm curious about you know, you said you've had some of these conversations people about like kind of Leveraging something like looking like kubernetes in other kind of contexts Do they go do they kind of come back to you like well, we can just write it ourselves It's a lot easier or I mean it looks really easy. Um, and how would you explain the that conversation? uh, well The conversations are a little bit like what you described, but there's also another piece that's important, which is there's there's some sense of Sunk cost like hey, we've already done. We've already got this far Uh, this is some cost policy. Why do we need to switch tools? Right. Well, you know, and that's that's a fair conversation and and it's just just because something can do something doesn't mean it should Right. You you got to weigh all the all the balances of you know risk reward, etc um, but the so my experience has been mixed so in some cases it's just cute idea Clever abstract theory never going to apply it. Thanks for the conversation chris. Um, and other cases it's it's eye-opening like wow that's actually Solves a real problem that we have that we are struggling with and now we have a tool to to solve that and Distributed systems are hard. I mean, there's a reason that there are uh Academic programs that have been researching distributed systems for decades, right? They're straight up phd's in just distributed systems They're still trying to figure it out. Um, those those are really complex systems um, but if we kind of bring it back to something that's less abstract and very practical um a and uh, you know sort of Over simplify not for any other reason that's easier for me to explain that way. Um, you think about a a linux system when you when you start an application we had uh, a few different generations of of You know system d and and upstart and and traditional Like sys five in it systems There's there's the application itself And then there's some custom logic that in the old days was a shell script and you know Now we've got service definitions and service files and things like that. Um, but that's custom logic explaining how you launch that application and kubernetes Has that same capability? Um already and we use it every day. Um, we use the like the underlying principles are Things called crds, which would be a custom resource definition Which is essentially an extensible way to to build a schema that describes The state of the system that you care about and then a custom controller associated with that crd Now you have a kubernetes resource. It looks feels behaves like a kubernetes resource, but it has a this custom controller We build the operator framework on top of that infrastructure And now you have an operator which is code that can run an application In a in a specific way including understanding lifecycle management Maybe backup restore the kind of auto scaling You're responding to metrics that are specific to the application of looks like i'm you know, sort of not keeping up with my Slo and so I need to add extra instances because throughput, you know peak load or whatever That's already a control plane activity and that's work that we do already and the thing that I find kind of interesting and and actually Exciting and it's a different take on on the world is um that that control plane activity just managing a single application like that An operator to to run a database and now your database feels like a database as a service running on kubernetes You're tapping into the The intelligence of a human being an sre that knows how to set up and run a database And you're codifying that intelligence in in in code to run the system And it's just huge like, you know, right like I don't think people appreciate like right there. It's a big deal and then But wait, there's more and then You could take multiple sre's from different perspectives Combine their experiences and expand and improve that same operator Which is code so now you're bringing the an open source community aspect and code development aspect to Operations and how do you build and manage systems? which mostly open-source software projects are about features functions capabilities and like Exercise left for the reader. How do you actually run that thing? and and as you mentioned you've got container sprawl or whatever you want to call like it gets more and more complicated even though The primitive is basic and easy to understand it just gets complicated because you have so many things interconnected and just the sheer volume so you have to have code and automation to manage that complexity and to me, that's why it's so interesting that there's this whole world that's um opening up that's about community development collaboration making operations really A first-class citizen in the world Which is also including the control plane systems that are what operations teams depend on to run and operate Infrastructure and platforms and systems and and all that so it's really interesting I mean because what I I never really thought about this before but it's um one of the things that I would regularly talk about when talking about like modularity and app streams um was basically being able to or even like some of the earlier things that we did that were similar is like one of the things that's really nice about rpm Deb to right just package managers in general is that um you can encapsulate There's some expertise encapsulated in the packaging format That you are benefiting benefiting from The problem I was highlighting with like modularity was that it's it's not very discoverable So, you know, if you want to run an Apache web server for example on rel You just you know yum install HGVD, right However, if you want to host your html pages out of people's home directories You have to go and set up like public html directories and all this stuff In fact, there is actually a package that does that That will set it all up for you based on you know, the guy whose name is Joe Orton Who sits in England who's like an Apache expert? Who knows how to set it up? And but knowing what that's called is very very difficult And so It's kind of like a smattering of those operations that you're talking about right because What what's so difficult about our operational environment? A lot of the times these days right is like, you know, we're all building on the shoulders of giants these days, right? So as a result, we have to run all of these relatively complex systems before we even get to our code, right? You know like databases and web servers and you know caching servers and cell termination and you know every other thing and so Like kind of like rpm encapsulating That expertise from somebody who knows what they're doing And then what I think is particularly interesting, which I hadn't thought about it either before Is that idea of there's now now can be a community around that expertise and and actually have the arguments and the fights and the whatever about the best way to do it Um And then you know, but then actually I can just take advantage of it, right? I don't need to actually understand how to run a maria db Well, right? I can you know, I can run one on my laptop, but it's it's not going to be robust, right? Um, you know, so I think that's really interesting and I think that's um, I don't think operators get enough of that credit. Um, Because you know, maybe it's partially just because they're so new, you know, um but Let's uh, I guess what I kind of wonder then is like, okay, so here is this brand new concept really. Um, that has been enabled by the extensibility flexibility, whatever you want to call it of kubernetes Um, it because it's a control plane that allows for other control planes What do you foresee other things that it could do like that or what kinds of things? What kinds of problems do you think could be solved this way? Well, there's some self-serving stuff that we could focus on which is, um If you think about The primary focal point for kubernetes is is running applications across the cube cluster uh How do you run? Collections of cube clusters Like so it's self-serving in the sense it's very kubernetes centric still it's not like branching out into completely different worlds. Um So, you know really really easy example, um or something that i'm curious about is uh, take Take a a system that's a kind of a a complex distributed system. That's not about only containers running on kubernetes. Um, so maybe edge The there's a set of Devices that run off cluster You can You can those devices could show up as kubernetes resources and you could manage and control those devices And you could even use something like a type of controller that's built around uh an ansible runtime so that you're You're sort of control plane loop is is a an ansible playbook So you don't have to invent something entirely new or write something completely in go for example um and You know now you could create a system That has maybe some core logic and some sort of traditional Applications running in containers and in the core cluster and then a representation Of a whole bunch of off cluster resources that you're managing and in you know giving sort of an integrated view of your whole system from this single control plane Yeah, get keeping it sort of practical like these are things that people have to do already today uh, uh, other you know other examples might be You want to leverage just a SaaS service that It's somewhere out there. Um, you can build a resource That that's a proxy for that service and create policy and and a control plane around how you access that service as a as a custom resource and again it kind of It's a real thing applications today have to coordinate some of their work with with a SaaS service It's just it's just part of part of life more like huge amount of time How do you bring that into one framework and and make you know the the the point of doing that would be so it's easier So it's better faster cheaper whatever um There's there's a lot of just those sorts of opportunities that I think are are the the easiest ones to get started with um, and then you know the the The bigger You know kind of what abstractly could you do with the generic control plane? um control plane logic itself is is often a pattern um, and You know, you've got some notion of of resources and inventory You've got some notion of a feed of data. You've got some notion of actions that you take when based on certain events um, you know that There's there's some common patterns there that you could apply to a whole set of Of of different domains that is where it does to me get really interesting, but You know kelsey had a an idea of It's just a physical world So you you talk about the IT ot convergence, which is another buzzword where ot is operational technologies operational technologies be like where technology and the real world meet so servo motors connected to a network to to You know change the flow of a big of air or liquid through a pipe or something like that That's an operational technology You could imagine a world that's really just managing physical infrastructure But a control plane running on kubernetes. So there's no the application is like you're You're whatever your your factory But the the control plane is sitting supported in kubernetes and a very different look or view of the world From what you think about kubernetes and running pods and applications and in containers Mm-hmm So we have a question here that I think is kind of relevant Um Do you think there will be or are there already other kubernetes distribution similar to open shift by serving a different purpose in user? To which I would say there's different versions of open shift to an extent right like different Uh, you know footprints that you can use for those purposes, but Like when I think hpc Like when like the supercomputers, right? Like I don't think those are running kubernetes, right? Like or are they? Some are some aren't yeah, not not to the not pervasively, um, it's not the norm the So first to answer the question. Yes, um, I do think there will be A way to do to build kubernetes distributions that focus on particular areas And you could You could say open shift has a focus on The enterprise and enterprise applications to to make it super broad But pick a specific use case and that's that's already it's already A possibility and you already see that to a degree. Um Scaling kubernetes up or down Tailoring it to a particular form factor You know some interesting stuff at cube con around cube and home automation Very much not an enterprise world just a very different application. So yes, I do think that's a reality and um the the I liked Kelsey's description of kubernetes comes with some out of the box stories like how to run applications and But you get you know, if you think of it more of as a blank slate You get to create your own stories and that's where it gets interesting. Um, and it can do lots of other things. So absolutely When you get into a space like hpc specifically um, there's a there's a A world that's focused entirely on on scale and performance, right and the it world evolution went through a virtualization phase Which really didn't support the needs For of hpc. Yeah virtualization creates overhead, especially in the io path Um, and if you're trying to squeeze every cycle out of the system, you you don't want to Include any additional overhead Containers don't introduce that same type of overhead because the application Is confined into a namespace and resource controlled with seagrass, but in the end It's just an application running directly on the linux kernel underneath. Um, so it's a it's a better technical fit to the hpc world But then you have this whole generation of stuff that's not needed kubernetes And it's taken on, you know a life of its own. There's grid schedulers that that do very specific and specialized type of scheduling kubernetes ability to be Tailored to specific scheduling requirements is an example of how you could see these worlds coming together And then the hpc world itself is evolving To take on more and more gpu accelerated machine learning type workloads Which is already something that we focus on in the kubernetes world. So you see these these places coming together And so I think it's really interesting Yeah, yeah, I envision right like all gpu clusters at some point right because that's the use case they need Yeah, they might have a yeah, they might have a small, you know cpu to kind of Manage the muckery of the platform, but it all the work is being done on You know some nvidia gtx whatever rtx kind of card and letting it rip because yeah, I mean They just are super things right work that way already. I mean there's The the host processor is doing more than just Placing workloads because it does it is optimized for a certain type of math right and so but but that control plane logic of Place the vector processing on the gpu and the integer processing on the On the cpu is is is like again over simplified, but just an example of how you could imagine Systems evolving. So yeah, absolutely. I think it's it's a really interesting space and there's this convergence of Use cases that that draw towards okay, maybe kubernetes could be a really useful tool for hpc and maybe there's a a scheduler and A set of resource controllers that are very gpu-centric and connectivity-centric that would be the right way to tailor a cube distro for hpc For example, I haven't given it a lot of thought but yeah, good example. Yeah, thank you Cool. Was there was there more questions or uh, no, I mean that was the only one I haven't Been able to answer we've been talking about crds and operators and deployments versus Uh, deployment configs and routes versus ingress things like that, right? um, well, so I did have another just a thought I've kind of had for a while is like Is what would you say is there a strong good comparison between like the linux kernel and kubernetes? Um, because in in my mind at least right they they share a lot of similarities. Um, and you know, I kind of wonder is like You know, could the linux kernel essentially evolve into being something like kubernetes? um, or you know, I don't know vice versa is quite the right word, but like or some some some amalgamization of the two, um because like You have a lot of the same Goals in the in the two different pieces of software Uh, I'm just kind of curious if you had thoughts on that Well, I mean they they live at different layers in the stack and so there's There's probably real value in keeping some separation and allowing these two independent layers to to sort of evolve on their own But they're tightly interconnected. Um, and the the you know the The way that you leverage the underlying operating system at the cube level is Direct, I mean, it's a one-to-one mapping. You you do a little bit of setup. You create a namespace you do some resource allocation and and Set up the resource controls around those controls around those resources You pull an image you make sure it's connected to some kind of a virtual switching infrastructure and connected to any pv's that it might need the storage side And then you launch the the application as an application running on linux Um, I mean it's just all that stuff sort of disappears and now you have this little vertical slice of your own linux I was sitting underneath you and um, so are there specific things that we can do Between the two projects to advance them both absolutely and and it's part of why I mean, I don't mean to make it a commercial for for red hat But it's part of why we deliver open shift integrated because we like we deal with this every day We've got dependencies capabilities that are driven in the cube layer that create dependencies on the underlying operating system Right, right So we have to do that. There are extensibility notions in the os and extensibility notions in kubernetes And they're not the same Um, and it's less clear to me where where those intersect and then You I mean you could imagine a scenario where You want to do an intelligent scheduler for linux um intelligent air quotes meaning you're gathering data you're You're building a model and you're making scheduling decisions based on that model You probably wouldn't want to do that in the kernel Because of the complexity of running like a machine learning infrastructure in the kernel directly I think you and I have had this conversation before in fact Yeah But if you extract that out, um, it fits a general purpose machine learning workload that kubernetes would support well Um, and you you want to have a feedback loop between that that controller or that out of the kernel? sort of scheduler Hinter And the actual scheduler itself So that there's there could be real tight integration between those layers And maybe you're doing that because of all the applications you're running on kubernetes and you want to intelligently place them for resource consumption concerns I mean clouds aren't cheap and if you don't pay attention to how you consume cloud resources It can get really expensive really quickly. So you know, it's not just turn it off when you're not using it but also optimize it when you are using it and Some so just some some interesting potential examples. Um, there's also Uh networking and storage Where and observability where with a building block inside the kernel like ebpf What could you do that's connecting those different worlds? Um, observability is a critical part of Any operations and understanding big distributed systems and we have this tracing infrastructure that ebpf can support and and you know, you can tie that back to a specific IO path for for networking or storage to to give, you know instructions back to a Program that might be doing in cap de cap or for something pretty pretty straightforward as at the networking layer Right. It's funny because like, um The more, you know, I kind of think about it or the more we kind of talk about it or whatever the You know, it's kind of like I I get throwbacks to You know, uh, kind of relatively early in my career when we talked about heterogeneous grids um And that being like the the thing right is that you know, the way you could run a whole mess of workloads Was by you know, using all the desktops, you know overnight, right? You know, and then there was the the blank on the name of but the the ai Sorry, the search for aliens, uh at home. Yeah, say it home. That's it. Um, you know, or the protein folding one or whatever it's like um, what you You know, at least what i'm imagining you're describing Right in a sense like is is we want, you know, that that technology that control plane or whatever to to be sophisticated enough that It can it can really make intelligent decisions about, you know, kind of how things are being distributed Um, there was also some interesting research. Um that an intern of ours was doing at red hat But they were like doing the research at boston university Um, basically where they would look at the various providers doing serverless functions So like aws lambda or, you know, google functions or whatever They would actually price comparison them based on the workload Um, and so because you you pay differently depending on the kinds of resources you use And so they would actually pick which cloud to put it in Based on the workload that was being driven so that they could get the cheapest price You know, and obviously if it was research more that it was like something that you could fire up and and make work But you know, they had working demos It was pretty cool. Um, and so you can imagine that optimization, right? And that's and kind of going all the way back to the very beginning of this conversation If for me that's hybrid club like You know is that I don't know where this stuff's going to be running But it's going to be running somewhere and I have guarantees on its performance And to some extent I get to the point where I don't care where it's running Um, you know, which I know is difficult for most ops people, but uh, Yeah, it's terrifying on one level and it's performance. It's like Uh, there's policy compliance around where data sits and soft data sovereignty and data governance Like there's a whole set of things that could become constraints around how you build this system, but if you can Define clearly those constraints and understand how how they're, um, managed Everything that's within bounds becomes within bounds and you have a real flexible system that that still is operating within You know performance Promises and right Data compliance promises and all the things that are important. Well, I remember Gmail actually had a huge problem they wrote some articles about this is that um, they couldn't meet the I think it was the eu at the time requirements about data where data landed Because they didn't actually know Where your actual email was, uh, so, you know, I have the same experience That's a different problem Um, but uh, yeah, so I know it's super interesting. Uh, so let's transition. I think it's kind of a good pausing point We're almost out of time Uh, and I we can talk about some internet points real quick and then we can kind of wrap up the show Chris, do you have a few more minutes to stick around or do you need to jump? Yeah, how fast do you need to uh, I think I can Stick on a few more minutes. Yeah. Okay, cool. Uh, because you know, I I think this is hilarious So hopefully you will too. We'll see um, so I'm going to share the slides again. Uh, so uh on this show, uh, we, uh, give out points for, uh, kind of watching the shows and filling out some codes and You know making suggestions for the show and that kind of stuff. Uh, and so then every Time we have an episode, uh, we give out our sweet sweet internet points Which at the moment only have intrinsic value, uh, but we we uh, we're joking around about our liaison, uh, who, uh Keeps telling us the same story about where we're going to get to extrinsic value. Um But so Nurendev, uh top of the stack with 5 000 points, uh, netherland's hackum And chris just by way of explanation. We have completely made up how to pronounce people's nicknames So, you know, there's there's some backstory, you know But basically what I do is with with the forum to kind of protect some privacy I I give people a place to just say give us a name to say on air And so we have no affriction with 3,800 points. I'm sorry netherland's hackum with 4,900 points Um, and then joe fuzz who still seems to be a little m.i.a. We need to know why he's uh, they are not, uh, submitting points If you know joe fuzz region of jones, right exactly. Um, and then we have detective konan kudo Who is steadily rising through the ranks here and then bacon fork. Uh, I saw them today Yes, but we they did not collect points for the last episode. So, um, we uh, we need you to do that. Um However, I think I forgot to take my handy handy, uh note so that I could put the points Codes in so give me one second and I will do that. Um But I usually have the links pre pre allocated. Um, but that should be it So, uh, chris, thanks so much for coming. We really appreciate it. Uh, you know, I like When we can talk a little bit about, you know, like the future, um You know as well as kind of the practical of what we're doing And uh, you know, I'm really looking forward to redhead summit next week Um, I hope hopefully everyone on the show is uh, you know in the audiences And then kubecon eu despite the time zone, uh, that looks like there's going to be a bunch of Interesting stuff there. We're actually running some office hours on the channel But most of the regular shows aren't uh, going to happen during kubecon Yes, because most of those people are going to be in time zone theoretically. Yeah, exactly. So it's a little bit harder It's harder to run an iam show, uh, you know, when you're, you know, five a.m. Right, uh, so Yeah, uh, so thanks so much and uh, I don't know. Should we call it a should we call it a wrap? Well, there's one question chris if you don't mind answering Is your office always that clean? Because if you look at my office, it's not right like people want people are curious if you notice you can't see any Yeah, if I could easily move like the camera is kind of in a fixed location Move the camera you would see the mess You would see the mess. I mean I got Stuff on my desk papers random stuff like it's it's uh, yeah um I mean the honest truth is I don't do anything back there Right. I do everything up here. So up here is a mess and back there I can just kind of ignore it and I did a one time sort of pass to make it work for for presentations and No, I'm not I'm not a person who excels at cleanliness and aesthetics and you know design spaces and I've failed that long time. Yeah, thanks way perhaps Yeah No, but thank you so much chris. We really appreciate you coming on. Oh, absolutely. I enjoyed it and you know cube con plug Clayton is giving a keynote on control planes, right? Yeah. Oh cool. I should probably check it out. Yes, totally digs into some details He actually he's going to talk about a project that he started which is um looking at cube just as a control plane so coincidental to the to the conversation that I had with kelsey, um, but But shows how much interest there is kind of percolating up around different parts of the cube community. So cool stuff neat neat That is very cool awesome Well, thank you all for joining out there in the crowd and we will see you Here on the channel here in a couple. Uh, well not a little under an hour. We're going to be talking about Day two operations part two on the open shift admin hour next so stick around Uh and when in doubt subscribe to the calendar because I know time zones are hard. This one will help you Quick like and subscribe, you know, and also. Yes. Follow us everywhere. You're watching us and hit that like button, please Thank you very much Take it easy everybody. Stay safe out there The whole point of this is that kubernetes by itself The whole point of this is that kubernetes by itself isn't all