 Nice. And welcome to the future of Linux panel here at KubeCon. My name is Vincent Batts. I'm CTO at Kinfolk and have been in the container space for a long time, so lots of familiar faces. And I will kick off the introductions with our panelists today. What's a brief question to get the conversation started? So, Maul on it, if you will, does it still make sense to run a general purpose operating system, Linux operating system in this cloud native world? Let's kick off our round of introductions. Christian, why don't you go first? Sure. So, I'm in the middle of a transition. I was at Canonical for 15 years and I'm about to join a new company in Brazil that's through an acquisition of a side business that I had. This company is a big retailer called Magazine Louisa or Magalu. They have a huge technology team and they're big users of Kubernetes have basically every single technology stack across the company. So it will be interesting to see from the inside what the user perspective is. So being at Canonical for 15 years, you probably have some opinions on general purpose Linux. Yes, I think so, yes. I had opinions before that already but my time at Canonical cemented that as well. Is the OS relevant? I think this is a big technology transition and those are always interesting and produce kind of unexpected results. The big questions I think are going to be, how will the distro vendors react to the change that cloud native brings? If platforms and distros basically have built this value of being a bridge between upstreams that release at their own cadence that develop on their own timelines and priorities to what users expect in terms of maintainability of long term support or predictability of release. How does that value translate into a world where developers are picking all the components that go into the container itself? What about the interface between kernel and the user space since you don't have a kernel inside the container? You have to use the host container. So those are the interesting questions that are there. I won't spoil for the rest of the panel, I have some of those. I think those are the big existential questions for people that are building general purpose Linux in general. Nice, nice, nice. Next up, Dusty. Why don't you introduce yourself and give us a little opinion. Hey everybody, my name is Dusty Maeb. I'm an engineer at Red Hat. That's kind of how I know Vincent. We used to work together a little bit. I started out my career even before Red Hat in the telecommunications space but then came to Red Hat and since have focused on kind of the container West space. First with Project Atomic, Atomic Host, and most recently with the acquisition of Coral West Inc. The emerging of container Linux and Atomic Host into our upstream Fedora Coral West, which is where I spend most of my time. And then also Red Hat Coral West, which is part of OpenShift. So that's kind of where I've been as far as a general purpose OS in a cloud native world. Obviously we're getting more to the point where special purpose OSes or container focused OSes seem to solve some needs that platforms have on top. But at the same time, I don't think necessarily general purpose OSes are going anywhere soon. I mean, we really leverage in this special container OSes. We leverage the ecosystem that the general purpose OSes provide. In our case, we have Fedora, we have RHEL, but there's rarely a new container OS that you see that starts completely from scratch. So there's always a role to be played there. That's fair. All right. Thank you. And Tasha, how don't you go next? Hey, I'm Tasha Drew. I'm director of product incubation at VMware in the office of the CTO. I was responsible for launching the VMware Tanzu Kubernetes Grid Service for vSphere. And I was part of the team that launched Project Pacific or vSphere with Tanzu. I'm also co-chair of the multi-tenancy working group for Kubernetes and co-chair of SIG usability. And as far as the question about operating systems, I think that there's sort of like an interesting historical separation of operations teams and application development teams that resulted in this drive towards having like this golden image that may not have anything to do with the applications being deployed on it. So I think that a lot of things have gotten baked into the operating system layer that the applications deployed on it don't actually need. And being able to have more purpose built operating systems that serve the needs of the applications running on it makes a lot of sense in terms of just minimizing your footprint, having fewer attack vectors and not shipping a lot of stuff in that lower level that you don't actually need. So I think a lot of it kind of comes down to the build tool systems that we have. I still think there's a ton of value in the people who are doing all of the backbreaking work of maintaining the base operating system components that everybody leverages and the community that works on maintaining those as well and proliferating them. So yeah, I kind of have a little bit of a mixed bag opinion. Yeah, that's a split view on, but it is interesting that even, you know, saying ops versus apps versus apps teams does kind of give two completely different like focal points of the same set of software. Darren, that probably leads pretty well into how you have operation teams versus enabling apps teams. I want you to go next. Yeah. Yeah, hi. I'm Darren Shepherd, CTO and co-founder at Rancher Labs. Rancher, we do multi cluster management Kubernetes and we've also done some container optimized OSes. Yeah, so that's my background. So regarding this question, I could probably like a monologue for the next half an hour about the answer to this question. Don't do it. Yes. So I'm going to keep it short. But does it make sense to run a general purpose OS in the cloud native world? I would say no. But it's actually significantly more nuanced than that because you know, it's like, what do you call a general purpose OS? And, you know, it's like, well, if I'm saying the answer is no, does that mean that the death of Ubuntu or Red Hat or something? No, that's not true because it's really all of the definitions of these things are changing. The assets that come from a Linux, the ecosystem or vendor are all completely still relevant, but they're just packaged and delivered in a completely different way. Because like, I don't even really like the term of container optimized Linux because the reality of when you look at these things is they almost have nothing to do with containers except for the fact that they have Docker or Podman or Kubernetes on it. It's like container optimized Linux is still a general purpose OS because it can run basically any workload. It's just how the workload is packaged and delivered. So I think there's a huge transition that we're going to see as we go into the cloud native world of like how users interact with the Linux distribution, what they're expecting from it. And I also see a lot of interesting opportunities for Kubernetes and Kubernetes distributions to effectively replace what is Linux distributions today. And sorry, that's a little off topic. I'll just say something real short and I'll stop. It's just that the Linux distribution today is really a kernel and a set of user space. Right. And so there are a bunch of user space packages that run on the kernel. Right. So when you look at Kubernetes, our view of a Kubernetes distribution is very naive right now I think is very myopic is because we have Kubernetes but then we have a whole host of applications and software that run on top of it. And like you see that from the CNCF landscape, you know, this gigantic slide of all this crap, good software sorry not crap. And so, you know, how, you know, there's there's still the fundamental need that Linux distributions always providing which is like delivering packages that run on the kernel. Now we kind of need the same thing in the cloud native space. So it's like, but those those packages that, you know, run on top of Kubernetes, they still have Linux bits inside of them they still have glipsy and they still have, you know, Java runtime and you know, so this is just all changing that that's basically I do think lots of lessons learned of software packaging in general that have been invented and reinvented in so so many ways, and we're just going to reinvent them again and the container space so Good, good, good. So then, with that, I would like you're talking about, you know, everybody's kind of mentioned like the way that these things are changing and even to some extent kind of implying that obviously there's enough players in the scene that are interested in seeing it play, you know, play out and I think one of these things that's interesting is that it's like, yeah, I don't think that any of the general purposes are going. OS is are going away and I don't think they need to. But in some ways the pie is just getting bigger. So like, what, what purpose they did is fine but the cloud native space is expanding out. And so with that you've seen, you know, different cloud providers come up with either their own, you know, optimized OS Linux OS, and then, you know, all of them are trying to compete for different kinds of managed Kubernetes offerings. Do you think that that's going to have an impact on the kind of broader market so like if we say okay there's general purposes and then there's all these derivatives but now there's like a competition for optimized derivatives and people won't have maybe what they were familiar with, but it kind of narrows down, you know, for whatever particular use case like cloud vendor optimized. What kind of impact do you think that'll have on the market. Who wants to go first on that one. Let me let me take a bridge from what Darren said I think Darren's points the main point is that while the, the companies themselves and like the core assets that they have may not change like what what gets delivered to end customers I think has to change like there's no question. And the thing is that the, the piece of the operating system that runs underneath the container in Kubernetes, that is going to that is, although the kernel is really relevant it's so small. There are components there is so little that there is like actual value in the cloud provider whoever's preparing the whole stack, giving optimizing that entire layer. So I think that's something which is absolutely a, a likely trend to come that they'll optimize this deliverable which is there to say like, how much is that actually Linux you know okay it's Linux from the kernel perspective what else is there like there's not a lot there. That's just serves enough scaffolding for you to boot the containers off. There's one wild card on that though which is that as you move into enterprise, everyone says, Oh, hold on, but I need to run my asset tracking or I need to run like my, my IDS pieces on this layer as well so I think that's the only piece where I'm not sure what the future will look for that but that something specific to be delivered for that that small, atomically, updatable, safe to basically roll back and forward I think that's definitely coming. The question is this thing like what do you do, like when somebody says oh I need to put put the BMC agent on this and people are like, well, you can install any software on this layer like, yeah, what any packages and what do I do are probably even more scary than that. Like, somebody's looking at a checklist and they say well it says here I have to deploy to X. Is this X. Well, no, it's a better derivative, you know, yes. That question doesn't compute in the context of where it's landing. Yeah, like ISV, ISV certification is a big deal too. You know, it's like a lot of people are like I'd have to run this OS because that's what you know this vendor has certified on, which completely changes when, you know, as vendors are starting to package, you know, their third party software on top of Kubernetes. What does it now mean to certify, you know, and that's kind of interesting space. I mean, a question for you there and on that like, can you run a workload certified for SUSE on a non SUSE kernel underneath the container. Yeah, and even even even dusty could wait into that one a little bit. Yeah, and well, and what does it mean, you know, is there even honestly is there even such thing as can I certify for Kubernetes in general, you know, or am I actually certifying for EKS or, or, you know, like VMware suite or whatever open shift. You know, those are interesting questions. So, but going back to because I already forgot Vince, like, what was the original question? No, this is good. It was it was more like, is it a useful trend that we see these kind of optimized derivatives because there's a few, you know, players in the scene and you know, like you've even had experience with having kind of an optimized Yeah, that's right. Focus. But is it a useful useful trend or is it well, I mean, so I just impact on the market. No, I said, well, I mean, it's disruptive for sure. Like, so there's there's like the bottle rocket and like Google's container optimized. You know, Microsoft, they haven't announced, I don't think a Linux, you know, container distro or whatever but I imagine one would come at some point. Honestly, I think it makes total sense for these cloud providers because it's like, what's the touch point of the customer. It's like the customer, or the end user, they really want Kubernetes. So why do they need to care about the nodes and what's running on the nodes the interface is Kubernetes. And this is why like, it's tricky, especially if you're a Linux vendor of like, Well, what does this mean now because effectively that Linux layer is the kernel itself or the container Linux, whatever is, it's fairly commoditized for most, you know, it's like people don't really want to pay for it that much. And, and they since they don't interact directly with it, it's very hard to differentiate because you know it's like Ubuntu for example like their, you know, rise to fame or whatever it's all about kind of users interacting with it and join it. And if I'm not touching the node OS anymore like, so it's like the only way to differentiate, even if there's, I would say maybe there's not even a need to differentiate anymore because you know if you just say it's commodity. But the only way to differentiate is through capabilities. And that's where I still see OS is being somewhat important right now is especially let's say like an edge space where I've been working a lot recently where it's like GPUs and devices and, and all these things don't matter because they very much touch the OS. So it's like well your OS is not capable, you know because it doesn't have these whatever proprietary drivers and that kind of leads towards like you know if you need BMC your, you know, it's like, it's the capabilities not so much that the, the actually interacting and using it, I guess. Yeah, Tasha, Tasha do you or dusty you want to write into that one. I mean, as far as, you know, different cloud providers coming up with their own distros. Is that a useful trend. I think you can go both ways right. It's useful because anytime you bring new ideas to the market. There's things that can be learned. But at the same time, it also increases fragmentation a little bit like there's now more options and if you happen to be rolling, you know, trying to set up Kubernetes yourself. You know, here's a new option to consider which is options are good but they also might confuse people or lead them down the wrong path at some point. I'm not sure. But I think Darren had a good point which is it depends on who you are right. Are you just clicking a button and having the entire cluster set up for you and it's completely managed. Not, you know, you're not managing the cluster if that's the case. Then, you know, the role of the distro does have a lot less of a role at least as that's your concern with right. The OS is still very important and we see that every time there's a new security issue that comes out right. But who cares who's responsible for it right is is the big part and we've seen the the the lines move more and more towards you know more as being taken care of by something that you outsource you pay for it. It's a cloud platform and you care just about your app right and that service that you're providing to your customers or to your users that is giving them value. And so, you know, Darren is right. If you are a cloud platform and you're offering that abstraction to your users and your users don't care, then it's probably a useful trend for them. If you happen to somebody, a big company or whatnot, who where you are responsible for that platform maybe you're outsourcing the infrastructure level stuff but not the platform as a service the Kubernetes level stuff. And you also want to be able to take that to other cloud platforms and not be locked into one necessarily. And then you do care about the OS still right you're still making that choice and you want to choose your platform based on that. So, it depends. It's kind of like as we move into this cloud native world it's all kind of reducing the burden on the the operator or whatever even if you're running yourself you know data center or whatever you know it's you know I think you know in the early days and now Fedora core OS really kind of pioneered this this kind of a self driving car kind of mentality of, you know, it just you treat it more like a phone or whatever just updates and you know reboot so whatever it's so it's all about reducing the maintenance, which, which a lot of that also is about reducing the variability in the OS. So, like if you say like the trend is going towards reducing the maintenance and the variability. That also means that like, I think fragmentation doesn't get as bad you know it's like, you know so much thing. It's like it's the thing is that a tiny thing that runs underneath underneath the container the line of the containers underneath that what is there. And I think one of the interesting things that's happened is the the LTS kernels, you know from upstream is kind of been I think that's been a big deal that like, there's actually LTS kernels upstream. So, that's allowing more distributions and things like that to just say kind of piggyback off of that, and that kind of keeps the whole industry in sync which is really quite nice. Because it's, you know, if you're, if you're driving you know Debbie and is coming up off of that or whatever they're trying and what's trying to kind of align to those LTS schedules, then it also reduces kind of the fragmentation. Because at the end of the day that the the biggest thing you care about from the Linux, you know from the container side is effectively the, you know, the Linux API and API, you know as long as that. Yeah. So, Tasha, you were starting to jump in your industry had kind of a race condition there. Is it a useful trend or I mean like it is we've almost touched on another thing of like the case you Kubernetes users even care what's running underneath. I mean, yeah, in an ideal world they wouldn't have to but like, as we were saying earlier when you start getting into what software what your software is certified to run on from a container and host perspective like you. So in a nice fee perspective, you really need to like minimize the possible number of combinatorial, you know, combat, like the matrix like like, let's make this achievable like to actually say we support our software. So yeah, I would say that it becomes just a supportability problem for the ISDs versus as an application developer do I want to care I really don't write like I kind of just want to care about the exact changes I'm making to my application and have it be as atomic as I always end up relying on something that's at a lower level that ends up making things complicated from the cloud vendors perspectives I think releasing their own Linux distributions makes a lot of sense just from cost reduction and optimizing for their exact use case. They have a more limited number of hardware that they need to support so they don't need like something that can really run on the huge swath of things you find in on prem data centers. They don't want to necessarily pay a vendor every time they spin up Lennox, you know they want to keep all of the money from the customer instead of just a smaller amount so yeah I get it. I don't know that it really translates to on prem use cases where people are really consolidating their entire security posture around a limited number of Lennox and Windows operating systems. You know, whether we agree with it or not they want to run their virus scanner against every single node you know so you just start running into like the exact use cases and like that security checklist so yeah. Yeah, gosh that security checklist so that's that's probably pretty perfect segue. Are there different kinds of security considerations that you'd have like for Lennox in general, when you specifically running Kubernetes so like in this, like, we, you know we said cloud native line large that's meaning like schedulable workloads that can be put on the cluster somewhere like so what kind of you know what kind of security or different security considerations are there for Lennox when scheduling it on a Kubernetes node Kubernetes node. So jump into that one. I want to just highlight something that is important to note as well like from an OS vendor perspective like there to that there's operating system beneath the line that runs underneath the containers, but then, like, the containers are not built out of thin air like there are pieces in those containers that come from the like usually that the large part of that container will come from the operating system like yeah, and I think that's something which from a security perspective is the very complicated unresolved kind of transition to Kubernetes because even if you statically link you still had to have a build route. Well, well, the thing is that I think the link already like makes everybody else look at you and saying okay statically link. What's done in the 90s. Yeah, what happened to the components that went into that statically linked thing like like okay like where do they come from and so now you've got this huge provenance and attestation problem that the distribution is very neatly solved for you because we have to design every single binary that comes out there and so you know this binary is built by a robot inside a system that in fact I was part of building a canonical you know that that robot built that binary. You can go to canonical and say that binary there has a it was compromised. So you have that perfect chain inside a single distributor. Now you're saying, guys, do whatever you want, you know, go to GitHub, find this thing like get cloned away or even worse like npm install your way into this mess. And now like, like that is the actual security problem that distributions ironically are very well set up to solve because again the DNA distribution is to serve as a bridge between the madness which is upstream and the predictability which end users expect right. And I use npm as the example because it's like the the the lsd edition of that madness right so there was a there was a great tweet here recently of like you know this that way air it's okay ignore it and wait you know introduce this many new packages this many vulnerabilities please donate to the The first 10 minutes of anybody using npm was like if you're like for like an old school person is like what the hell did I just do you know like this thing imported this dependency which like 10 lines. What the hell is that so anyway, but the point is that that madness has to be resolved some way, right, they're going to like somebody is going to have to give to give these people that are developing on the side, a set of components that they can use safely, you're saying statically linking that's cool but somebody has to be able to press a button and see ICD and say hey, bring in the new versions of these things. Oh, but they can't be API breaking otherwise the app breaks, you know, like this this sort of things which distributions are very known for like we don't break the the applications if I break your app, I apologize, file a bug, I'll revert back that that that breaking change in the API because I keep stability for however long I promise to maintain distribution like that has to be there's kind of an academic world like otherwise like otherwise it's like it is proper dystopia right otherwise like you can't use anything I can't use my phone, I can't use the computer because everything is hacked across the stack. So, so that to continue to derail it because I want to bring up a couple points you touch and we only have like five minutes so get it. Oh shoot. Well no but I think this is really really well whatever it's interesting because I'm saying it but Yeah, you're biased. I see. See the um but no if you look because I you know before I was saying it's like I think the role of the Linux distribution is clearly changing because all of those assets like let's say like your rpms your Debian packages or whatever. Like those are all valuable and they're well curated that's an extremely difficult thing. But what's different. What's happening now is like the kind of traditional the way that things were before is you would install this generic thing and then you would kind of mutate it into what it was going to be. And you know that's the way puppet chef configuration management solve all those things work you know, Ansible is it's kind of you form it, you know it's like this clay that you form into what it's supposed to be once it gets there. And we're shifting all of that earlier into the pipeline and so it's now basically I want to take all of those assets those rpms or Debian packages, and I want to do one of two things I want to build an OS image which then turns into the mutable container of Linux, or I want to build a docker packet or docker docker image, which then becomes your container runtime so it's like, we're moving this kind of mutability and assembling of all this stuff earlier in the stage to produce these like reusable assets that we can more easily track. And you look at the big advantages of containers in general, a lot of it has to do with distribution management pipeline. That's where you get a lot of the value out of them. And it's like by shifting everything kind of earlier and more predictable. You know it completely change could change the game so it's like, those assets are still useful and I don't think they're going to go anywhere, but it's just presented all differently. But people have to. This sounds like that. The benefit of when people would, would solve things at compile time versus runtime. Like, yeah, yeah, exactly for it to some Python or Ruby library to explode on the edge and to lead into security is that like, I don't think necessarily the security requirements of containers change, but security and the models that we have are changing such that we can make it more secure because you know we're building like we're creating immutable assets that were deployed so we can cryptographically and make sure that the right thing is put there so it's like what I'm seeing with containers and stuff and all this kind of immutability is that we can greatly increase the level of security, because we can you know like secure boot fully verify things DM Verity verify the OS if it's mutable, the whole stack we can basically can verify everything that's going on because, you know, we're changing the way that we're packaging and delivering Linux as an OS and in the software I like that a lot. Tasha Dusty. Tasha, what do you have anything on say on that. You know, it just kind of occurs to me that like the one thing we're really not talking about is how some of these cloud focused stories into Lennox really are under investing in the wide community of people who spend a lot of time maintaining all of the various packages and capabilities that they're all leveraging. And, you know, I just start thinking about like, is Amazon donating to like open SSL right like I mean, it just kind of feels like there is like a certain lack of understanding of how many hard working people there are like throughout the world who are constantly using Linux and like keeping really important packages up to date that we then all consume and leverage. And actually like kind of one of my pitches sometimes is like when people are worried about security and enterprises, I'm like you should understand every package you're deploying and make sure that you're not relying on something that has one guy in Berkeley like desperately trying to keep up to date, like on his free time. It's like five years ago, which is what typically happened actually. Yeah, yeah, not only that the recent xkcd about one one person out in Nebraska that broke the internet or whatever but no and it is crazy because often when people hear that of like, oh you should you should actually be familiar with the code that you're importing whether it's go lang or npm. Like, and it's usually it's like the gut reaction is that it's that's such an overwhelming reaction, or you know, ask that people just laugh it off as a joke. And it's like, that is the joke. Like, you didn't own it. Therefore, good luck. Have fun. Dusty, do you have any comments on that one. Not, not specifically to what Tasha just mentioned. I mean, obviously, yeah, I mean, there are so many worked in the fedora community as well. So that's not a small community either. No, not not at all. And there's so many people that need many more things for every, everything they do every day in the fedora community in the Debian community and the Ubuntu community and the gen two community everybody. It really is, you know, many different people all over the world, many different companies all over the world coming together. And, you know, I think I think some of the cloud providers are getting to a point where they realize that if they can't just survive forever without heavily investing in this part that they they started to build up. So, yeah, I think it's starting to get a little better but I'm just one person. I mean, in the in the cloud providers, you know, kind of, you know, it's like open source kind of started with this ideal, you know, it's kind of like the random person contributing and in their free time doing things. And there's still a lot of that, of course, but it's been tricky over the years has as open sources also become the business model of so many companies is is people they're paying people to work on open source. There's a lot of people who get paid don't work on open source versus the people who kind of do it in their free time. And so it's very difficult to balance that because a lot of people are making a decent living working on open source and there's some people who are kind of, you know, giving up their time for free and they really, you know, kind of really shouldn't or maybe they don't want to. And it's difficult but I think the cloud vendors have definitely like recognize this and if you know you look at all, all three like the big three of Amazon Microsoft and Google. All of them over the last couple of years have had major pushes in the open source space. Yeah, so it's like, you know, I think things are changing, changing for the better, but there's always there's always more I mean I see. I always, I feel so bad for open source maintainers burning out. You know you just see it so often. Yeah, it's one of those big things that I have to remind people often of like, even if you work for a company, you know, it's still your name on the line when you put up a PR or LGT and something. Yeah, don't forget that it's still you as a human. Yeah, yeah. So it's a tricky one. I mean I don't have any answers or anything. Yeah, it's something. Well good. Well, and wrapping up, I mean, it's, it's pretty interesting. I think we've covered how the operating system, whether it's general purpose or container optimized is still kind of the glue between different expectations, you know, kind of working with the upstream, having their own cadences, working with the different companies involved, and then, by and large, you know, we're reusing a lot of these. The work done and the kind of like infrastructure and methodology, whether it is for kind of an operations or an application persons piece of it where you whether you own the whole stack yourself or you wanting to pay somebody for it. And either one of those cases, most people are wanting to push for something that's more supportable something that's more predictable. So I think that's probably why anybody could argue the case for both general purpose or something container optimized is because they want it to be supportable for their use case so Karen, Christian and Dusty, I appreciate you all for your time. And if there's nothing else and we can probably open it up for a few questions.