 So now we have Colleen Walters presenting you on Container Linux with REL. Thank you. Can you guys hear me okay? Yeah, looks good. So before I get started, I'd like to start my talks. Why am I here in front of you presenting on this stage? To me, I joined Red Hat because free software in particular is really important to me. When I was listening to Chris's talk this morning, just thinking about just why diversity in particular, you know, why is that important to us? Software today is just fundamental to our society. It's just a layer that underlines everything, right? And I think Richard Stallman in particular was incredibly visionary in not accepting proprietary printer drivers, right? And now just think about how fundamental the cell phone and computers are to our life, right? Free and open-source software are just fundamental, right? I also in particular just love working in the free and open-source software community, collaborating with people and companies across the world. It's really great. And I think Red Hat's a great place, great place to do that. So that's why I'm here. I want you to think about that, you know, what motivates me. But in this talk, what I hope to get you is the motivation behind a lot of fundamental changes we're making for OpenShift 4.0 and after Red Hat acquired CoreOS. What are the themes? What are the things that we're trying to change? That's what I hope you'll get out of this talk. So when I submitted the talk, it was based on one that was given at the Red Hat Summit by Brandon Phillips, CTO of CoreOS, and Ben Breard is a PM at Red Hat, works on containers. And so the original version, Brandon Phillips gives a lot of good motivation for what drove CoreOS. And Ben gives a lot of background about on the Red Hat side. So I'm going to give you some more detail today on the merger of these things. So before we look at the road ahead, we kind of have to look backwards a little bit. Like what was the road that got us here? Obviously Docker sort of encapsulated Linux containers in a way that was very compelling. Linux containers have existed for a long time, but Docker packaged it in a command line interface and tooling and added some concepts like the container images that really just struck a spark. It was a model that worked and just a level of usability that hadn't quite been captured before. And after that, where we started to see, so I helped create Red Hat Atomic Host, which is a container optimized version of Red Enterprise Linux. And when I abbreviate CL, that short for container Linux, I should have spelled it out there. And just a brief background on that. So originally it was called CoreOS, but then CoreOS the company grew to have multiple products and container Linux basically became one of... CoreOS, the operating system got renamed container Linux because they also created Tectonic, which is a distribution of Kubernetes and other things. One thing I find surprising is sometimes container Linux and Atomic Host are sort of used in the same sentence as Rancher and LinuxKit and things like that. And just it's interesting to me because they're on one very high level similar, but on the technical level very, very different. Rancher and Linux get both say, okay, well, you run containers on top of the OS. Why doesn't the OS itself... Why isn't the operating system itself assembled out of containers? And I thought about this a lot and I just... It's too fundamental a change. I struggle to see the actual value in doing that and I think Rancher is starting to switch back. But I won't dive too much into that. Just that's part of the history of these things. And obviously after Docker, Kubernetes came along. And Kubernetes, you know, there's the orchestration battle. And Kubernetes, I think everyone would agree, is sort of one for the most part, that orchestration battle. I mean, you will see people debate what should I do at the small scale. I had a customer ask me at summit, you know, I have two computers in a branch office. Should I use Kubernetes? It's interesting discussion. But obviously at the large scale, you know, we have distributions of Kubernetes. There are a lot of them. OpenShift is Red Hat's distribution of Kubernetes. Tectonic, we acquired from CoreOS. And it's actually really interesting because OpenShift had a very strong developer focus. There's a lot of added objects. So when we say OpenShift is Kubernetes, you can literally use the kube-ctl command line with it. You know, everything you saw in the previous talk. But there are additional API objects. So it has builds. It has routes, which predate ingress and things like that. Tectonic had a lot of components that were very operator focused. So the union of these things makes a lot of sense because you got all the advantages. The developer focus of OpenShift with all the operator focus of the CoreOS technologies. And that's mostly the operator. When I say operator here, I'm actually more mean system administrator. It's confusing because there's also operators, which I'll talk about later. All right. So like I mentioned, it's actually really interesting because container links and atomic hosts have a lot of architectural alignment points. Atomic hosts, we say, okay, we're going to take all the Red Enterprise Linux components, you know, the RPMs and bake them together in a different delivery format. And container links is very much similar to that for Gentoo. It's taking Gentoo eBuilds and providing a service based on those eBuilds. And it's not container links is derived from Gentoo, but it's not really exposed that it's Gentoo. And that's sort of true of atomic host as well. And they also, yeah, they come out of the box with similar alignment points. So one of the biggest things that started with is basically this idea of transactional or image-based updates, right? In the previous talk, in Chris's talk, and if you use Kubernetes, you've realized the value in having image-based updates for your applications, right? But having that sort of thing for your operating system is really valuable because it means you know you can go to a particular state. One of the things that happened later in the evolution of these things is we started to design ways to extend the operating system. So you have this concept of a base operating system image, right? I have this thing just like an application is at this point. One of the tricky parts, though, is that not, and I'll talk more about this later, is not everything goes to the same life cycle. So in container links, there's this component called Torx, which is basically like a kind of boot time add-on management. So it's intended to still be immutable infrastructure style. You boot into the OS and you add things on it, but you don't ever remove them or have them drift over time. In Red Air Prize Linux, it's on a host. Actually, in Red Air Prize Linux in general, we have this concept of system containers, which are a way to define bootstrap containers. And at the top, a host also has package layering, which I won't talk about today too much. One of the most important takeaways, and this is the number one thing that we're changing right now, is that in both container Linux and Tectonic, right? So you have a series of layers. You have container links. It's a base operating system with container image. It has Docker. And on top of that, you would install Tectonic, which is Kubernetes, right? So they naturally, well, we support them moving at different life cycles. And the same was true with Red Air Prize Linux and Red Air Prize Linux Atomic Host and Red Air Prize Linux in general and OpenShift, right? You install an operating system, and then you install Kubernetes on top, right? So, yeah, I'll dive into this more in the next slide. So this has been a real, really big challenge for us over time because we tried to create this separation for a good reason. You don't want to, you want to update your operating system, right? You, you know, next meltdown, Spectre comes out. You want to be able to update your operating system and have that applied to all those containers, right? But a lot of people, pretty much, no one wants to just have their Kubernetes installation or their OpenShift installation just change day to day, right? That's something you want to plan for. That's something you want to control. And the, so we tried to support that by having the operating system and the cluster, Kubernetes operate using different technologies at different life cycles. The problem is, though, there's actually a lot of interlinkage between those. So the kubelet is the part of Kubernetes that actually talks to Docker and runs containers. It also does a bunch of other things that Docker doesn't do. And basically today, the kubelet tries to talk to that engine. So the engine is shipped by the operating system. And that Docker engine, or the container engine, could be cryo2, uses SE Linux, which is very much a part of the host, right? So we have this linkage across all those things. Another good example that the kubelet handles today is storage, right? So it can provision SEF volumes and things like that. Well, SEF has a kernel module, right? So we have this kind of linkage from the high level all the way down. And this is today in Red Enterprise Linux. This is very visible in the fact that we have the Red Enterprise Linux base channels, right? That's the base operating system. And originally we said, okay, well, that's a little bit too slow for the container engine. Let's create extras. So now we have two repositories, right? You can install Docker from extras, and then OpenShift is a layer on top. And then we have multiple levels of supported OpenShift versions crossed with multiple versions of the operating system and the extras repository in there. It's very, very complicated to manage. Now, obviously, and I'll talk about this a little bit later, OpenShift Ansible handles some of this for you, but there's no getting out of the inherent complexity in all this. And we can't fill out a bootstrapping model. How do we, how do you bootstrap from that single node OS into Kubernetes? So I'm going too slowly, so I gotta speed up. One thing that is interesting about container Linux is basically there's a single stream. You want container, actually, just a random question here. How many people have used container Linux today? A couple, a couple, okay. So, yeah, it's a very streamlined user experience, right? You install the OS, it comes out of the box with automatic updates, and that's really it. That's part of the idea is you don't think about the operational complexity of an operating system. You just install it, and the team, I think, did a really good job of keeping track of the latest versions of different upstream components. So the Linux kernel is one of the biggest ones, and a challenge has definitely been regressions in the upstream kernel, but also keeping track of the latest system D. But a challenge for container Linux was Docker because, again, Docker is tied to Kubernetes. So how do you life-cycle those? Well, it's a lot more complex for Atomic Host because we previously had three distributions, Fedora, CentOS, and RedArp Linux. Now we actually have four because container Linux is part of the family. And that made talking about Atomic Host a lot more complex, having these three things. But it is one of the most powerful benefits, in my opinion, of containers. I mean, there's a lot. Again, in the previous talk, you saw a lot of benefits of Kubernetes. Keeping that host operating system up to date, making sure that, okay, you know, the next, again, next specter comes out, you see that command, you don't have to think about, okay, why have these rel5 boxes over here? I have these six ones. How do I want to test that? Put all that stuff in containers and keep the host up to date as a really, really powerful model. Okay, so that was the background. And what I want to talk about is the road ahead. So the RedArp is a little bit more complicated than container Linux. So we're going to create, we're in the process of creating Fedora CoreOS and Red Hat CoreOS. So we're carrying forward this two distribution, two stream thing to try and fulfill, to try and deal with the fact that we have different audiences, right? Before all this stuff, we had Fedora and RedArp Linux and they target different audiences, different use cases. If you want to participate in the development, we point you to Fedora. We have moved faster for long-term operational stability of RedArp Linux. Some of that is going to continue. So Fedora CoreOS is in progress. It's going to be a successor to container Linux in 2019. Container Linux, again, if you're running it today, don't panic. It's going to have a long life cycle until next year. But the idea is Fedora CoreOS, we want to preserve that experience and move forward. Whereas it'll feel like container Linux. You install the operating system, you have a container engine, you can do what you want after that. Whereas in contrast, we are totally changing what we're doing with RedArp Linux, Atomic Host and OpenShift, where we're merging the CoreOS technologies into what we call Red Hat CoreOS, and this will solely be targeted for OpenShift use cases. So rather than having this idea, okay, well, I'm going to provision the operating system, then I provision the cluster on top, we really come out of the box with a very opinionated model for OpenShift and Kubernetes. OpenShift, which is Kubernetes. And again, I should emphasize, if you just want the Kubernetes aspects, OpenShift is Kubernetes. Now, we're having a lot of discussions about how exactly Red Hat CoreOS will work and what its relationship with RedArp Linux will be. The way I'm thinking of things now is we want to orbit RedArp Linux, but we do want to move faster in a lot of cases. So a good example is tracing and debugging, right? A compelling value is maybe some of you out there have applications that you've built on top of RedArp Linux 6. Today, you can put that in a container, run it on a RedArp Linux 7 host, but tracing and debugging is advancing rapidly in the upstream Linux kernel. So if you want to use the BPF-based tracing stuff, new cutting-edge stuff to debug your old RedArp Linux-based app, we want to have a faster-moving host where you have all those newest features and you can have your applications stay stable. So that's part of the idea. So there's a couple core ingredients. So what do we actually mean when we say we're merging the CoreOS container Linux technologies with the RedArp Red Hat technologies? One of the number one things from the CoreOS container Linux side is this ignition tool. So previously, it's worth looking at a comparison with RedArp Linux. There you have Kickstart, which is usually a way to provision bare-metal machines. You can use it to provision cloud images too, although that's more unusual. More popular in the cloud environment is this project called CloudInit. An ignition is targeted for both these use cases. Use it for bare-metal, use it for cloud, and it differs a lot from CloudInit and then it's a boot-once configuration. So you pass it a configuration and either it succeeds or fails. If it fails, your machine does not boot. And you can see... you can trace the failure in a console log. It's usually best to have your ignition configs tested locally and then before you scale them out. But one of the fundamental flaws of CloudInit is that it runs on each boot and so you can have your system state sort of drift over time if you change the CloudInit configuration. And again, a whole part of this philosophy is at a scale-out model, you don't want Pat's systems or systems that drift in state, right? So ignition, it's a boot-once config tool. And it has a lot of more powerful features because it runs in the initial RAM disk so it can provision disks, it can change your partitioning and things like that. And for both Fedor Chorus and Red Hat Chorus, hopefully Red Hat Chorus, that's a tricky one. I'll get to that later. I think one of the things we messed up with with Atomic Host was not having automatic updates on by default. Yes, we have a transactional update system, but I think it's really a powerful mental shift where you go to a model where updates are actually automatic, like you don't think about them, you can have control over them, right? But just having them on by default is a fundamental difference. So we have some work on basically polishing the OS tree layer, which we're going to carry forward but I'd like fewer operators to have to think about it. Again, I like that mental model where you provision the operating system and it just updates. And if something goes wrong, you can roll back, you have visibility and control into it. And there's a lot of other things that we're merging together. So from the core OS side, there's a strong commitment to security, right? It's fundamental to what we do and that's also true at Red Hat. And we want you to be able to manage it. And one of the things we actually just recently accomplished is getting ignition with SE Linux in forcing mode. So ContainerLinux has SE Linux, but it defaults to permissive might as well not be on because if you try and turn on, there's a lot of traps. So we actually just finally got ignition working with SE Linux, which was fairly tricky because it run very early in the boot process. So it's a little infrastructure where provision your systems and they stay that way. And the concept of the operating system as a cluster, which is mostly taken over now by Kubernetes. I'm going to get into that later. I just need to check how much time I have. How much time do I have? Sorry. Yeah, Dan's telling me it's not showing. Okay, so I'm going to go pretty quickly over this because I think I'll have a couple minutes left. We're discussing if you're interested in participating in development. And again, to me free software is important. Working in a community is really important. So if you're interested in participating what's the future of Fedora Chorus and transitively the future of Red Hat Chorus because we wanted to derive from the other. We have a tracker issue where we're discussing a lot of different things. So for example, ContainerLinux comes with network D, whereas in most Fedora derive derivatives we use network manager. ContainerLinux simplifies a lot is they don't really have an installer per se. You just have a disk image and you DD it, copy as an image direct to a disk. Whereas with Atomic Coast we just reused Anaconda which means we had Kickstart and a lot of choice. I think we've sort of come to a conclusion we want to again be more opinionated about the installer. In particular, the partitioning layout less choice there. A lot of discussion about how do we ship Docker and how do we manage host updates outside of the Kubernetes context? Update, Lifecycle and Cadence is one of the biggest and trickiest issues so we're discussing all those things. So let me dive down into a little more detail in the Red Hat Chorus side. I just talked about common ingredients across both although there will be some differences. The biggest, by far the biggest change and I mentioned this before in CoreOS is that the Kublet will be part of the host will have everything you need to run OpenShift versioned and life-cycled with the operating system. So again, rather than that two-phase install it'll just be there and there won't be a kind of bootstrap phase where you have to maybe install a different version of Docker or something like that. A lot of work went into the CoreOS tectonic installer. It's actually very different just to talk a little bit about the differences in a minute but the next generation installer again is going to be a lot more opinionated it's going to derive significantly from the tectonic installer. It'll use ignition and manage the host install to a far greater degree than the OpenShift Ansible version. And here's one of the biggest parts that I'll be working on. So I'm an operating systems guy I've been working on updating operating systems I've been in that area mostly for over 15 years now so yeah and when I say so briefly a brief background on an operator. An operator is a way to define a way to manage an application like it was part of Kubernetes itself and there's a lot more detail on the CoreOS operator page but the idea here is that the operating system update itself really the operating system itself should feel like a Kubernetes object. And the CoreOS guys basically wrote some code that I'm going to demo later if I don't run out of time for managing those operating system updates. So again it'll be automated on by default but very controlled so you'll be able to use kubectl to control the operating system update. Another change that we're making is basically taking this OS tree concept which is like the operating system image it's a different format than container images for various reasons. It has SA linux where container images don't or I won't go into that but the idea is we'll basically mirror encapsulate that operating system update itself in a container which is another part of making the operating system feel itself like a container. And one of the biggest things once you scale this out I was just talking about updates but actually how do you manage the configuration of all your machines so building on top of ignition and tools and technologies from CoreOS will have a machine configuration operator that's making sure that as you schedule a change for your systems whether that's networking related or cryptographic maybe you decide to change a system cryptographic policy right? What we're working is basically saying okay we can apply this change directly to your machine and if we can't we'll be able to take down the machines and do a rolling reprovision so all this is going to be integrated with Kubernetes itself so we drain the node and you know get all the containers off and have a sort of seamless update and kind of moving away from again immutable infrastructure any kind of container drift. And we're likely going to come out of the box with podman or cryo only so this may be one of the bigger differences between Red Hat CoreOS and Fedora CoreOS because Fedora CoreOS I think it would be very traumatic not to shift a binary called Docker certainly for anyone transitioning from container Linux but again for OpenShift for Red Hat CoreOS we don't want you logging into the nodes too much you know the idea is you're up maintaining everything through Kubernetes you hopefully don't need to care too much what's underneath the hood except for the cryo features so probably come out of the box cryo only. So it's worth talking briefly about the other path so we're only going to support Red Hat CoreOS for OpenShift installs there are a lot of valid use cases where you need to install OpenShift on top of Red Enterprise Linux as it exists today and this will very much definitely be supported the OpenShift Ansible code base I think last I heard one of the larger Ansible code bases out there is going to continue and it will be used to handle the kind of bring your own I have a custom requirements I don't want to make this leap to Red Hat CoreOS you'll be able to use OpenShift Ansible it'll feel very similar although it's likely that some parts of OpenShift Ansible will converge with some parts of the former tectonic installer because there's a lot of overlapping concerns around provisioning certificates all these types of things there's always a sort of core tension with OpenShift Ansible because Ansible is trying to define a cluster state but then Kubernetes is also trying to do that so again one way to think about it here with Red Hat CoreOS is that Kubernetes takes over that role okay so let me check I don't have too much time but let me just briefly show talk a little bit about a demo so this is code ported from container Linux that's talking to RPMOS tree from Atonic host so it's streamlining operating system updates and again the core problem here is if you just type yum update on a machine or RPMOS tree upgrade you really actually want to make sure the machine has its containers well you want to manage the reboots have the containers the update drain the machine have the containers move to other hosts and have everything basically be coordinated you don't want all your hosts to reboot at once and that's basically what this code is doing here so what you just saw let me pause this right here so one of the most important aspects here is again Kubernetes is controlling the operating system update so in this code the update state is annotations on the node object in Kubernetes and what you can see here is basically the operator there's a node agent that runs on each node and it's updating these annotations and saying okay I have an update I need reboot do I need to reboot or not and we also add a little bit of metadata around versioning so this will allow a web interface to say okay across my whole cluster what are the versions of the individual host right in a very natural Kubernetes focused way right again all that state is maintained in etcd and basically what this operator is doing let me see if I can skip ahead to a more interesting part so here we're deploying the operator and again the the operating system update is just part of the Kubernetes yeah so here this is demoing running the rpmostry status command which is showing the on an individual node the version running into a package diff but if you want to look at the logs you know why is maybe you want to you're trying to debug why a particular node isn't updating you can actually use the kubectl logs to look at that operator so rather than having you know something like okay I got an SSH into this machine and look at syslog or journalctl it's all part of your cluster and basically ideas right now it's fairly hard coded any one node will go down for rebooted at time obviously a lot of you want to control this in a much more sophisticated way this thing's actually not really an operator in that sense right now in that it doesn't have a lot of configuration itself so that's one of the things we're gonna be enhancing is adding a lot more significant control over this you know if you have 500 nodes you can obviously have more than one rebooted at time in most cases so I won't go through this whole demo because again I think I'm yep so that was a demo of container Linux ported code which is integrated with Kubernetes but for Red Hat CoreOS it's actually gonna be a much higher level cluster operator that controls those updates because it'll be very integrated with the machine configuration and saying okay I need to reconcile this ignition configuration and make sure that this node goes is in this particular state it's about yeah configuration plus operating system version which combines to form the cluster okay so in summary it's a lot of technology from CoreOS container links that we're merging together with Red Hat technologies and definitely I think it's sort of too bad that some things are gonna become more complex because again I really like that simplicity of container links today that concept there's only a single stream you get it and you install it and you know the updates just go in the background but I think the new split will actually be worth it because we're trying to again manage both multiple audiences but the number one change we're making here is open shift installations which if any of you have tried to do it it's not hard but there's a lot of things that we're making dramatically simpler here and I can go in yeah not too much time so I'm going to do more detail so that is about all are there any questions okay yeah so you see Fedora CoreOS is having other uses then OpenShift is OpenShift origin going to be using Fedora CoreOS is there yeah okay so the question is about Fedora CoreOS and other uses and what about OpenShift on top that's a very complicated question what we said so far in the fact is that we definitely want to support people using upstream Kubernetes on Fedora CoreOS that's a very valid use case and the degree to which as we'll work on supporting a kind of true scale out enterprise thing on top of Fedora CoreOS unlikely our product focus is going to be on Fedora CoreOS but you know if you want to play around with containers you want to do a small scale thing you want to do something custom everything we do will be free and open source and we'd like Fedora CoreOS to be where you enter things so can we build around OpenShift origin on Fedora CoreOS I think we'll probably make that work but you may lose out on some things like the operator based updates like I because we've definitely learned that if you want to support this at scale you need to have everything tested and versioned together whereas with Fedora CoreOS we're kind of still maintaining that container linux model of the operating system and engine will move forward and you can do something else do what you want on top and that will continue to work right there's a lot of technologies that manage that work on top of container linux today to manage Kubernetes we're not going to break those that's for sure that's another question in the back so these are some of the prior updates for Kubernetes specifically right I saw a presentation by Brandon about the Operator Framework was that related to this stuff or yep so the question was about the operator based OS updates and the operator framework and is that related to this the answer is definitely yes so the code that I demoed does not use the operator SDK yet I think when it was written for container linux so the SDK and the operator framework and idea have seen a lot of traction grow and again it's a way to manage your applications but at a technical level today it doesn't use that but it uses the same ideas if that makes sense and as we move forward especially with the next OpenShift 4 and merge tectonic components you're going to see increasing use of the operator SDK there to make sure that everything's consistent as a way to manage all the components of your cluster my question so you've spoken about Fedora and you've spoken about Red Hat Enterprise Linux what about CentOS where is that going to fit into this whole scheme yeah okay so the question is what about CentOS that is a complicated question so again CentOS today since it was acquired is kind of the wrong word but brought into the Red Hat family one of the things that one rule that CentOS serves is is where we publish the source code for Red Enterprise Linux so now it's tricky though because Red Hat CoreOS is technically life-cycled with OpenShift so it's likely that we'll be publishing our source code through OpenShift although that's not completely defined yet so it's likely that we'll at least have CentOS in that rule now one thing I would I don't think we are initially going to work on making a CentOS version of Red Hat CoreOS but it yeah I can't say that we'll definitely publish the source code but whether we'll try and emphasize that because I think the thing is Fedora CoreOS as much as possible to serve that rule because you really want your host to be newer and this is a sort of tension with the Red Enterprise Linux model versus containers and Red Enterprise Linux the operating system kernel lives with all the user space and the apps but you really want to have the host be newer so to me like I always wanted in the Atomic world people to run Fedora Atomic host and now Fedora CoreOS to make sure you have the newest versions rather than CentOS which is older so that's a very complicated question I guess the short answer is to be determined but I hope not to add too much complexity to the graph of operating system families if that makes sense Dan So is the versioning going to be different and there's not going to be a version Fedora 28 version and a Fedora 29 version of CoreOS correct? So the question was about operating system versioning so obviously today Fedora has major version numbers actually Fedora Atomic host is today life-cycled with Fedora itself whereas again container links really doesn't you don't think about that version number today you install and it does have a version but it's just kind of been a long life stream so that is still to be determined to a degree I think that we need to capture again that user experience of container links where you don't have to think too much about the operating system because you've containerized your applications and you can go to the newer kernel and newer system D with less fear I think having a stronger more focused integration testing story will allow us to move in a longer single stream I guess the sort of classic example is Python 3 you're like in that something where that may happen across the Fedora major version and if you're writing an application you care a lot about that but as far we don't want you to use Python for managing a host right so there you don't care about the host version so that's still to be determined and so honestly is the version number for reddit corvus I think it's likely to be four right which will be the next open ship version but so it's a very tricky problem because again we have we need to think about how people think about changes in Fedora itself which Fedora corvus will be derived from but we don't want to give the impression that you may have to reprovision every six months right we'd like to have a long life cycle for an install at least from an operator perspective an assistant perspective any other questions I think that's it all right thank you all thank you Colin so we now have a lunch break which will be served in the second lunch and hope to see you at one