 So, hey everyone, my name is Adam, I work in Red Hat as a software engineer. I work in Fedora most of my time and on the build system and all sorts of things like that. Scott. Yeah, I am a product manager so I focus a lot on the why and like what we're trying to do. And I tried to keep my fingers out of the what and how or the how I should say but I end up doing it too much and so that's how I end up. I end up with talks like this with Adam. So we'll go through kind of the why, the what and the how in this talk on like building a smaller container than it is. I think we can jump in. So, we're going to go through a little section where we kind of like dive through like over the last like seven-ish years, you know, I've been talking to users at conferences and customers and all kinds of people that basically use containers. And like the one of the one of the resounding sort of like themes is always like, I want smaller images, why are these images too large? How do I make these images smaller? And so obviously like over that time we've looked at all kinds of like ways that we can make them smaller. But in the last like 18-ish months, especially working with Adam, we've definitely finally, I think we're on to some really, really interesting like techniques and ways to make container images smaller while still retaining all of the all of the advantages of using a Linux distro. And so we, this is where we get serious. This is where we're going to dig into serious business. So first, I'll walk through like some of the things that I see people ask for like they always ask for smaller images, but like what they really mean, what they really think they mean is that they want a smaller attack surface. And by attack surface, if you haven't heard that word, what that means is, is, you know, the less permutations of software that you have in your environment, get to find a bug and get into your environment. So, so if you have like 22 different versions of glibc and 100 different versions of open SSL and, you know, 50 different versions of engine X and 200 different versions of Apache. That's much more likely that like one of those versions of those, you know, 500 different pieces of software is going to have a bug that will let somebody get in. So like if you only have one open SSL and one Apache and one engine X and one glibc, it's much less likely and it's much easier to track the CVEs for, you know, for those, for that software. So like the attack concept of attack surface is that within an environment, within a software environment that's controlled by a specific organization, you know, the minimum number of different permutations of piece of software. What that's what they actually want, but they just say they want smaller container images, because they're thinking about attack surface within the context of a single application or a single container, but that's not actually the way it works. It actually applies to everything in the environment, not just like a single container image. So they say smaller container image but like I mentioned, you 1000 different container images that are like five megabytes each is actually five gigabytes of attack surface. Or in that case, it might have actually been better to have like, you know, 200 containers, you know, that are that are that are much bigger that are like, you know, 15 megabytes each right or whatever but like there's this balancing act of like less container images that could be slightly bigger versus a bunch of very small container images. It's not quite as black and white as everybody thinks to get to the actual less attack service, like smaller individual container images as part of the battle but not all of it. And then another thing I think that users want is they don't have to think about it right like they just want, like, like there's this concept now of like consumerism that just you know I just want to build it down with this thing and use it. And then I think finally, we jokingly said, you know, they want cool tweets they just want to like tweet out like some of my more popular tweets are like when I show some of Adam's work or some of our work on on on what we call UBI micro on on Fedor on using this sort of technique. If I can produce really small images and tweet those out that's pretty cool so I think that's what users want in a nutshell. And so, to try to solve those things like we looked at we've looked at like over the last couple years working together like we've looked at a bunch of different things like so my journey on looking at smaller images started probably seven years ago but like some of the many of these are dead ends by themselves in and of themselves. So we looked at scratch builds. We looked at minimizing the dependency tree within like Fedora and well, we looked at distro list which is is really like the concept of using binaries that somebody else provides to you in a container image although I'll dig into that a little bit deeper. And then we looked at compiling from scratch, which is something that we've done in Linux forever you know like a patching you know, compiling Apache from scratch for example, and then we also looked at the concept of again like less container. Which I'll dig in a little bit deeper. So, next slide Adam if you don't mind. So the first thing is scratch container image builds right like so we thought about the concept of like using yum to install into a scratch directory and then copy that into like, you know into the container and so like using like multi stage builds along with, you know, building, building basically from scratch with like nothing in a in a directory to start. The problem with this is is you're basically competing against like the Fedora contributors, or the rail builders, you know the rail package builders, or, or Debian or whoever, you know whatever Linux distro. And they're pretty good at it like these are not dumb people like they're pretty smart and they're pretty good at it so like, you're not what I found myself is better than them and I went down and I started trying to build him from scratch and I realized, Oh, these are like the exact same size of the ones that are coming out of Fedora and a boot to and row and blah blah blah. So like I was like, All right, so this is not. This is not a magic bullet in and of itself to do scratch builds right because, because what happens is it's a dependency tree. You know there's a dependency tree and certain dependencies get pulled in and it is what it is and like those dependency tree dependencies get pulled in, no matter whether it's somebody at Fedora at the Fedora project doing it or or a rail package manager, or if it's me so like it ends up pretty much being washed by itself so this was kind of a dead end. The, the next thing we tried we I looked at immediately was like well how do I minimize these dependencies. So in this graphic this is a graphic that actually one of Adam's tools produces. And, and it's pretty cool because it will kind of give you leads like, like if you're an investigator trying to figure out like how do I figure out what's getting pulled in that's so big that's making this image get bigger right. So it's really nice. But then what I realized is like, Oh, I want to, I want to change the dependency tree here. That actually gets really hard like you have to be like, like I got the bright idea one day to try to look at the Apache spec file in Fedora. And I was like, I'll just fix this I was a sys admin forever I've done of spec files. And I realized, Oh my God, the Apache spec file is one of the most sophisticated spec files. And then trying to minimize like turn certain dependencies to weak dependencies and do all that myself. I quickly realized was like a lot of SME knowledge like a lot of subject matter expertise that I didn't even know I didn't have like like when I'd compiled Apache from scratch I just pulled in whatever dependencies and it just worked and I did, but I realized I didn't have an actual strong grasp on the actual dependency tree like it's a very different level of knowledge about a certain package. It's even different than what maybe the developers of Apache or the users of Apache actually know so like the packaging skill set is definitely its own thing. So this again kind of ended up being a dead end for me by itself. Finally, I looked at distrilis and I figured it was easier for me to let cletus hikes aka tanner guy of world renowned fame and with a very, very popular tweet, where he explains what distrilis is, you know, there's no such thing as distrilis just another dependency that you know, jokingly, I mean this is obviously a joke, but, but, but in a nutshell, like, if you look at like this, the distrilis, like there's one there's a I forget exactly what the community name is but I think it's something distrilis on GitHub. It's a Google project. If you look at that like how some of the some of the software that you're trying to download is compiled from scratch and is like its own thing and so you're expecting that community to manage those binaries for you and all the libraries that those binaries are compiled against. And then others are actually just using Debian packages. So like what I quickly realized was like, there's no such thing as distrilis like you either manage the dependency or some other team manages the dependency, but somebody's managing the dependency. You're managing it from scratch or you're managing it as part of a set of escrowed binaries. And by escrowed binaries, I mean like binaries that some other team builds and saves somewhere right like, you know, that's that's basically the kind and I quickly realized that is a Linux distril like you can call it distrilis, but it's a Linux distril like you're essentially offloading that subject matter expertise to another group of people to go do that for you. And I found some challenges with that in that like a distrilis group, you know, is not necessarily have the same track record as like Fedora like Fedora has been managing for 15 years, you know, 1520 years. So you know that they that there's like organizational knowledge of how to manage an escrow binaries. And that's what they do for a Linux distril that runs on bare metal. So like, it's not really that different for inside of a container image. So, again, distrilis by itself, it feels like a red herring. So, finally, as I mentioned, you can you can have somebody escrow those binaries from scratch, you know, for you, or you can build them yourselves and escrow them yourselves. Now you you have the option you could compile from scratch every time you do a computer container image build, but that would be terrible, because that would take forever, or, you know, and you could cash them on your build servers, or again, you could escrow them and some server somewhere. And now you just became a little bit not super fun like this quickly. This was probably the quickest dead end, where I'm like, I don't really want to do this because now I now I'm in the business of building a distril and in the business of building container images from scratch. And now I'm two steps away from what I actually want to achieve, which is just have a small image. I just want a podman pole or a Kubernetes, you know, you have a file that pulls an image I don't want to, I don't want to have to like do all this work myself. So, this is where like, like, like the final one we looked at was like, well, what a burning environment right like, like, the config management world we very easily understood that like having one Linux distro to manage was much easier than 22 different Linux distros because there's always these small differences between them. And even though the config management tool helps you, it's still easier to minimize the number of Linux distros. For some reason, once we move to containers, we dump everything in a tar ball and we were just like, we, we like we don't care what's in there and we're like, it's fine. But then you realize, but from a pure attack, you know, surface perspective, like, I just gave a very general example here I just picked random, you know, or relatively recent versions of Fedora Ubuntu Alpine and UBI and I added up the different versions, you know, you'd end up with 11 different versions of OpenSSL, three different versions of Muscle, C Library and eight different versions of G-Lib C. So you basically have 11 different C libraries and 11 different OpenSSLs. And you go, what, what value do I get from that? Right? Like, that's actually way, way, way more than attack surface that if I just standardized on like one in this example, I just use UBI8, but this works with any Linux distro. If you just standardized one version of one Linux distro, you end up with a lot less, you know, a lot less attack surface because you ended up with one G-Lib C, one C library, one OpenSSL library. And I use those because those are two really key important libraries to think about when you're thinking about containers. And so finally, when we pull all this together, we've, we looked at this and we realized like none of these by themselves is completely a magic bullet, but when you pull it all together, there is something magical. And so like a lot of Adam's work is going to cover basically like using a high quality dependency tree. Like I still think the Linux distro still matters. So like I did a talk at FOSTA last year, like the Linux distro still matters because they're really good at managing dependencies and escrowing binary. So like that seems dumb not to use that. The existing tooling, the tooling that they build also still works really well and they're really good at managing dependency trees and pulling that. Now minimizing the dependencies, this is actually a place where Adam's going to go a lot deeper because he's done a lot of work and people don't quite realize how hard this work is because there is actually no single expert within a Linux distro that knows how to minimize it. It's actually, it's actually, you know, hundreds and thousands of people working together each building their packages. And so like it really has to be a collaborative team effort to like do this. And so, so like Adam will dig deeper into that it took a lot of work to get us on the right track there. Again, choosing a single Linux distribution obviously we're doing this within Fedora and RHEL ecosystem because it's what it's where we work but like any single Linux distribution could do this but but we're going to talk about ours in particular. And then finally after you have all of that the output the effect of it is like that you will have smaller images, and then those smaller images will share the exact same libraries with all of the other small images in your environment. And now you've truly achieved what you want you want a smaller attack surface. So I'm going to hand it over to Adam to dig deeper into a lot of what he's done over the last 18 months. All right, cool. Um, so yeah, let's talk about building small image, be showing some examples how you can do that because I expect because you're here you I expect you want to build stuff yourself. So I want to demonstrate a few things how you can actually do that with a Linux distro. So I'll be using Fedora as an example, because that's where innovation happens but like everything I show will work in basically any RPM distro that uses that uses DNF, which is a package manager. I'll be using the concept of multi stage builds multi stage container builds, and the main idea is basically like when you buy a car, right, you only get the car and they won't ship you the assembly line with it that would be that would be horrible. Um, so if I think about this concept and package managers, I can do the same thing right. Um, the package manager to build the container image, but I don't want to ship it because that's that's no use in the end. And Fedora's package manager is DNF and yes, a few features that can help us actually achieve this. You can install into an empty directory so you don't have to install into your system but actually somewhere else. You can disable weak dependencies so in the RPM world there's many dependencies and some of them are higher some of them are weak and weak means that they are kind of optional right so you can disable all of the optionals and then intentionally pick what you actually need that reduces the size. And you can disable documentation packages because you don't need those in containers. Those need to be very small. Combining that with some knowledge about the actual ecosystem. For example, we have alternatives, because it's a huge, huge group of packages, and there's just a few of them that just might provide the same functionality. And there might be a huge difference in terms of size or other things. Um, there are two in particular I'll be using as an example, and these are gleefc minimal and bagging cordial single. I'll show them in a bit. So let's try to build a Docker file. I'm, I'm using a Docker file because that's very widely known format how to build a containers. This will work in any other format that you're using with portman with builder. I'm using this because most people know it, but the concept is the same everywhere. So let's try an example engine X you probably don't need to build engine X container yourself like you can get them from red hat or fedora, but this is an example of how you can build an engine next right. You might think that this is the best way you just do from Fedora 33 run the package manager install engine X. And you end up with 224 megs image, which is not amazing. So how we can make it slightly smaller. So I'll try using the scratch build that Scott said, right. So this is the multi stage built in the container. So I'll say, Hey, I'll build, I'll install everything into a directory and they build a new image just from that directory. So there's no package manager in the end, but something really weird happened. It just grew to 450 megs. Why? This is very weird, right. So what if I apply some other knowledge that I shared, I can disable weak dependencies, I can disable documentation. But what makes the biggest difference is actually choosing some of the alternative packages in the dependency tree and these are g lipsy minimal unpacking for you till single. These are just smaller versions of, of those packages and it suddenly dropped to 116 megs. So these, these added to side by side. And you can see how with a few tricks, you can actually build an image that's reasonably small with still using the Linux is throw. So this is using packages that are maintained by hundreds of experts. It's always complete where you can always install it. It works. It's maintained. If there's an issue, you can open a bug that'll fix it for you. This is what this show does. And if you need an update, you just rebuild the image as you would normally do, of course, you want your images to be immutable. So that's how you get updates in. And that's how you can do much, much smaller results. I have a few other examples you can do. I call the distro as this is probably bad name, but you can build a small base which only has a g lipsy and bash in it. And that's 38 megs. And this is how you, how you get there. And you can use it as like a general environment or just put some go lang or rest binary in there to run it. There's no package manager in there. And if you want to get even smaller, you can build a 1.8 meg distro less in quotes busy box base. Again, from Fedora from an RPM, but there's no package manager in the, in the results. So you can just copy this out from the slide will show them later. And you can build yourself a tiny image. So these were just a few examples how you can actually use distributions to build a small image, but don't include the package manager. And this is possible by actually minimizing the dependency tree, right? That's why we're able to build a few, that's why we're able to build smaller images. And I want to share a few facts about that. So Scott mentioned this right dependencies are complex. And this is the Fedora repo. And it's like a dependency graph with all the nodes packages and many various dependencies in between them. And you can't really just approach it as an individual and try to make it smaller. Right. This is really complex and hundreds of very clever people maintain these individual pieces. So you need something broader to actually optimize this. So that's where Fedora's objectives come in and minimization is actually one of Fedora's main objective. So the whole community is actually working towards minimizing the dependencies so we can build smaller images. And this is a graph of Fedora base image sizes. Basically you can maybe see on the right side of the colors change. I've been building this graph over time, but around Fedora 30, I have two entries here. This is where we started the minimization objective. And that's where the images sizes started to drop quite significantly. And this is again only possible with many, many people working together and making sure that the dependency tree is organized is optimized. And also that the image is using only what it needs. And for that we have a tool. This is actually available for you in public. You can just see it and I want to share a few things you can do. This is looking at nginx, which is a web server. It's what we built before. This is analyzing it in various space container images and on various architectures and these are not necessarily images that Fedora builds or ships. This is mostly just installations that we expect people to do on them on their own. So we want to make sure that these are as small as possible. We analyze all of this and this is like all the Fedora teams. This is all available to all of them to just monitor whatever they need and you can see at the bottom right. This is the, this is the nginx image that we built. And yeah, there's dependency graphs that you can see over time. There was some weirdness in the middle, but overall not growing package list with details about each package. And even you can drill down to the interdependencies between individual packages. I can really show you things in detail and this is available on this URL. You can go there right now and you can just check it out and you can even contribute your own workloads to put them into the monitoring. So that's my part. I guess I'll hand it over to Scott. Yeah, so maybe next slide. So some of this work, like you can see it today in Fedora, obviously as Adam showed, like, like, and we mess with this all day, you know, all the time in Fedora. But also you'll see this work arrive in rel, some in rel eight, some in rel nine, mostly is kind of where you'll see it. I mean, we're kind of done with rel seven, but, but I wanted to at least put out there that like, again, these, these are like a pretty decent base image that standard that like where you're going to see a lot of this work arrive. So like, we have something called UBI micro coming in rel eight, four, which shouldn't be in the next three ish months. And so you'll see what we did get it down to about 38 megabytes uncompressed, and I believe compressed, it might be 13 ish megabytes. So like it's pretty darn good across the wire like 13 megabytes and then 38, and that's with a full that's with a real G lib C. We are looking at other ways to minimize the dependency tree like like I'm working with Carlos O'Donnell to like look at ways that we might trim out a megabyte or two here and there for like TZ data and, and other other like components of G lib C. So there's still we definitely have a roadmap of things where we can like continue to focus on driving down the dependencies but like you quickly find that like it touches a lot of different teams and a lot of different clever people like Adam said. And so it's, it's, it's a tough problem but it's one that's fun to work on. Again, like we said kind of destroy this images like like again these are this is a joking word that we use because like in the image, it doesn't have the assembly line packaged, or maybe another way to say is like, we package the assembly line in a build container in the first stage of the build, and then we only deliver the car in the second one right and so, so like if you need to move the assembly line around you can. You could, you could save that assembly line and cash it and keep it around, and then know exactly how to rebuild that image. But, but in general you add in production you only want the, you know, the car and that makes sense. And again, it's not necessarily distro like serverless and all these other less words, it's somebody else's you know somebody else has to manage that. And in this case we think that like letting fedora and the little Linux distro itself manage that makes sense. I already kind of mentioned like the dependency tree relate rel nine you're going to see a lot of this I mean, I'm hoping to see even smaller and rel nine, like when we have a rel nine micro, you know, I hope to even get it down in the 25 ish megabyte range we'll see we can get to that but getting to that and then maybe even looking at a, a, a, a, like maybe even a busy box image just by itself as like a base starting point. Definitely some other ones that keep your eye out for like probably rel eight five ish time frame is around having like an open SSL based image, and an engine x based micro image and an Apache based micro image those are three they're high on my priority list like I'd really like to get those three out by like rel eight five time frame. And so those would end up being then 120 megabyte range for like a patching engine x and I think even smaller 80 ish megabytes I can't quite remember the size off top my head for open SSL, but they should be pretty good even with the rel eight dependency tree but then with again with the rel nine dependency tree which will inherit a lot of Adams work you'll end up seeing a lot even smaller images but then you know, I bring this all together to like try to explain like this is how you actually get to a smaller tech surface right like you have to have standardized C library standardized open SL standardized, you know, XML libraries things like that, like have all these standardized and then and then have the minimum, you know, each of the images needs to be minimized but then the entire dependency chain, your tree has to be also minimized so that like the tech surface within your organization is actually minimized and so this is kind of the path we're going down and like we think it's a pretty good path and happy to answer any questions. If you guys have any. Okay. Adam captain was asking does minimizing depend dependencies go hand in hand with the objective of reproducible builds to keep the auto factory metaphor going can these techniques help ensure we are building the same car. Yes, absolutely. This was one of the core principles behind the minimization objective like there were options right we can just install something and then just tear things apart and make it small but terrible and not reproducible so the decision was that we actually want to keep using RPM dependencies as they are unmodified right so we're doing changes before actually the installation so this is during the packaging and during the build process so it's still just RPM tree and it's still reproducible like the way we built packages in Fedora are in in a boat system that just keeps track of all everything that was in the build truth at any particular time and you can really reproduce like to the bit every single build and that's what definitely we keeping. And so yeah, the answer is definitely yes that's that that's very important to us. I have a question from Cornell. What do you think about nix and building nix OCI images, while overriding certain package attributes, for example disabling PAM support the kids package. Does it count as making your own distro. That's an interesting question. I don't, I'm not super familiar with nix. So I don't have a good answer to that off the top of my head. How. Yes, Adam. Yeah, sorry, no, no, no ready expert in next day. Yeah, I guess it's a strange question for us because we basically deal mostly with like fedora and rel. And in the fedora and rel world we would probably never disable things like Pam like we want. We want to keep the way the binaries function in a container. So the containers are just fancy processes and container images are just fancy files that live on the operating on the disk then we basically want to keep them functioning quite similarly to to the way they function in a native traditional Linux OS. So like we really try to steer away from like, I get that I start to squint my eyes and go, I don't know about anything that would like make the binaries behave differently in a container. We already have enough confusion. And you'd be surprised how many questions we get. The vast, vast majority of the questions I get are trying to take something that exists in Linux and then make it work in a container I'd say that's 85% of the questions I get. And so like making things work differently, kind of defeats that so I think I don't think that's a technique we would look at deeply. And also like we're using glib see the standard C library and it's kind of big but like that's the standard that that everyone uses right and that that's what we want to use as well so there's this additional ways how you can optimize it like, for example, you with the busy box but that's not what we're really aiming for we want to use the standard ways. Okay, let's go to the next question. Is UBI 9 is out. No UBI 9 will be based on rel 9 and so when rel 9 drops will release UBI 9 at the same time on the closest thing that you can see fedora fedora if you go and play with the fedora images, you're going to get very close to what what you'll have in rel 9. Okay, questions from Renee. What was the improvement from from performance perspective of minimizing site as showed for NGE. Are there any performance metrics. So that's when that's when I'll jump in Adam if you don't mind because I have I've had some recent conversations around this that are interesting. So, minimizing the dependency tree and all these things will not really have an effect on performance. The two main places that will affect performance in my opinion are glibc and the kernel are and probably open a cell library. You know those those three things will matter and, and actually I just got a recent report of somebody using a partner using. In fact I still have a I have a call with them next week I believe if I if I remember right, because they want to understand why their database is performing better on UBI than than other Linux base images. You can break this question down and say well why does why does nginx or why does you know a database perform better with this library than that library with these binaries and those binaries. I mean that's always going to come down to the kernel glibc and then the libraries right and so like over this journey that I've come from the container side learning more and more about glibc and like all the work we do in that in that tool chain. I realized that that team does an immense amount of work to make the glibc that like red hat produces in fedora and in rel and even just glibc in general, even upstream. Like they do a tremendous amount of work to like make that work really quickly and they worry about runtime performance. The, the minimization would only ever have to do with the pull time of pulling the container image but the runtime performance is all the work of the glibc team and then the kernel team and then how that stuff's compiled and how it works. And so apparently there are customers and and and users that are seeing different performances based on which which library I did I did research muscle see a while back again I didn't we didn't do like crazy benchmarking but it definitely looks like there are system calls that are significantly slower and like other if you know engineering is always a trade off right like the there's a reason why glibc is bigger it has more functionality and it's probably been better tested and blah blah blah. If you go to a smaller library like muscle see there's definitely function calls or system calls that don't funk that don't perform as well. I suspect that that's where I would focus your time if you're worried about engine x actual runtime performance is the the kernel, the C library and like the SSL libraries things like that. Next question from Camille. Have you considered also using lip curl minimal and curl minimal alternatives to leap curl and curl in the base images. Yeah, show I think that's basically with whatever I show with the glibc minimal and this minimal versions of other packages and yeah that you can definitely use those. This goes back to like knowing the knowing the ecosystem and knowing like what to pick because this alternatives. I kind of I kind of wouldn't want to have maybe a list or something or way to make this more obvious for people so like they don't need to just know, but they can refer it something but yeah this is definitely part of it just making the smarter. See tree that work the best for whatever you're doing. And back to the performance. Um, you might also want to consider just like sometimes smaller might be maybe not that performance so these are always trade offs right but yeah if you want to optimize for size just like go for all the minimal packages and you get very small image. Yeah, and to jump in on that the one about the lip minimal that's interesting like on the discord server I'd love to discuss like would you view it useful to have like a curl container image or a W get container image that's something I'd love to discuss a little bit deeper like that's where my mind goes like it might be easiest for us to produce a container image that already has those minimal versions in them. So I'd love to discuss that a little more if you have some time. Okay, question from Pavel. Do you use for scheduling and triggering the image builds from Jenkins? Well, Adam could probably answer Fedora and I could probably answer Ralph. Okay, will there be a question from Ted? Oh, no, no, no. Oh, no, that wasn't the answer I was I was handing it to Adam to answer like what using Fedora to build. All right, yeah, Fedora I think uses OSBS which is how would I just open shift open shift yeah but yeah open shift container build system that that's what Fedora uses and you can find like sources for all the container builds on src.fedora. I think it's slash containers. That's where you can find all the sources for the Fedora packages. Yeah, and the same is actually true for Ralph like internally we have an OSBS system that basically is a huge cluster of of servers that are all running open shift and then that build system actually is what builds our container images and there's this dependency tree graph. Like if we rebuild UBI we have to rebuild our images we have to rebuild all the middleware images etc. There's a huge I think red hat might have one of the biggest build dependence. No, but but it's probably it's pretty big. We build millions of images per month. My question next one is from Terry be a cent OS stream nine container image. Terry why are you asking me this. No, I'm kidding. Yes, it will likely be. Yes. I'll go ahead Adam, were you going to say something. Oh that though I would just be very surprised if there wasn't so yeah. That's, yeah, in fact, in fact, Brian Simpson is working on a CentOS eight stream image right now. As of today we're building it off of UBI we're basically using a UBI eight image and then running a YUM update against the CentOS stream things so that like basically UBI is the since since CentOS stream is a future version of you can think of it almost like a raw hide. It makes sense to just like update the the slower moving you know you know and like from you know you can think of you can think of CentOS stream as rel.next. And then you could think of like UBI as rel. And so when you want rel.next you just run a YUM update and just a few things should change like you know, but are just a small set of packages. There's only like 180 packages in in UBI so like you're on a YUM update, you know, it might be five packages get updated so like that's the way that's the way the plan is to build the CentOS stream images as of today. Okay, there was also a question from James regarding realm nine when it's going to GA but Terry already replied that it's in the middle of 2022. Correct. Yep, that's right. And the last question the funny one from William. Scott, can you remove your hat so we can see your new mane. I am. I'm too bald to remove my hat but you can see my my you know you can hear I don't care. This is my hair. This is COVID hair. This is like whatever a year of no haircuts. I believe that you made our days like you know, if it's a bit more colorful with with this last picture of you which we can project. That's all questions. We still have like two minutes to go. So if you have any question, it's the last moment to ask. I have no idea what my pandemic hair makes no sense to me. You've always had that. I know. Thank you. That's all I believe. And I would like to thank you, Scott and Adam for presenting the workshop. We had great questions. And there is one more from Pavel. What was the biggest issue when minimizing the dependencies? I don't think I know like the biggest issue but it's just getting everyone on the same page and figuring out like what to do because there's if you if you say smaller right like what do you mean smaller like do you want to minimize the repo do you want to minimize. And then there were like many things and we need to just discuss this and figure out like what specifically we're actually making smaller versus like what are we keeping right so we're keeping IPM. But we want to maybe make the dependency to be more optional and stuff like that so. So those things and then I don't know it's it's that they went like a huge problem it's it's an interesting work definitely. So just like yeah getting everyone on the same page and making sure like people actually find time and like on top of that actual you know jobs and everything to focus on this. But yeah, I can think of like something really really huge. I think it's going. Yeah I concur Adam like for me it was just getting people excited about the idea of minimizing things it's it's grunt work in a lot of ways right like if if I have 50 million things on my plate somebody says hey by the way you need to make things smaller I'm like like I'm going to get grumpy like it's a getting everybody excited about like no this matters like especially in the container world. That's right in my mind the hardest piece. Yeah and sometimes it actually means like making the packages life harder because they may be using some huge library to make this piece code simpler right to just like call a function. I'm asking you hey you need to remove this very useful library because it's massive like that's actually a burden for them but it benefits everyone else right and then we need to figure out like what to do about it so there's that many tradeoffs that's it's mostly just tradeoffs.