 Hi, everybody. I'm Nolan Diabla. I work on the containers team at Red Hat. And I also work with the developer experience team, which is, and both of those are the subject of the talk I'm giving today, which is how OpenShift builds container images. I'm actually going to focus on a number of subjects. And let's see if that. Nope. Oh, it's not on. There we go. Number of topics I'm going to be covering today include the key bits of software. I'm going to be covering them, not from the viewpoint of what they work on, but mainly how you can fit them together, whether they're libraries, if they can be remote controlled, how they get remote controlled, how they piece together. And then I'm going to basically describe how we use them in 3.11 to do container builds, what changed between 3.11 and 4.0 that causes to change how we did things in 4.0, what we changed in 4.0, all the things we broke, well, not all the things we broke, and some things we're going to be doing even now as we move toward the final release of 4.0. So are we with me so far? I know Dan's going to tell me to slow down. OK, so the first thing that we'll have the first player in all this is Kubernetes. It's officially listed on its website as an open source system for automating, deployment, scaling, and management of containerized applications. But in short form, people will call that orchestrating containers or a cluster, and probably more things than containers now. It's controlled over GRPC. That's the important thing as far as this conversation is concerned. So if you're running it, you're using a, is this my water? No? You're probably using a remote client like kubectl to drive it. There we go. And also, it's kind of a big deal. Just a second, I'm a child at heart, Dan. The next thing you're going to consider is OpenShift, which is Red Hat's version of Kubernetes. It's controlled also over GRPC. In fact, you can use the same standard kubectl command to talk to it. OpenShift also has, historically, its own command my client named OC. But at this point, the commands are very, very similar in terms of what they can do. In fact, I see Antonio nodding along. So are there really big differences at this point? OK, thank you, Sally. It adds application lifecycle management. And yes, I crib this exact text off of its website. It adds DevOps tooling. And the most important thing in terms of this talk is it adds builds. And if you're not excited about this as I am, well, you're not going to like it much more at the end of the talk. Dockardy has traditionally been the runtime that Kubernetes used to actually do the heavy lifting of running a given image as a container anytime it needed to do something. If you were running a workload on your Kubernetes, it was being run by Dockardy. It downloads images. It runs containers. It's controlled using a REST API, which is different than an RPC API, but the end goal is the same. You're over here. You tell it to do a thing. It does the thing. The REST API includes a build endpoint, which is how it used to do builds. Docker build was the command line client that used to give it builds. It would use the REST API. You would give it a context directory. It would tar up the whole thing and send it over to the server, which could take a while. And it was a noticeable step that had to happen every time you did a build. At that point, the image build is done completely remotely from where you were when you invoked the build request. It's done entirely by the engine. None of it is exposed to what, well, none of the inner workings of what's going on there affect you where you are. If you're on a different machine, all the build happens on the machine where DockerD is running. The engine parses Dockerfile, which is included somewhere in the build context. You can tell it exactly where. It figures out what to do. It runs those things all inside the engine. The base image is pulled into the storage area that the demon is using. And the image that you built is also saved alongside that base image. One of the benefits of that is, once you've done that, you can run the new image immediately because you never had to push it anymore or pull it down from anywhere. The next base you probably haven't heard of is Image Builder. It's a tool that OpenShift uses to parse Dockerfiles. Stay with me. And then it would actually execute the build using a set of callbacks. Now the Image Builder command line program implemented callbacks to do these things by talking to a remote Docker demon. So it's very similar in execution to Docker Build, but it's a library and a CLI wrapper. And in particular, that library lets us get away with not having to implement parsing logic ourselves. So that's the thing I really like about Image Builder. Sourced image is yet another OpenShift project, which, if you're familiar with OpenShift, you can use to do content-based builds, things that you're not going to compile so much. The primary target here is things like Python scripts or PHP environments where you don't actually have to compile code, but you do need to have a runtime there and you need to format your code that's in your repository. You need to put it in a particular place. Well, essentially the goal is if your runtime is also your build tool chain, sourced image is the thing you want to be using. It uses information in your source tree to tell it how to lay out the contents of your repository into the build, into the runtime image in order to make your code runnable. Sourced image internally uses the Docker client library that it imports from Docker itself to start a Builder image using a remote engine and doing all the work again in the Docker engine. And at the very end, it commits your image. Now commits a fun thing because it isn't the same as Docker Build, but the end result in both cases is a new image. And as of 1.1.11, which is not yet in Fedora, you can actually tell sourced image to not do any of that, but just instead spit out a Docker file. Now this was also very useful to us. And again, it's also implemented as a library and a CLI wrapper. Now OpenShift Star Builder, there's not actually its name. It's a wild card in that NAP font I couldn't put them both in. Traditionally, the OC binary did everything for OpenShift. There was a build, you know, when you were running things, it was running them out of separate images, but it was all essentially the exact same binary. It was just being invoked under a different name. One of those names is OpenShift Docker Builder, and it does a whole lot of things. For multi-layer builds, which, well, are the default for Docker Build, it would run Docker file builds using DockerD, again, using its REST API. It was actually implemented using Go Docker Client, which I guess I skipped over. For single-layer builds, which was an optimization you could tell to you, it would use ImageBuilder. And ImageBuilder would go through your Docker file, run all the commands locally. Well, if you encountered a run instruction, it would tell the DockerDman to execute the command in the container that it was working on. Anything that didn't require running a thing, it would just modify the configuration locally, and then it would commit it all at the end. So no matter how many instructions you had in your Docker file, ImageBuilder could just build one layer, which was cool. It, in many cases, produced a smaller image. OpenShift STI Builder is yet another build image that essentially wraps OpenShift, sorry, yeah, sourced to Image, to produce Image. These are actually invoked by OpenShift when it goes to build an image. OpenShift builds in broad strokes. There are two things that factor into how OpenShift actually kicks off a build. The build trigger, which is what tells it to start the build. And that's invoked either through a web hook or in command line call to OC, or another image being updated. This is the things that make it look like magic so that when you push to a repository, the web hook gets invoked by your source code management system and that's what kicks off the build. The build is actually done using a build pod, and this is something that's really clever and that's like if you didn't know how it works and the fact that it's just a build pod, as in Kubernetes it doesn't really care much about it. This is fantastic stuff. The job of the build pod is to run either OpenShift Docker Builder or OpenShift STI Builder. And if your source code happens to be in Git, it'll check down the source code first. It runs that in amount of volume. So it's actually running multiple containers in that pod. The first one checks out your source code into a shared volume. That volume is again available to the builder image. And the builder image has to build the image and then it needs to push that image to your registry. Somehow, OpenShift does not care how this happens. This is the opportunity that we had going into 4.0. And it also needs to tell the framework if it's succeeded or failed. The next thing is redeployment. You usually have a deployment config which again says, okay, well, if the image was updated or if a tag was changed or if I modify the deployment config, any running positive using that image should be migrated over to using the new version of the image that I just built. And that's the magic of pushing your code to your repository, having it build and having all of your code that's running the old version of your code switching over to the new one. So in 3.11, this is all done, actually still done by the build controller. Its main job was to read the build config to determine where all this stuff happened and essentially what it needed to do and build a pod specification to run in your cluster. That's all that that says. Yep, more of that. And here's an example build config that I actually generated yesterday using one of the examples that OpenShift suggests and I could scroll through this. The interesting parts here for me are the source code. This causes it to check out your source code and the fact that this is a source strategy which means it will be invoking OpenShift STI builder. And yeah, nothing else useful there. This is the build pod it produces. It's a bit longer as you can see from the scroll bar. The important part is that it's creating one container that invokes OpenShift STI build and these are some of the variables that we're setting. That's fine. And we'll scroll down. And here's an init container that's wanting to check out the source code for your command, for your repository. That's, I mean, the slides are probably, you can scroll through this at your leisure or you can use oc get-o-yaml to examine them on your own cluster. So a bunch of stuff happened between when we started planning 3.11 and when we started planning 4.0. So it's too much to explain. Let me sum it up. Kubernetes adopted CRI. Scopia became a thing. Scopia got split up into containers, image, and a wrapper. Antonio, you're not nodding along. Is this not how it happened? It is how it happened? Okay, great. Super cool. Run C became a thing because it got split out from Docker and everyone started using Run C to run their containers. Cryo with containers, image, and Run C started being a thing that could run images, well, the current containers for Kubernetes. We added container storage underneath Cryo so that it could actually not use up as much disk space when it was running your images. Openers have got interested in running Cryo and in 4.0 there's like, yeah, we're gonna do it. But it needs to be able to build and Cryo as an implementation of the CRI very carefully limited in scope does not offer builds. In fact, it doesn't have commit, which means you can't even fake builds. So we needed something else for the Cryo users. So Builder became a thing. And then we realized, oh yeah, we have all these libraries that we can put together to change fundamentally how we do builds in OpenShift. So 4.0, first pass. OpenShift Docker build now uses Builder internally inside of the build container where it's invoked to do the build. OpenShift STI build uses the functionality that Source 2 Image has to produce a Docker file, which we then feed into the exact same logic for building the Docker image. Well, for building a container image in the container, looking at Danny's, give me the stink eye. Builder no longer remote. This is the important thing and it's the biggest takeaway for what we changed for 4.0. And well, since the host no longer knows what's going on, well, we need to do the pushing of the image ourselves, but we already had that capability. So now that that change happened, some stuff broke. The first thing was all that new build logic had a lot of dependencies, like a lot of them. We do a lot of their development remotely, but I'm pretty sure that first PR, at least one OpenShift maintainer got up on their chair and ran out of the room screaming. A lot of those dependencies are Linux specific. An OC traditionally has been a command that ran on all kinds of operating systems. Oh, okay. And since that logic was Linux specific and it was a ton of new dependencies, OpenShift Builder became a separate repository. So OpenShift, STI Builder and OpenShift, Docker Builder are no longer the exact same binary as OC. That changed, but it made the testing a lot easier. But building it in a container also has other side effects. In particular, cache layers from successful builds just aren't there anymore because they go away with a container that built them. Cache layers aren't available if your build failed and you pick up again. This is the thing that people are noticing and they're not pleased. So a successful build, retrying steps, also don't get to be recycled or reused. Images pushed aren't available instantly on the host because they were never on the host to begin with. They were always inside of your container. This is a slowdown in deployment. Also, because the image is being pushed from inside of the container, it doesn't have any of the configuration that you have on the host. Recompressing the binary in layers that we've already pulled down can't waste time. I was just in your talk where you mentioned this. And just happy to, I too, I think. And yeah, there's a ton of optimizations that could be done, things that we used to get for free when all the builds were done on the same host, even if they were being done on behalf of users who didn't like each other. And builds are still run privileged. That's something we're gonna be tackling at some point, I think. So before, the other big thing is that OpenShift 4 is very opinionated about how you run your cluster. In particular, it's got takes. One of them is you can't depend on the node's configuration or reword it. You can't force your configuration problems onto the node. So we needed a way to get things like CA configuration and build configurations into the build container. So now it's all in config maps. Config maps are or will be available cluster-wide so you don't have to configure them for every project and namespace. I wish Adam were here. Then he could tell me exactly where we are on that one. So what's next? Bug fixes. We need a couple of bugs that we need to fix in the last and in the next few weeks before 4.0 goes out. Multi-layer builds aren't currently enabled because we're still doing things the image builder way. And when we tried flipping that switch, we found a bug in build that we need to fix. There are also some bugs in, well, just regular bugs that run into, like pulling images when the base image isn't fully specified. That one's boring, but I needed to have more than one thing in the list. The next big thing is unprivileged builds. I would really like us to get to a point where the OpenShift build is not in a privileged container and Dan is not gonna like that part because when we were talking earlier this week about cache layers, he started spinning off multiple ways to share those layers, which are not gonna work once they're unprivileged. And there are also experimental things, which come and go. And I'm sure you all have some ideas too. So this is the pointers to the packages that are involved. Are there any questions? Anyone? Okay, came in on time. Yes, this one. This one? Sally. So the OpenShift Docker build, will that be a name that just such a, it's in the ring so it can't be named? Well, the name of the command that the build pod invokes is controlled by the build controller component, which is part of OpenShift. So we could rename the command if we wanted to, but the name directly reflects the name of the build strategy, which is Dockerfile-based. And we didn't change the name of the Dockerfile just because we built Builder. So. We could call that we should Dockerfile Builder. We could. As long as it was coordinated between the two repositories because the build controller that creates the build pod spec is still in OpenShift origin, whereas the binary is built out of OpenShift Builder. So as long as it was coordinated, the name can be changed. I don't know that there are plans to do it right now. The build happens. From the cluster. Yeah, well, the builds happen wherever the cluster schedules the build pod. That hasn't changed between 3.11 and 4.0. In 3.11, the build pod would then call out to the copy of DockerD that was on the same host that the pod was being run on. So if you then tried to run your new image on that same node, it would already happen to be there. Whereas in 4.0, we push it to a registry and then the container goes away. So whether you're trying to run your new image on that node or another node, they're still gonna have to pull it down from the registry. Is that as far as all coming out? 4.0 is. It's gotta be out by the red net, so. Right, it's supposed to be out about two months before. Oh, awesome. Okay. I wish I had the schedule memorized. I probably should. You can go to github.com slash installer. There you can read try. Also go to try.Oneshift.com and try it out. Yeah, the installer is in Overshift slash installer and I highly recommend you going and checking it out. You can run it in Livert or AWS right now, most of you know, but those both of you are opposed to this. Yeah. And by way of a demo, I was about to kick off a build so we could look at the logs and how they're different, but, oh, Star Build. Were there any other questions? Okay, thank you very much.