 All right, folks, it looks like we're going to get started. Don't worry, it's a fairly short talk. There's not a lot of new information here. So thanks for showing up. Hope you're all still awake after lunch. Anyway, my name is Nolan Dyeboy. I work in the containers team over here at Red Hat. We're not over here, over there at Red Hat. And primarily, I'm working on building images and build in tools to do that with some of these really fun people. So today, I'm going to be talking about building images for multiple architectures, which is a thing you're going to want to do if you want your images to run on something other than the AMD64 style boxes most of us are probably using. So to give a brief rundown of where I'm going today, I'm going to give you a very quick summary of how we do things now when you're building. I know Scott, earlier in his talk, said you should always be using a cloud to build your images, which forget he said that during his talk. So once you push your image to a registry, I'll give you a quick rundown of what happens when you're trying to pull it from a registry. And then go through the differences, well, how it's different when you have images of multiple architectures in that registry. Then I'm going to show you how to build the necessary information needed in order to supply that registry with the information that a client will need. And then we're going to run through a couple of quick examples. And actually, I need to start one of those right now because it is a capture of a script that I ran last night because running the actual script over a slower network would be a really bad idea. So we'll see. And actually, the capture takes 12 minutes or so to run anyway. So I'll come back to this actual script later on in the talk. Right then. So you're familiar with the process of building an image of your own. You build the image. You push it to a registry. You either give it a tag name, or you remember the digest that it got when it got there. Then you tell all your friends about this wonderful image you built, and they're going to pull it down and run it. You probably want to delete all the evidence from your actual hard drive. So when they go to pull the image from the registry, their client tells the registry, hey, I would like to manifest for this version of the image, usually the tag or the digest, from this repository in you, the registry. And the registry supplies it. Now, what's in that manifest? Basically, it's this. It's just a list of the things that make up the image that it'll need in order to reconstruct the file system. So going back, the client pulls down these things, builds the file system, sets up the namespaces, gets your container running, everybody's happy. So what's different when you've got multiple architectures in your image? Well, the stuff I've highlighted in green. The client asked the registry for the manifest for the image, and instead of handing it back the manifest for the image, the registry hands it back a list of manifests, which looks something like this. The important thing here is, well, this one actually only lists one thing in it, but you can see that it has the digest. Is this a pointer? Yeah, okay. The digest of the manifest of a particular image. And the platform is the interesting part because that is designed to let the client select what it thinks is the best image for it to be running if there were more than one thing in this list. Right now, there's not a whole lot of options, and it may just say, you know, I can't run this, and that'll be the end. So once the client has selected one from the list, and assuming it finds one, it's okay, it then goes and asks the registry again for the manifest for that digest. And then the process continues as it would have before. It builds the file system layers, launches the container, and for the most part, everything works. Everyone with me so far? Good. So if you're building images for multiple architectures that you wanna push to a registry, this is what's changing in your process. You're gonna build the image, you're gonna build it again, you're gonna build it again, you're gonna build it again, you're gonna keep doing this for every architecture you've got. Now, you can push it from your nodes where you're doing the builds, or if you happen to be putting them on one host, you can save it all for later. You do need to keep track of some information about all of these, because then you need to build that manifest list. Yep, and then once you've done all that, you build the manifest list, you push the manifest list along with the images to the registry, or if they're already in the registry, you just push the list by itself. Then you tell all the friends about the tag or that you applied to the manifest list or the digest of the list, and then, again, they're gonna tell their clients to pull that image, and they'll lean on the wiser, it'll just work. So the tool that we have for building manifest lists is the new build a manifest command. This is very similar in model after the documentation for Docker manifest if you've seen it on the web. There are in various steps, you create a new list, you add an image to the list, add actually takes some options, but hopefully our goal is that you'll never have to actually manually supply these, because we'll be able to dig out all the important information from the images that you're supplying us. We also keep track when you add an image to the list. We keep track of where it is, so we can find it later if you wanna push it to the registry along with the list. Manifest push has two options. Actually, the main important one is dash dash all. If you don't supply dash dash all when you do a build a manifest push, we just push the list. If you've already got your images in the registry, this is the fastest way to go. If you don't already have them there, push dash dash all will transfer the contents of the images to the registry, even if they're located in other registries, whether they're on your hard drive or wherever, and then build RMI's, how you clean things up on your host. So yeah, particularly the note at the very end is that the only place where we don't currently detect everything correctly is ARM, we still have a bit of work to do there. So there are more or less four ways that I can at least think of that you can build the images. The fastest way is definitely going to be use the hardware that you actually have for that architecture. Folks at Amazon love ARM machines, so for them they can do that. For the rest of us, I only have one laptop, you can run a VM for a different architecture on your machine. It is gonna be kinda slow, but it will work. You can also cross compile on your machine and then use build scripting to copy the binary into an image. Just be careful you tag the images architecture correctly, otherwise things will get confused down the line. And the fourth one is a little hybrid. You install QAMU user static or QAMU user magic, I should call it, you know? Anyone? Okay, fine. Anyway, what this does is it lets you build an image for one architecture, but emulate when you run commands, it'll emulate the necessary architecture just for the binary. So while the buildup process you're running is running native on your machine, anything in a run statement in a Docker file or in a build or run command is actually running in your target architecture. And this actually does work. It's faster than running cross arch, but definitely slower than getting actual hardware. So this is a demonstration script for using build up with inside of a script to build from multiple architectures. Those of you who are proficient in batch will see that up here, I'm just creating an empty Docker file and then I'm just cycling through my target architectures, building it. And then once they're all built, I create a new manifest list, add the multiple manifest list. This is me getting around some of the stuff we haven't done correctly for ARM yet. Hopefully you won't have to do that in the future. And then I pushed them all. This actually works. I have pushed this image. So if you're running a non-exity 664 architecture, you can pull this right now and run it. And Dan, there's your fedora with a ping installed image that you would love so much. And that's all well and good. We can also, actually let me see how far along the playback on that one is going. Yep. It's in the middle of building the ARM of the Arc64 one. This is an emulated copy of DNF running inside of Builda, which is running natively on my AMD64 machine. So this is actually a slower step than the AMD64 one above it, and it's gonna keep going for a while. So let's just keep that going. That's all well and good and it's gonna get to the end. Let's do something kind of evil. The thing about manifest lists is there's not a lot of requirements placed on the format and you can do kind of some crazy stuff in it. So for example, I'll create one and I'm gonna call a cursed image and you'll see why in a minute. So people who pull this image on ARM64 will get Fedora. Perfectly normal. You can see I'm referencing an image that isn't actually on my machine. It's in a registry. That's perfectly okay. We'll download some metadata about it, add it to the manifest list that we're building. Everything's fine. People who pull my image on AMD64, however, will get CentOS. It's gonna be a little confusing. If they happen to be on PPC64, they'll get Debian. And if they're on S390, I still get Alpine. Nothing stops you from doing this. This will work. You should never do that. You should definitely not tell people to try to base their images on that. I'm gonna go ahead and do this live. So, let's see, open another shell. Okay. Let me put this down for a sec. So when I timed this earlier today, let's take about a minute and 20 to run. As you can see, it's adding each of the, it's created a manifest list and is adding things to it. And now we're pushing the manifest list. Now normally this would be very slow because it's going to have to pull down all the blobs for all the images that I mentioned, but because I also pushed it last night, the blobs are already present on the registry. So this push won't actually take nearly as long as it would if you were starting from scratch. Thank you. And that's the end of manifest push-ash-all. So let's see what happens when we run this image. Okay, that's running natively on AMD64. Let me switch over to a VM that I've exercised into, that is Arch. I'm sorry, that's ARM. Let me do the same thing. That one's gonna take quite a bit longer than my AMD64 which I'm switching back to right here. Okay, so AMD64 I got back sent to us eight. This is gonna take a while. Probably should have lined up something for this stage of it. Anyone here from Cleveland? No? Where's Scott? He said he's from, anyway. Any questions while I'm waiting for this to finish pulling in an emulator? Yes. Okay, the question is, do we plan to add these features and flags to Podman? Yes. Right now the implementation's a little rough in Builda. I'm skipping over some things where we've got bugs that we had to fix, but the medium term plan is to get manifest list building and pushing into Podman itself. Are there any other questions? Yes, Micah. Well, there's, okay, so Micah's question is, is there any work being done at the OCI level to make the manifest list or as it's called an OCI image index format, more secure or reliable? This, what I'm demonstrating here is not necessarily a bug. I mean, it's working as advertised. The criteria by which the client selects which of the entries in the list it's going to use is not super well-defined. And I'm actually, I'm not following OCI conversations about this very closely. So perhaps, like I'm going with the spec that we have now. Is Vincent here, would he know? Nope, maybe not. Finished building and running my container. Are there any other questions? No? Oh, come on. Yes. Sorry, could you repeat, I missed some of the early part of that. Yep, QM user static works and it is independent of your container runtime. Like it wasn't, I'm not even sure it was designed for containers. It works really well with containers though. For Podman, which doesn't quite know about all these things yet, like I'm using a couple of non-documented options right now to speed up the demo. Once we figure out how it's supposed to work, we'll document them in full. But if you manually look at the manifest list and pick out the digest for the architecture you want, you can tell Podman to run that image. And if you have QM user static installed, it will run that image. And that's how we actually proved out that this was actually worth pursuing way back when. And as you can see, having pulled the AR64 version of my cursed image, it now, the OS release file in it says it's Fedora, whereas on ANV64 it says it's sent OS. And I think you can see how the rest of this goes. And that's the end of the demo. So thank you all for coming. If there are any more questions, I have five minutes left. Yes. I have to confess, I haven't actually run Docker manifest directly myself. This implementation was guided by reading the docs on it though. Okay, so if it is a difference, I would like to clear that and clean those up. Anyone else? Apparently I had five plus time for QA, so like I said, it's a short talk. All right, thanks a lot.