 It's gone quiet, so I think that means I'm meant to start. So I really was not expecting to be in this room, and I'm kind of astounded there's so many people in this room. Are you sure you're in the right room? You're laughing, but. OK, so my name's Adrian Mote. I'm a technical community advocate, or DevRel, if you will, for ChainGuard. At ChainGuard, we build minimal low-CVE container images. So this is one of my favorite quotes about containers. It's probably a decade old. It's by Brian Cantrell, who is a, you know, if you ever get a chance to see a Brian Cantrell talk, do. He's always amazing. But he said, Docker is doing to act what app did to tar. So way back in the day, we used to use tar balls to ship software around. You know, you just put your stuff in the tar.gz and send it around. And that was OK. It kind of worked, but there was always problems with dependencies that would work in my machine, but not in somebody else's machine. And then we started using things like putting stuff more stuff into the package managers and Linux distributions. And that was a lot better, because our package managers would take care of figuring out the dependencies and so on. And then Docker took it even a step further by packaging everything up down to the operating system layer. And that got us much better reproducibility and solved this. It works on my machine problem. So that's kind of what that quote's getting at. But even if you look at your package manager formats, your devs, your APKs, your RPMs, et cetera, if you squint, they're really just a tar ball plus a little bit of metadata around versions and dependencies. There's not much else. Oh, two. And it's the same with container images. They're really just packaging for an application. It's just a file system and some metadata. And theoretically, at least, they're not that difficult to create a container image. There's even standards documents that tell you exactly what files you need to get in there to get a running OCI container image. So you might think, well, everybody will create their own container build tooling. But we've not really seen that happen, or at least we've not really seen people create good container build tools. We've maybe seen a lot of build tools, but not a lot of fantastic build tooling. So what are some of the things we would like to see in an image builder? Well, ideally, if you want to your best of image builder, it's going to be able to build a minimal image. So what do I mean by that? I mean, our image should only have our application and its immediate dependencies. I don't want lots of operating system components, other things bloating the image. It should be fast. You don't want to be waiting a long time for images to build. Reproducible, that's an interesting one. So this is kind of a sliding scale. So Docker's fairly reproducible. Even at twice, you get more or less the same thing out. But you don't get something that's a bit twice reproducible. As in, I don't get a binary that if I compare it, I'll get exactly the same thing bit for bit, because there tends to be things like file time stamps. And the UC build IDs, I think we solved the build IDs one. And then there's some other points that may or may not be important to you, like simplicity. I mean, typically I'm going to prefer a simpler build tool. But if you want to do something like organization-wide, you may have a platform team that's responsible for your build tool, and then therefore simplicity may be less of an issue. Some people may really care about S-forms. I suspect that might be a bigger thing in the future. I had planned to talk about S-forms, but I don't know if you've read the abstract from my talk. I put far too much in there. So that's one of the things I've trimmed a little bit. Generic, so another thing we see with build tool and is you get things designed for specific ecosystems. And we'll see that in a minute. But you also get build tool and design to build anything. So this is where we came from. So if you were using Docker 10 years ago or eight years ago, then you did a Docker build something like this. And this still works and still a perfectly valid way to build a Docker image, especially if you're just doing something locally. So in this case, we're just using the upstream Golang image. We're compiling our Go application and we're setting an entry point. And that works. As you're probably aware, the problem is that we've still got our build tool in that image. So our final image doesn't just have our application. It's got all the build tool and it's got all the stuff in the underlying Debian operating system that we don't really need to run our application. And ideally, I would like to get rid of that because it's just a source of potential CVEs and problems. Yeah, with Docker, we still have this problem that's not entirely reproducible, but it's fairly reproducible. And then we got multi-stage builds. I can't remember exactly when multi-stage came out, but that was really a huge step, in my opinion. So if you combine multi-stage Docker builds with minimal runtime images, then I think that's actually one of the best ways still to build container images. So in this example, I'm using Chingard's Go image to build our Go application. I could as well have used the Go image from before. I am doing ZGo enabled equals zero, which will give me a static image. That static image I'm copying over to the Chingard static image for the runtime image container. And in this way, I end up with a very small final image that doesn't contain all the build tool, and et cetera. And it'll just be a few megabytes in size. In this case, I've used the Chingard static image. You could also use the Google Distress image. They're honestly fairly similar. Some of you might be saying, well, could you also use Scratch, like the completely empty Scratch image? And that will work, in some cases, during a static binary. Where it falls down is that a lot of applications in Linux require or expect a few more things from the host environment. For example, TLS certificates. It's a really common one. Like if you want to talk to anything securely, you're going to need TLS certificates in your container. Other things include a user. Some applications expect a Linux user to be present. Slash temp directory, things like that. And that's exactly what the static images give you, just enough to run the average static binary. So yeah, that's a huge step forward. You're typically not going to have any CVEs in this image, unless you're pulled in CVEs where your go dependencies are something like that. We're still using Docker, so there's still some issues around reproducibility. The other thing is you need to have the example I showed there work for if you can compile a static binary. But say you have a Java, a Node, a Python project, then you need to use a different image, a minimal runtime image for that stat. So if you look at chain guard images, we do have minimal runtime images for all those. So go check out our images if you want to use this approach. Google also have several more. But the example I showed there was with Go. And there's actually an even simpler way to get a minimal image with Go, and that's to use this tool called KO. And in that case, there's not even any config files unless you want one. You can literally just type KO build in your Go project, and it's going to build a minimal image container with your Go application. And I think it actually uses a chain guard static image by default. So if you're using Go, that's definitely want to check out. But you might be thinking, well, hang on, you're talking about these distrueless images. How are these distrueless images made? And can you use the same build tool that's used to make the distrueless images yourself? And the answer is yes, you can, but you might not want to. So Google Container Tools distrueless is built with Bazel or Bazel, whatever it's called. And what they basically did was take Debian and hack everything out of it to get a minimal image. Chain guard distrueless images, we have our own build tool called APKO, and we use the Wolfie operating system, or Linux distribution, which is our own Linux distribution. OK, I'm going to try and do a few brief demos and see how this goes. OK, so first one is Bazel. Am I in the Bazel directory? That's the first thing. We're in dagger. That's not going to help. Can people say this OK? Make it bigger? It's good, OK. One person talk, make it bigger, OK. I think the problem is, is that the bottom? OK, so this is the example. Bazel examples from a company called Aspect. Aspect are great. They do lots of stuff around Bazel. Bazel is phenomenally powerful. It's also phenomenally confusing. Here's the build file. I probably shouldn't say that. It's just very powerful, and well, my feelings will become clear from this example. So this is the build file. It's actually reasonably straightforward. You can see unsurprising things like this, telling you what the sources are, what we're going to build, the libraries, the binary. We're going to put in a tar file. The OCI image uses a tar file. It also uses a distro-less base, and that's kind of it. Interesting, well, another bit is we have a modules file. And because there's a lot of had big focus on reproducibility and supply chain to Bazel, you have a modules file that specifies the versions of everything. So that's up there. And notice we actually pull in the Google distro-less base image here and specify the digest. And that's kind of interesting, because ideally you would think in Bazel you'd want to build everything from scratch for supply chain reasons. There's not any really good reason why we'll pull in an external base image rather than creating one here. OK. Now, I've built this before, so this should be fast. There we go. So yeah, it's very, very fast to rebuild things. The first time you build something, it does take a little bit longer. And that's built. This tar file, it is a Docker image. I can do this, and it will load it into Docker. I'm not going to bother running whatever the image is. I'm just showing it as a Docker image. One thing that I, and it is also very small, LSDashLH, so that tar balls 10 megabytes. So we are, again, minimal images out. One thing I expected to happen that doesn't happen, I'm not sure why. So if I do Shasim on that, I get that. And then if I rebuild it, I get a different Shasim. So I assume there's a timestamp or something like that in there. I was expecting to get exactly the same Shasim, but clearly I've done something wrong or I don't understand something. I did try and look at this, so I typed this command. And this kind of sums up my feelings about Bazel. This is the help for the build command. This is probably a halfway. It's insane. It's still going. There you go. So you'll forgive me for giving up at that stage. So that's Bazel. It's very powerful, but yeah, it's also very complicated. But yeah, you get minimal images out of it. I think the big thing though is Bazel will build everything. It builds your Go code. It builds your Java code. It builds your C code and provides complete supply chain solution. It's not really just for building container images. So if you want to get into Bazel, you're going to know what you're doing, I guess. Oops, didn't click that. OK, so the other alternative is AppCo or APKO. I've never, like, we built it. I still don't know how to pronounce it. Yeah, we can do an example of this as well, which is hopefully a lot simpler. I do have some old build files lying around, but the only file of interest is this build.apkofile. So all we're specifying is the repository. In this case, we want to get APKs from the Wolfie repository. You could also use Alpine. You can't mix and match them because Wolfie is compiled against G-Lib C and Alpine's Bazel. But yeah, you can use the Iser here. And we're saying we want to create a container image with the packages Wolfie-based and C-A-C-Diskits bundle and also setting some metadata on the image. And that's it. You can't do anything else in AppCo. Everything inside your container image has to come from APK. You can't pull in random files. You can't. There's no equivalent of Docker run commands, for example. So it's much, much simpler. It's powerful, arguably. But OK. And then if I do AppCo builds, that's what you get. So hang on. Can I pull a hash? So what we're saying is AppCo build, specify the architecture, then the build file, which is build.appco. The name of the image, which is going to be Wolfie-based. So this image is literally just an image with a shell and not a lot else. And then the file that it's going to save out. So it's created this Wolfie-based.tgz file. Again, you can do Docker loads. That loads the image. Again, it's going to be fairly small. There's a little bit more in there. So I guess it's maybe 15 megabytes or so. Oh, six megabytes. Perfect. If I do Shasm 8 and 256. So I get this very nice Shasm in this case. It starts with the word added just by random chance. So let's RM that and rerun the build. I get exactly the same Shasm. So that is bit-wise reproducible. Of course, there is a little bit of a gotcha. Oh, not gotcha, but something to be aware of. In this case, I haven't specified the exact versions of these packages. You, of course, would need to specify the exact versions if you wanted this to be reproducible in like a few weeks time. OK, so that's apco. Yeah, simple declarative. We've shown it to be reproducible. You get a minimal image out. I've written Composers well with Dockerfile. So we use apco to build this to list minimal runtime images that you can then use with Dockerfile to put your application code on top. Because you probably don't want to be building an APK for your application code and using apco to create the final image. And you can, and it's not a bad thing to do, but I guess most people aren't going to do that. And the other thing is, of course, if you use an APK, you're dependent on multi-packages. You can create your own packages with either APK tools or Milange, which is our way of creating APKs. And interestingly, we also have a rules APK for Bazel that was created by Aspect, again, I think. And that solves the problem I mentioned before about Bazel pulling in an outside image. So if you use rules APK inside your Bazel build, you can end up with a fully hermetic build from scratch, which is nice. OK, this one is interesting. Canonical came out with something called Chisel Containers. This is basically their version of Distralis. And they've managed to create a very minimal. The two examples I've seen are .NET and GRE. I'm not sure they've released many other images. But once they have released, they're definitely minimal. No CVEs or low CVEs. They look pretty good. The way they did it is kind of interesting. They used something called RockCraft. So RockCraft, I think this is what creates the image. I couldn't actually run it because it's all Ubuntu and snap-based, so I couldn't install it. I guess I could have created a Docker image, but I wasn't actually that interested after I played with it. So this is going to create, I think, a Docker image. And this is basically telling it what packages we want to install. So this is saying, install the hello package. But you can also ask it for what it calls slices, which is continuing the chiseled net of force. So I guess you chisel out slices. And then you type something like hello bins, and it'll only install the binaries from the hello package, assuming the bin slice has been defined. So that's interesting. And the next question, of course, is how do you define these slices? So this is the example from the RockCraft tutorial. And we have an example of slicing up the open SSL package. So what we're saying here is I want to create a bin slice that only has open SSL, c rehash, and the libraries required to run that. So what we're saying is, yeah, we're going to need libc, libssl, some config files, and these two binaries. But I don't want anything else. So it's going to cut out a lot of stuff. And then it says it does need the config. So down here, we also have a bunch of config files that we're specifying in a separate slice. And my sort of issue with that is kind of right on the slide as well, because these are all hard-coded paths. So what happens when the APK changes a path for the names of file, presumably a slice breaks and you have to update it? So it does seem a bit manual and potentially error-prone. So it definitely seems kind of useful. They've had some good results with it. I'm not entirely convinced how generally applicable it is. Also, it very much ties into the whole canonical ecosystem with charms, et cetera. Buildpacks. Has anybody used buildpacks? Yeah, one or two. It doesn't seem hugely popular. Yeah, buildpacks, the main thing about buildpacks is kind of easy to use and automatic. So I'll pick up your config and do a build automatically. I didn't realize this, but there seems to be multiple implementations of buildpacks. I used paqueto, so I can find the sample. But I wasn't overly impressed with the defaults, I have to say. So yeah, there's no buildpack config here. It's just a simple Go project. You run it with pack. There we go. So I say in pack build, my application, using the paqueto Go build pack. And we want the Jami base, so it's using a Debian base image. I've run this before, so it should be fairly fast. The first time I had to download a whole bunch of images. There is some support for S-bombs. I don't think it analyzes the Go depth, so OK. And that's built my image, my app. Now, unlike some of the other ones, it's not built atarable. It has actually loaded that straight into the Docker cache. And my problem is that by default, it builds quite a large image. So it's like 121 megabytes. And if I do, I'm not sure if they're keeping it up to date, or it's just that base image has a lot of CVEs. But it really doesn't seem a great default. Well, the other thing that happened, and I haven't even figured out how to fix this, is if I try to run it, it turns out it's built AMD64 image, even though I ran it on an ARM platform. I kind of thought I would have figured out, hang on, you ran it on ARM. Let's build an ARM image. But it didn't. I'm sure I can't just pass a flag somewhere and it will fix that. The other thing I should say is, I've forgotten the guy's surname, but there's somebody called Ram that wrote for the news stack. And he wrote a great article on using Wulfi as a base image for build packs, which will solve quite a lot of my criticisms there. Anyway, that's build packs. I guess the big thing is I've shown it there for go. There are examples for Java, Ruby, Python, et cetera. And again, it's like zero or low config that should just work sort of thing. Buildkit and dagger. Yeah, so this is one of my favorites. So dagger is actually built on top of buildkit. If you're not aware, buildkit is the engine underpinning docker build. Interesting thing is docker files just scratch the surface of the power of buildkit. I think they always want to create better front ends, but they never entirely manage it for whatever reason. But dagger does try to take advantage of more of the power of the buildkit engine. Now, dagger isn't designed just for building container image. It's really designed to solve the whole CICD problem, where you're trying to debug CICD, but it works differently. You can't run it locally. Or at least it doesn't run the same locally as it does remotely. And you end up with 20 commits that are all like, works this time. And it always continues to fail, and you're pulling your hair out. So that's what daggers aimed at. Oh yeah, let's see the dagger example. So in dagger, the real powerful thing you can do is define your build pipelines in code. And you can kind of see there, like it's designed for assembling powerful build pipelines from multiple functions. In this case, if you go, there are front ends for various different languages. And what we're saying here is we're defining a function, build, and publish. And there's multiple modules that you can already use in dagger. So dag is the short for dagger. We're using the Wolfie module. So I think it was Solomon himself, or dagger anyway, have created a Wolfie module that creates a Wolfie-based container. So this dag-Wolfie.container is going to return a container object called CTR. And then in this step, we're calling the Golang module, and we're calling build container. It's going to build our Golang source, and it's going to build it on top of this container image that we're passing in. And then we're going to call publish, which is going to upload the image to a registry. Again, this is just the basic example from the docs. I've not written this myself. I'm very running. That was lucky. So to sort of call dagger functions, it's dagger call. So we're calling build and publish, which is the name of that function we defined a minute ago. We passed a couple of arguments to do with a build source and build args. And one really nice thing about dagger is it gives you quite a nice output, I have to say, and updates on what it's doing. It does everything in parallel and caches everything, et cetera, et cetera. So yeah, it's really quite a fun output, I have to say. And there we go. It's built that, and it's pushed it up to this registry. So I should be able to do, I should have changed this to, say, HelloCubeCon, shouldn't I? I didn't use a local bin. Hello? I don't know. Oh, no. Oh, it's because I didn't spell local, right? Also, my thing is going off the end of the screen. There we go. OK, so that's dagger. Very powerful. You can define things in code, but it's really designed for the CI CD problem more than just building containers. The real strength, though, in dagger is the modules. So they have something called a daggerverse, and people are submitting their own modules. There's modules for things like APQ and Wolfie already. So yeah, I really do recommend checking out dagger. It's pretty fantastic. And the last thing I want to mention is Nix. So I thought Nix was going to be a fantastic solution when I started this. Like, has anybody else looked at Nix? Put your hand up. How many people agree that it's like going down a rabbit hole? So theoretically, at least, Nix is a fantastic solution for building containers because it's designed to be reproducible, and it should be possible to create minimal containers. The problem I had is there's 1,000 different ways to do things, and everybody tells you a different way to do it. So I started down here at the bottom with the example from the docs. I never actually got that working completely because I think there was some issue with macOS that apparently is possible to solve, but I don't know how. Then there's actually a really nice blog by Mitchell Hashimoto, and he wrote about how he builds things with Nix and uses mixes that were Docker files. So quite an interesting approach. I don't think it's an idiomatic approach, but he has a lot of success with it. Then when I talked to somebody, they told me, no, don't do that. Have a look at this example that uses Flakes. I pointed me at this code, but it was more than I could get going. And then two weeks ago in Hacker News, somebody had a talk called Building Better Docker Images with Nix or something. And they had a fourth way of doing things. So there's four different ways of doing it, and I do not know which is the correct way, or if there is a correct way, or if it even is an idiomatic way. And that's my problem with Nix, so I'm not even going to show an example. Yeah, but I did find this great quote in the Hacker News comments about that last article. Somebody said, Nix, I love it, but sometimes it feels like being a morty on Rix's adventure to compiler land. And yeah, I really liked that quote because that was very much my experience. OK, so wrapping up, what would I recommend? I think it's like both Bazel and Dagger kind of trying to solve bigger problems than just building containers. So Bazel builds everything. It doesn't just build containers. It's going to build all your source code, et cetera. And I guess Dagger is as well. You're only going to use Bazel if you want, like, supply chain guarantees and things like that. I guess you're going to know what you get into when you start playing with Bazel. Otherwise, I wouldn't touch it. Dagger, I think Dagger's going to be pretty big. I think it's a really pretty nice solution to the CI-CD problem. And for that reason, it might end up being used in quite a lot of smaller projects as well, as it is quite easy to get going with. So if you have trouble with CI-CD, and I think we all do, then definitely take a look at Dagger. After that, if your project is just like a small project, and it's in a specific ecosystem like Go or Java, I would probably look to see if there's a specific build tool for that ecosystem. So if I had a Go project, and I just wanted to create a container image for it, I would just use Coal, because that'll work. And it's like zero effort, and it gets me a good result. I've not played with it too much, but there is jib for Java that I assume does something very similar. For a generic tool that you can use across multiple teams, et cetera, I would still be using multi-stage Docker builds with digitalist-based images for your run things. So please go and check out the chain guard, digitalist images, and see how they work for you. But honestly, for the majority of people, I think that's the simplest good solution, where you end up with minimal low-CV images that you can run in production. You're obviously welcome to play with AppCo. For most people, I don't think it's I think you're only going to want to play with AppCo if you have a specific use case that AppCo is good for, because you're going to need to put everything you need into an APK. Next, honestly, there's no point in putting it next year, because if you like next, then you're not going to listen to me anyway, if you're going to use it. So yeah, go ahead. Next probably is a fantastic solution. I just couldn't find it. And yeah, on that note, thank you very much. That was all I have.