 To close out a great day of talks, we have Will Christensen and Jason Plumb. Will is a DevOps engineer with a background ranging from software tools development for non-destructive evaluation at ASIC to Linux administration. His experience with development on scientific engineering and manufacturing software, coupled with an affinity for command lines and VIM, have led to a life focused on automation. Presently, Will works as a cloud engineer at Change Healthcare. Jason is a staff engineer on the distribution team at GitLab, where he focuses on providing a large number of deployment methods for GitLab the product. He's worked extensively to build GitLab's cloud native helm charts along with GitLab's Kubernetes operator to enable the ease of use and deployment that GitLab customers have come to expect but the power of configuration and orchestration that Kubernetes provides. Today, Will and Jason will be taking us on a journey into the realm of container build optimization. With a talk entitled, One Layer Builds Dock or Not Included. Containers can bring a number of benefits but the way containers get built can quickly increase your technical debt. In this session, Will and Jason will walk us through step by step how to build one layer containers and will share the benefits of doing so. Hello, everybody. Thanks for coming out today. Today, we're going to be talking about One Layer Builds, specifically Dock or Not Included. So first off, your presenters today. I am Jason Plum. I am a staff engineer, distribution and GitLab. I'm Will Christensen. I'm Will Christensen. Also, I'm sorry, a DevOps engineer and open source contributor. And just like Jason, we also mentored people coming up and just trying to make the community a better place. So let's do a quick dive into what the actual issue we're going to talk about. It's fairly simple. It's the big oof. And in that regard, what? Yeah. So when things get heavy, there are problems. And Docker containers, well, let's just say containers in general, they're stacks of layers in the end. And the problem is when the cat gets too big, you need to stop feeding it so much lasagna, especially as thick as that apparently was. First off, a lot of people don't use multi-stage builds. These allow you to build things in stages and only put the necessary result into your end container. The next one is some layers are kind of fat. And then you go and build on top of that. And there's nothing you can do to trim that out by the nature of how layers inside of containers work. Think of it as checking in a tar ball of all of the images for a game into a Git repository. You think, oh, it gets compressed. It's not so bad. And nobody will need it if they don't do a Git pull. Well, now it's in the object. And even if you do Git RM, the moment that you pushed it anywhere else, everybody now gets a copy of that for perpetuity until somebody rewrites history to take it out. And in the case of an image, when you rewrite history to take it out, you're rebuilding the image. It's not the same image anymore. There's no way to take that out, period. Then, well, you've experienced this one personally, so I'll let you take the lean. So the impacts for how things stack up become a major pain in terms of later on in the container build process, you can start dealing with people trying to remove layer or remove something that they thought from the actual container itself. And it won't work. And over time, there are certain inefficiencies that stack up. And if you don't watch out, for instance, if you were building a Java container, you may see a container sometimes as large as seven gigs and many, many layers. The registry costs will balloon over time. And the biggest issue that we've seen is when people have their pull policy on their Kubernetes cluster set to always, well, every time you make a change and you include in your build process, you were paying a lot of money for your network transactions, your storage costs. But most importantly, your time is taken up because the larger the containers are, the longer it takes to populate in the registries in all the clusters downstream. And just as a simple example of what we're talking about being fat layers, here is dive and we have basically pulled the default Ruby image, the public Ruby container image. And it's eight layers. Eight layers doesn't seem like too much, sure, but it's 881 megs. Just for that. If your runtime doesn't need everything that this might need, this could be 80 megs too much, could be 400 megs too much. Hard to exactly tell. And if you look at the line right here, primarily they did apt get update. There's more involved, no argument. But that's 528 megs. That's a lot. So let's look into what we can do about that. First off, let's look at what it is to actually build one layer containers. Right? So if anybody's familiar with building containers, have you ever seen from scratch in a Docker file? This actually means zilch zero, give me nothing, give me a blank layer. And that is my base. Now, the interesting bit here is because you're working effectively with a completely scratch empty, nothing, nobody else has already preemptively put all this stuff in there for you. You can have a pre validated tar ball that contains a file system. You can bootstrap it from a distribution using debut strap or pack strap or even YUM or DNF with a remote install option. If you need to, you can basically copy your full running shrewt that you've been using for years as a production application and just pop it right into a container. The beauty of this is what comes as a result. Well, yeah, I was going to say the biggest benefit that I have seen is when you use things such as app get pack strap or the remote install is you can go directly from a repository. You can put it into the folder and you don't even need to populate even a temporary container with any of the cached files. And when in that side of that container that you are building, you don't even need to put the package manager, which reduces one more piece that may be a security vulnerability downstream and when you get to production. Who knows? Maybe there's something in those default packages that you don't actually need that requires Pearl or all of Python. But in the end, does your application need it? Step two, as a result, when we're talking about one layer builds is very simple. So let's talk a little bit about the tools. This one is build up. Yes, it's pronounced that way. It came out of Boston. It's build up. Okay. Yes. You can find more information at Delta.io. But the key things that you really want to take away from this is that it's specifically designed for building containers, not running containers, building containers that can be done with shell scripting or the CLI, or it can actually be used with existing Docker files by just telling it, you know, built from the Docker file. But the documentation for building from scratch with this project is actually very sound, very straightforward. If you have questions about what really building a container from scratch is, whether it's in Docker or with Builda, you can find solid reference here. Will, from your experience, we're going to cover the next few. Sure. So most important is for me is you can use Builda without ever touching any domain specific or some domain specific languages. You don't need to know much else other than do you know a distro? If you understand the packstrap process and you understand how the builds would go in any type of shell scripting, that is very easy to make a container. Just remember to clean up your cache files and you can make very small containers. And with only one layer, you don't have to come back and you're not haunted with previous previous efforts with the one layer containers. We are all sorry with Builda, they're all OCI compliant containers. So if you want to do a Docker pull, they are completely compatible in many of our builds that we do for open source projects, for work, we are able to build directly from Builda and use it in our Kubernetes environments with various runtime engines without a problem. Oh, and most importantly, if you are a Docker user, please consider looking at Builda just because working with the manifest for handling multi-arch builds is an absolute dream. And this is, this is where we go. No, really from scratch. And let's point out a couple very specific things. And well, you kind of touched on these a second ago. First off, because you're not operating in domain specific language in how a Docker file is declared and how these functions behave, you get less opinionated tooling, but nothing against the way the Docker file is implemented in any way shape or form. But if you're familiar, as say, assistant men and building out either existing jails or shrewds or just even a working file system directory for an application, then you actually know how to build the working directory for an application. That skill set is no longer, let me figure out how to do it in Docker file. It's instead, let me do the thing and then tell Builda to say this is a layer. It's the fastest way to go from that scratch, that bear file system into a container without having to necessarily learn anything more specific or have to worry about bumping into edge cases and optimizing over and over again until you finally get to that, that happy point that you think you can move forward towards production. And as I said, because that system administrator knowledge that, that SRE, that sysadmin, that architect view of how do I make this application function that you've got decades of experience doing, you have the ability to do these. You can integrate this skill set directly into cloud native with containers. It doesn't matter if it's OpenShift or Kubernetes or Docker or really anywhere you want to do it. Your skills are there. On top of that, we have the ability to actually accelerate the breakup of monolithic applications, not necessarily as simple as a large code base, but monolithic application suites. You can take something that you already know that you already got broken up and take just the parts of you that you need for that application to function and break it out into separate chunks. So you can build something that's got all of the parts in one. It's basically a big stack of bricks. And then you can start taking individual bricks and put them in separate containers. And your viewpoint of how did I build that whole application doesn't have to change. Only how do I ship this component for runtime? From a runtime perspective, for a get-up solution, you can bake, for instance, a Terraform container with all your providers built into them. And it makes swapping out those containers mid pipeline, very simple and very quick. We have noticed that there is a significant improvement with caching, because when you pre-bake all your containers with everything that you need, you can save about 20 to 50% of the time for your built. And if you are doing anything where you need to make API calls, you can specialize and put all the specialized libraries right in there to do each step mid pipeline very quickly. I have not seen this type of improvement, except for using multiple tagged runners, running our shell runners that have been built specifically for specific tasks. We also have the bonus of because you don't have an upstream for your image, you're not building off of, say, immediately Ubuntu 2004 off the top of my head, then you're not bound to whatever is defaultively in that image. You aren't bound to what may come up in terms of functionality or pre-existing users or items that are pre-installed. You can optimize it out of the box by having only what you actually need to operate your application. This makes it actually also very good because you can now do this doing early adopter. So say you've got a new piece of hardware or a new processor type that you want to work with. If you can build it, you can stick it in a container. If you can stick it in a container and you can get a runtime of some kind to do that, then you can get it on that hardware without having to do much modification from the viewpoint of the developer who's got hardware that's of a different type. And over on the right, let me point out, when I say because we're optimizing what's actually there, this is one of my images. I know. An example of as you come down through, I started with a tiny 69 megabytes, but then the next layer was another 111, and there's another 108. And then you get down here, oh my, what is this? Okay, by the time I'm down here, I'm adding another 100. This image is not small. This could be optimized to say at least. So as we are definitely trying to repeat over and over again, when you have one layer, you don't have to pay for the sins of previous work. Um, everything is always fresh. And it also means that you get better control over what you have and not worrying about anything that may be in the past. If you have ever had to, let's say, do a scan of a container using various tools that let's say something is flagged, but was deleted in the layer above, I have experienced being, I was being held back from using that project and trying to figure out, well, how do I go back and deal with that particular issue? And if you have to wait for a vendor container that has that violation, or sorry, that issue, you are pretty much just sitting around waiting for something to happen. When you have full access to one layer, you can address it. You get control with the one layer build. If you must, you can build your containers from the same thing your systems would. So if you have, let's say a capsule, internal mirror, anything for repositories that would use for normal patching or image builds, you can build a container from that directly. And that includes also internal libraries that you may need to build. You can include that all into a single build pipeline. Much easier to remove contents from a layer than it is to edit a previous layer. We know theoretically how we're told to remove binary from a previous layer. I haven't seen it done in practice yet. And from what I've read, it doesn't look very easy, and I don't expect most places to figure it out unless they're just really curious. And most importantly, I have noticed with one layer builds because we don't have to worry about any type of masking of previous layers. I know Jason and I talked about AUFS. There's a masking way. So if you try to remove something in higher layers, you may still pay the penalty for the space it's taking because you're not removing that. And it allows you to keep everything just neatly coupled. And when you go to compress that image, it will get the highest compression I have found for that image. So all you have to do as a contributor is just make sure are your caches cleaned up and are your temporary files for whatever you need to do inside the container removed. And let me tell you, when I've had to work with very complex images, whether in my day job or in the community work, I love dive. But if you've ever had to actually use it because you're dealing with an image that's been built on top of an image and on top of another image that's on top of another image that then gets to the application, trying to find out why there's a weird permissions problem. That's not a fun process. Dive is awesome for that. And we love dive. But you don't want to have to do this on a regular basis if you don't have to. Remove the problem from the start. Yep. So the nice part here is this can all be done in CI or CLI. This is not a point and click. Because it's not a point and click, it means that you can do this through whatever source of automation. We will later demonstrate how you can do this as a part of GitLab CI through our pipelines and using the registries that are available. But the nice part is you don't need to do anything horribly special. You already have all the necessary runtime available to you as an engineer, as a developer, as a contributor. It's there and you have access to it on any system that's available in the public right now. There's nothing that binds you to require a license or anything else. This is all open, usable and extremely accessible. On the left is literally the first container I ever made with Builda. And other than addressing in that container how to get a more minimal version of Bash, I still use it. And it's great. So if you're afraid of building containers from scratch, I started down this road and I found, at least with Arch Linux and Arch Linux ARM, it made the process very simple. So if you are restricted from using Arch, then you would replace line three with your Dev Bootstrap or your YUM or DNF remote install. That will essentially build the container that you need. With packstrap and Dev Bootstrap and multi-bootstrap, you can install into the container without worrying about a cache. With the remote YUM install, you do need to clean up that cache afterwards. But other than that, same process, pretty much every distro, this is where your sysadmins will come and all their experience will reward you with nice, simple, one-layer containers. Oh, and a major benefit is that as I talked about previously with the pipelines, if you need to make specific containers, you make one base container that has done nothing more than build a and build a. And after that, you can put on whatever you need to either do the job or to like, you know, to talk to infrastructure or you can make specific building containers that can build the artifacts that you need. Well, let's go ahead and show them how this works. Well, all right. So for the demo, we're going to build a simple pipeline. The pipeline is just going to be a very basic Ubuntu container. Please ignore the fact that we, of course, right before any talk, there was some work that we had to do last minute, but should be simple now. So we're going to go run the main branch. This will be renamed when you go to see it in your shared repository, which will be hosted on our Buzz create repository in GitLab to simply go run. Now, I do have to preface the fact that this runner is sitting in my friend's closet. It is a one node Kubernetes runner and the base for it is also built in this particular project, which will, which is a focal runner. And we're just going to wait for the runner to wake up. I have a friend that has this runner working in his closet. It is a one node Kubernetes cluster. Anyone can build this. The runner itself is been built from Ubuntu focal. There will be an example for what it would be like to do a pull from Docker hub for Ubuntu focal and then to build the GitLab runner with it with multiple layers. And then there's what our build process does for rebuilding the container. So once the runner wakes up, the duration is 45 seconds. And on internet, on consumer internet with consumer hardware, this is quick. Let's go take a look at the, at the size of the container. Now, this is just the bare bones container, no optimization. This is also using G like C, not muscle, no, nothing special. And it's 52 mix previously, there have been some optimization efforts upstream with Ubuntu, which by the way, thank you, keep it going. Where the size of the containers were a little bit larger, but now this is just the pretty much the regular one. And that's it for the entire demo. So having listed the demo, let's do a quick comparison between a simple build from Ubuntu. Now, mind you, we're not using the aforementioned slim images that are now made by Ubuntu and Debian, Fedora, et cetera, major distributions. But let's just look at, you know, build up from basically scratch, right, on Ubuntu 2004 and build up from scratch. Will? Yeah. So the left is pretty much everything from the GitLab CI.yaml, you know, our runtime environment or our build pipeline file. And I took all that through then to a shell script, just to build an initial container to get started, which is something that if you want to use this process is someplace where everyone starts. However, once I build that container, I run the pipeline and I reference the manual container to build the next one, which is on the right. After you build it once, you can build it again. And it's pretty much a chicken-the-egg problem. But once you're in the cycle, it's easy. You click a button and you've updated your container. So when you hear, hey, there's a CVE vulnerability, you go, oh, is it fixed? Oh, cool. It's in the repos. You click a button, minutes later, it's available, downstream is already taken care of. It makes my life as a DevOps engineer so much simpler. So what we want to do here is let's understand that we're proposing the concept of a one-layer build. That's not going to fit everybody. That's just kind of how it is. It's just like one size fits most. The most is the keyword there because some people are not the same. So a couple of counter cases to what we have going on here. But one, for example, happens to be the GitLab Cloud Native Container Suite. You're deploying a very large application suite. And if we have a common base, then every image that gets pulled to any given Kubernetes node actually now reduces the amount of time it takes to download the next image that might be running a different container. So for example, if you have our Rails code base that are serving the web assets and the API, those have a common base with the Giddily service and the Redis containers, right? So we can optimize for our runtime of our users by using a common base and then adding only what we need in as few layers as possible above that. Another obvious example for when you cannot use a one-layer container is if you're using Iron Bank, if you are in a highly regulated environment and you must use a regulated container, obviously certain solutions are not available to you. And while you can get things certified to go into Iron Bank with the solution, you're dependent upon what you're dependent upon. And there's nothing we can do to change that. Another particular case is certified Red Hat images that are included as a part of OpenShift on-prem. If you're going to have a certified operator that's going to be included directly in the OpenShift marketplace, that container has to be UBI-based. If you're going to do that for other containers that are going to end up in the Red Hat container registry that's immediately accessible as a part of OpenShift, they too need to be UBI-based. So you can work from a base image and then limit the number of layers that you put on top through a combination of the techniques that we've spoken about before. Now that we've put the caveats in play, let's make some additional benefits that are inherent to this approach. First foremost is the supply chain attack prevention. If you are familiar with certain security appliances, you know that if you leave your build infrastructure, if you do not audit very closely your build infrastructure and if you aren't aware of some of the latest techniques, your build infrastructure may be at risk. One of the best things you can do is make your build infrastructure more ephemeral. Constantly rebuilding and being able to get the latest to your build infrastructure will help you. The benefits I really enjoy is the great visibility and auditability. I don't have to tell anyone it's magic inside of the containers, I can show them. We can have discussions and it's really nice to be able to work with people that may have an issue and we can, you know, if the upstream distribution has an issue, we can build our own slipstream right into it, still want the layer for the container. No one's an EUizer and everyone's just happy. Future proofing. When your hardware changes, as long as your build chains are there, as long as your repositories are supporting that particular hardware upstream, you can take that by just simply getting build up or building build up, just trying to see if it works. And if you can build what you need with build up, you can build all your containers just by with the same knowledge you know of the distro that you're using. And finally, security scanning. As I mentioned before, with fewer layers, you fewer origins, fewer things that you need to validate before you release. And by making sure that you have the reduced attack surface and greater control, you can make sure that you can maintain the integrity of your containers. And with the ability to rebuild fast or quickly and simply, it also helps you with once again, that supply chain attack prevention. So let's give you a quick summary of the things that we've talked about. First off, save time, both for you and for your consumers. Save space, both for you in your registry, in your storage costs, but also for your consumer, because the less they have to have transferred to where they're trying to execute these things, the less it costs them to both in time and trends. And savings or savings, save money. So simple answer is 42. Four things I want you to understand in two ways. First, to minimize your images, minimize the layers. To minimize your layers, use fewer. To use fewer, make sure if you're using more than one layer, use multi-stage builds, whether it's in Docker or it's in Billa, please do this. Second, and final, if you can use one layer. And it's really as simple as that. Special thank you, Zach, Emily, and John, all of you not only helped out clutch to make this presentation, I can definitely say time was spent out of normal working hours and very late for every single one of you. So thank you. Could not do this without you. And with that, we want to say thanks everybody for coming out and feel free to join us at the happy hour chat.