 So, yeah, I'll start slowly and then first of all, welcome everyone to my talk and probably your final talk of today, so I try to make this easy and to relax, to sit down and listen to that. I also know I'm the last thing between you and the attendee party at the Guinness Storehouse, so this is a bit of an unfortunate situation to be in, but yeah, the other thing I wanted to say before I start, you might have always seen that, I have a bit of an injury on my left arm that happened on the weekend and the talk I'm going to give has quite a major part of a demo with it, so it's like a tutorial workshop, whatever you want to call it. I still have difficulties to type with both hands, so I'm not sure if I can do everything. I asked the conference if it would be possible to cut it down to a regular length talk, but then in the end decided I will try it that way, so we might finish a little earlier depending on how things go, but we'll see just as a disclaimer and I'm not making this up because my demos are so bad, unfortunately it really happened. Okay, so let's get started then. I also just figured out that I probably have to jump in and out of the presentation because I actually want to switch between screens where I have the CLI and wishes to your code and so on. This doesn't work when I'm in presentation mode, so bear with me, they will always take probably a few seconds to do that. I also have this top bar right there that I don't see, but you see, should be all right. So that's me, my name is Matthias, I'm from Germany, I work for a rather small consulting company called Novatech, I also have seen quite a few familiar faces here from the Linux Foundation, so I'm happy to be back on a stage and in conference with people, so maybe some people know me and if they know me they probably know me through my job at Novatech. What most people probably don't know, I'm also teaching at two diversities in the area of Stuttgart where I do things around distributed systems, while distributed systems is a bit of an old gray topic, so I try to do a distributed systems with modern technologies and containers are definitely a very important part there, so some of the things you probably see today will have its source of origin in the lectures. Now in a nutshell what this is going to be about, a situation probably many developers will be familiar with, to get your source code or your application into a container image and this is an entry level talk, so I'm not going to go through all the things, what is a container, what is an image, but I briefly introduce that and then I'm going to walk through various options and try to show what I think works better or worse with the different ones, also important to say I'm not bound or like affiliated with any kind of vendors or technologies, so we're not getting any money to say Docker is great or Docker is bad or whatsoever, it's more like I just want to highlight that there might not only be one solution and it might be worth to take a look at a few different ones because there might be better for certain purposes. So I've just mentioned Docker and it probably is difficult to talk about containers without talking about Docker, so the initial way most people will probably get to building a Docker container or like a container in general is a thing called a Docker file and I'm going to also use this as a bit of a reference, what can you do with it and what can't you do with it, what might be good, what might be bad and then have a look into various other technologies. I'm definitely going to mention things around cloud native build packs and Paketo, also look into open source projects from Google called chip and co and maybe say a few words about open shift source to image. Now for the format of this talk, I've never given it in this kind of style and I've also just recently added a few of some more things. I also offer this as a lab but I think 90 minutes might also might actually be a bit short for everyone to go through and compared to like I think the previous speaker here, I don't have a setup environment for you that you can basically log into and run. I mean if you want to follow along, you totally find basically what you need is a container daemon and for some exercises you might need a Java runtime and that's about that. But anyway, I will do the things live or I will try to do the things live and you are very well invited to follow along if you, but anyway this is not, if you want to play through the things by yourself, I just do that step out of the presentation real quick. So this is basically where the link goes to and there is like a lab walkthrough instructions that takes you to the various steps. It will also tell you what you need to do that. It's not much easily installable on any kind of machine and then you can just also watch now and play through the steps later by yourself if you want to. This is free and open and available so you can always do that. Alright, anyway, if you have your laptop in front of you and have the things set up, you are very well invited to do that alongside with me. Also, I would of course like to have this as interactive as possible so I don't have a super strict agenda how I want to go through this and where I place the demo parts and the theory parts. If you have any questions, please feel free to tell me and I'm always happy to do things on the side of if there's the possibility. Alright, so I think the very origin of this talk was actually a question from a colleague of mine which has already happened a bit of a while ago. He came to me and said, well, Matthias, you're doing quite a bit of things with containers and in their project, they need to build containers for the Java application and ask me what is the best Java base container, base container image, I'm sorry. And I'm a consultant, what would I answer, of course? It depends, right? I mean, I don't have a straight answer for that so I would of course want to know what are the things they're going to do and where is it embedded and what is the workflow and so on. But based on that, it kind of raised the idea to me to build this talk or build this evaluation as probably more people have that problem. So before we go into that, we first want to look basically quickly what is like a container or a container image or a container itself because that's of course a bit important if you want to improve your container image builds in probably to understand a little bit what is the technology around that. Now, I said I'm not going to lose much time there and most of the things have probably been explained today already. But in general, if you want to run an application, you normally run that on an operating system where you have some libraries and dependencies in between. Now, the initial idea of a container was basically to separate the application along with its libraries and dependencies from the underlight shared operating system stuff. And to basically make it portable and executable in various environments. Now, the idea of containers wasn't really to just build that container thing. It kind of evolved used coming from the technologies, namespaces, change routes and seed groups. I'm not sure who has been in that very first talk in this room today by Gerov. I was there too. That was really good. So that's also where I learned a few more things about namespaces that I didn't know yet. In general, this is just for you basically that everybody should be in the same page to say, well, if I use a certain kind of container daemon, this is basically a process or a server application or depending on which format it comes, that will handle all those things for you to work on namespaces, change routes and seed groups under the cover. So that you don't have to worry about that and have basically a need API to handle those things. So that's the part about containers. Now, container images are basically the templates or blueprints that you can have or use to basically run or instantiate as containers. So the container daemon will be able to load those images or grab them from various repositories and then start the container of whatever is given in such an image. And as we are looking, sorry for that squeezing, that's the stage and I can't stand still. So apologies for that. Now, looking into those images, this will basically be our most important thing because if we want to improve, build that the way our images are built, we should basically understand a bit about their structure. So in general, the way things started very much with Docker was basically that you have that concept of an image in the container with a run command, you wouldn't instantiate it and then you could do changes in that instance and with a commit, you could go back to a new image that basically would manifest all the changes inside of that image. So in case of a Java application here, you would say, I need a base OS container. I need to install the Java runtime. I need to copy that application. And then I had roughly speaking three layers with an Ubuntu Java and the app on top. This could of course get more complex and error prone of whatever type of applications you have installed. So Docker realized that very early. I tried to make a bit of a research on the Docker history. I think it came out in March 2013 and in May 2013, there was the first mention of this concept of a so-called Docker file. Now, this Docker file kind of works a bit like a script where you have different steps that you basically do. And then you summarize all that individual steps that I said before. So in this case, you would say, I have this base image Ubuntu 2010. Then I install the Java runtime. Then I copy a char file from outside into my container and execute the command. And technically under the hood, each of those steps would be like a new commit and a new instantiation of a container. And in the end, you have to finalize containers. And all those steps, I'm sorry, built the individual layers in your container. Now, if you do that, you could of course play around with these base images. So in this case, I took a plain operating system base image. That means it only comes with standard operating system binaries. Nothing prone to programming languages and such. There are also base images that already bring along runtimes or dependencies, libraries, whatever you might need for your certain language implementation. I was actually using this slide for quite a bit until I realized looking at the Docker Hub that this image is actually deprecated in the meantime. That means I should actually, I know this is recorded and I shouldn't say that I was actually running labs on this and nobody noticed that it's deprecated. So this is actually one of the important things also about this talk that no matter which option you take, you will have basically a certain source where you pull your information or your binaries from and you need to trust that source. And you also need to be aware of how often are things updated, is this being maintained and so on. And this is also the dangerous thing with those Docker files that you might pull in things and they work. But the part that you are interested in this application is most likely on that end, like where you build your application. And not so much on the base image side. So people tend to neglect that even I did myself. And then I realized, okay, it's still there, it still works, but I should probably update it as the people have told me so. So the webpage redirected me to that image. It was actually, of course, really easy to exchange. But there are more. So especially, I mean, not the entire talk will be based on the Java image. But in particular with Java, you can do quite, you have the problem of really many options of your base container image. And that brings in the bit of difficulties there. So in the end, I don't want to open up Docker Hub right now. But if you start searching there, you would find plenty of those images. Now, finally, we can go back to the question of my colleague. What is the best Java base container image? We don't know. We only know that there are many out there. And we still need kind of a decision process of what might be the best one to start with. Now, looking at the things from a bit from a different angle, it's mostly like, how do you actually use your base container image? Where is that being embedded in your software development chain? And nowadays, as a lot of applications are running on the technology of Kubernetes, a kind of typical workflow looks like that, like code, and then potentially test. I haven't reflected this in the chart yet here, but code, build, put it into a container, and run it somewhere. So the scope of the Docker file is normally the containerization step. And the idea of such a pipeline is, of course, to make things repeatable and basically gaining confidence and security in the process if you can re-execute it all the time. So the Docker file is one step where you have a repeatable process where you know this just works and this does exactly what it's supposed to do. Now, if this is already given for that, why would you not want to extend this scope and say, if it's so good for the containerization step and it standardizes things, why would we not want to use it for a build step as well? Looking back at the Docker file, that, of course, would mean you first have to install the build tools on the operating system image, then copy the source, build the source, potentially remove the build tools because you want to have your image as small as possible, install a runtime, configure everything, and run it. So that can get more complicated than the plain Docker files we've seen before. Docker also realized that and came up with a thing called the multi-stage support where it was possible to separate the Docker file into stages. So we're still talking about that script style here, but I highlighted this with different colors. In this Docker file, we're actually building two different images. So the first one we're going to build, and I'm using a Maven container here, so this one comes with a build environment for Java. This one would actually copy the source, all the source information into that container, and then build all the source and come back with a Jara file. And given that it's running in a container, we can, of course, go sure it will always take the same builder. It will always happen in the same way. So the outcome is always supposed to be the same, which makes, which can have definitely improve a pipeline as if people, in the worst case, check in the Jara files from somewhere where they build it on their own. So this makes sure the build step is basically standardized. Now, the next one is like the run step where we just take another, oh, I can see another deprecated image as a base image. And I have to update that slide. I'll just notice that. And then copy that Jara file from the initial container into the next one. So we kind of basically separated the build and run image step within the Docker file. So moving forward, the development within Docker didn't come to an end. In 2018, they started coming up with a new technology called build kit. And this was definitely very new as opposed to the thing they did with Docker files before. So it kind of went away from a simple shell kind of behavior where each step would just be executed and like a commit and run step and adding a new layer. So that really brought up, became like kind of an engine where a lot of things were improved. So especially caching was of the individual layers was completely rewritten. Security concepts were improved than images to become smaller but a flow lodging and so on. In the end, it didn't really make Docker files less complex. It more like turned them into bit of a programming language where you can have like, if this happens, then execute that stage or those stages don't depend on each other. So we can run them in parallel and so on. While it makes your Docker file probably more efficient, on the other hand, it increases the skill requirement to actually execute those files. Now, the old syntax still does work. The only thing you might see here on the top is that actually common file where you specify the so-called front-end image. So with the new build mechanism, you have like a back-end image that does all the processing and the front-end image which basically defines all the functionality which is possible in the syntax of the actual Docker file. So and then you can change also to things where I say, well, I'm going to try some experimental features which are not in like in a released version at the time. So one of the cool things that got also added with this new build kit implementation was the concept that you can do caching or mounting an external file system. While this was already possible, of course, with plain Docker containers, it wasn't possible during the Docker build steps. And now with this, of course, it makes it possible if you build your application and rebuild it a lot that you don't have to download all the artifacts over and over again just if you change one line of code. I mean, particularly people that use Maven or NPM will kind of know what I'm talking about here. So to sum things up, what we have learned from the evolution of Docker files, say what makes a container image good, you could of course see there is speed, how fast can you build it, the size, how big is it in the end? Like from a certain tendency, smaller is of course better because then you don't need as much storage. Also the structure in the file is important, the degree of standardization and security. Now one thing or one consideration that I wanted to go quickly into, and this might not be applicable to all programming languages, but for some it does, is of course I said it before, have your image as small as possible, that's all right. But I very often see people trying to squeeze out like every kind of megabyte on those base operating system layer and spend the time of building like small footprint, a base operating system images that they don't have to maintain. Technically the sum like the full footprint of an image is the sum of all layers. In the end, the application layer will most likely be the one that changes most because whenever you change some code you will get a new application layer. This sometimes might also change dependencies but it should rarely change the operating system layer. The point that I'm trying to make is the way a container daemon works, if it downloads an image once, it downloads all the other layers. So when it has downloaded that operating system layer once, it probably won't download it over and over again unless you explicitly tell it so. So size optimization on the base layer is probably not very efficient, but structuring of the application layer might be because in this case again in a Java example, the application layer normally consists of like class files, resource files and dependency files. And even if you just change a single line of code, the overall application layer will be rebuilt. Now there is something you can do here. In this case, you will basically extract things out of the child file and copy them independently into your run image. That technically means you get a container image layer for each one of those. And if you do just a single change in a source code file, then it will only change this layer in the class files and all the rest of the layers will remain. And normally this kind of gives a much more of a performance improvement because we're talking about normally kilobytes here. So this will be really, really quickly to load. The other layers will take long to load on the first run, but once they're there, you don't have to worry about them anymore. So I would recommend much more to invest in that space to say can we actually improve or optimize something than trying to find the smallest footprint base layer image because most likely it will be loaded only once. I mean, of course I'm not saying base offering system images with three gigabytes are totally fine. I mean, you keep them at a reasonable size, but try to optimize at the right end. All right, so let's look at the time. So we're half an hour in. So yeah, what I'm just probably going to do now is stop the slideshow for a second and repeat some of the things that I have just talked about. Or while I'm bringing this up also, are there any questions as of now? No? Okay, so I'm just going to try to arrange the windows in a better way. So I have my environment here. I don't think I have anything running. I only have pre downloaded one of the images here with one point two gigabytes because I wasn't I'm not super sure about the network here in the on the conference and I don't have any backup or fallback. If my demos won't work, they're not going to work. I don't have a plan B. So very, very honest style of presentation. Now, yeah, let's let's have a look into that first example. So I have many Docker files as you can see here. And I'm going to set Docker back to the way as it initially was. So this was the time prior to build kids. Probably have most of people have started that way. So we're already in the Java folder. And the way I've seen most of the Docker files is basically that they take the code from outside. So like I do now, people would build their application on their machine and then run put it into a Docker file. That is of course a bit dangerous because it's not guaranteed that the build will always be identical and reproducible as things on certain machines might of course change might maybe have built have been built with a different JDK and so on. This is really difficult to see later on once they are in in a container. Nevertheless, it will still work fine with such a Docker file. So we're going to look at a simple one. Um, this is the one way where I basically install the Java on top of Ubuntu. So let's just execute this. I hope this back and forth is not too annoying. But as I said, I definitely can't type it, um, with my with my left arm here. So as you can see now, for those who was doing this the first time, it basically pulls the images that it doesn't have from the repository, which is a Docker hub in this case, and then executes all the installation steps. And of course with this one, I'm trying to show what is what is a bit of bad here. You have to redo those things over and over again. And basically you're probably trying to solve a problem which has already solved by somebody else. Eventually you're going to get your image. So we're going to have this image, which is like six seconds ago. And if I look into the history of that image, so I'm just typing the seven nine E, which is like we can all, we can basically see all those steps that have been listed in the Docker file. So we have the base image here. We basically install the Java and here we have 20 megabytes, which is the chart file that I put into the container that I have previously built. In a similar way, I could use the deprecated adopt open JDK. And also very important to see here, there is no deprecation notice on the Docker execution. So it is an image which is still around on the Docker hub. There is a warning on that website, but there is no special tag or whatsoever on that image. So if I run that in a pipeline or I have that in a script from development, it will help for me to see that I'm actually building upon something which is not really recommended anymore. So this also increases the responsibility on developer side that they don't only need to check for their application, but they also need to validate whatever is in their base operating system container should also be fine. So I could basically build a lot of images and as the network does really well, so I'm going to do that now. I'm always happy with the demo works. But yeah, I think you get the point. In the end, we can see that we have a couple of images here. They slightly vary in size. So normally the Docker history command is something which can help you here to see on a high level basis how has this image actually been constructed. If you want to know more, there is an open source tool which is called dive, which I guess most people know in the meantime. This is like a command line browser for your container images. So basically here on the left side, you see all the layers in your image and then you can browse through them. And on the right side, you can see what has actually changed in the container. So if you want to really go on a deep dive in your application, in your container, then this would be a way to do so. I'm not going to go into that depth, but just for you basically to see. Okay, so now we can look quickly at those multi-stage files. So if I do this multi-stage right here, and we can have a quick look at the code again, if I ever find the right screen. So, so here you can see at first pull that build image, which is happening. Sorry, too many things open, which is basically happening now. So it pulls that image first, then it copies in the source code and it builds the source code. Now, of course, the advantage with this approach is I don't need any Java or whatever go Ruby implementation on my machine because all the build part will be happening inside of a container. And if I reuse the same base container image, I can also go sure that everybody who executes that file will get an identical run image in the end. So this is something I can probably not guarantee if everybody builds the code on their own machine and then copies it into a container image. The downside, as you can see, it's of course not getting faster because inside of the container, the container doesn't know that it has been built on that machine 20 times before. So it's always starting again. And I mean, it will notice it if nothing changes. So right now, we're using everything from the cache. But if I just go in there and say, well, give me a second, I'm changing something very minimal. So this is not a big change in the business logic. And run it again. Then, of course, you will see that this build command will run over and over again. So if that happens somewhere in a pipeline, it's probably not that much of a drama. If that happens on your development workstation, I think your developer might get a bit bored if they have to run through this all the time. Now for that, we will have a look into this build kit technology. So when this one has been built, has finished, well, I'll just make it finished. Yeah, I mean, we can watch it in the end again if you really want to. But anyway, so I'm using, I'm changing my Docker behavior now and try to build this again. So the first thing you will be able to see the output looks a little different. I got a better split down of the individual steps, what I have happening or what I have to happen. And now it's running again in that build process before it switches over to the run part of the application. And again, the first time it will take all the time because it needs to build its cache. But once it's there, actually, I'm just noticing I'm using still one which doesn't use the caching. So I should probably use that one that uses the caching. This is the multi-stage cache. Yeah, I'm getting confused with so many different Docker file names. So I probably have to type that. So this looks better. So if I repeat that here, this will already hopefully be able to pick up the cache from before. No, it doesn't. Yeah, so the reason is because I am using now this other front-end image, it uses, I think, a different mechanism for caching, which wasn't the case for the previous build. This final run, we still have to wait. And then we're done with this. And I can switch back to a few more slides. All right. So this looks okay. Now I'm going to do the same thing again. As I did before, I'm going into my application file and just modifying the method signature and running this cache part again. And when it comes to this part, you can see it doesn't download all the things. It just rebuilds and pulls in the cache from the file system. Of course, there are also pros and cons to that. I mean, in this case, it fully relies on the cache on my machine, which is outside of the container daemon. So potentially there could also be harmful things happen on my machine, which the cache would then pull in. If you want to do it clean and reproducible, then don't use the cache. If you want to do it fast, fast just for development purposes, it's of course a big help. All right. So I think the last and anyway, one thing I wanted to show, no matter what we did with the previous images. So if I do a docker history on that very last file, you will see that there might be difference in the setup here. What is what always going to be the same is the almost 20 megabytes of the application. So let's just see another that is DC2. And we see the same thing down. So we really haven't improved yet on the aspect of splitting the actual application part into like structured layers. I have this layered image here. So this basically is a three-stage image. The first one will actually build the application. The next step will split it into parts. And the third one will basically use those parts in the final image. So I'm invoking that. And that, of course, also uses the cache. So it doesn't rebuild it. And this extraction part only took less than a second. So we should have a new image here. I'm sorry. This one that has been filled 40 seconds ago. And if I do a docker history there now, then you can see that this top section has been kind of split down further. And we can also see that from the 19.7 megabytes that we had before, 19.6 megabytes are dependencies. The actual class part is only that. And if I now do a minor change again and rebuild it using that style, it will only have to exchange that layer in there. And if you do that very often, that would probably increase your performance of the overall process much more as if you would save one or the other megabytes in any layers down there. All right. So much for that. Let's go and see some more alternatives in case there are no questions. There's a question. That's the part which is kept outside persistently. Exactly. M2 is like the name for the Maven. No, it's not this one. Or is it here? Yeah. This is like basically, well, I think this is actually the directory in the container and where it's mapped to the outside file system, this is what the Docker daemon takes care of. I don't think you can specify that. Or maybe you can, but I don't know. But yeah, this is basically this command enables just that the mount, like the caching mechanism of Docker and has nothing really to do with that Maven part. It just basically tells that .m2 is like the cache folder of Maven. All right. So thanks for that. Oh yeah, please. Yes. Exactly. So yeah, if you have like an npm or whatever kind of build mechanism for other languages, then you need to be aware, which is the directory you need to have in your container where the build process will look into. Thanks for clarifying. All right. So going, stepping away from Docker, now that we've seen what you can do there and what other technologies that might relate to it, a friend of mine has kind of sketched this time scale of container and build mechanisms evolution. And a couple of interesting things there. So you can already see that a certain technology is called build packs already existed prior to the arrival of Docker. This is definitely one thing that we're going to look into. Those build packs have been initially brought up by a company called Heroku. They're later on adopted by Cloud Foundry, made open source there. Then other few approaches was like, for example, putting the Docker build mechanism into Maven or Gradle, like Spotify did. And then not only those things changed, also a lot of things in the container world changed. So there were new container engines. There was a specification of an image format, the OCI, which made it a lot easier for companies not being Docker also build images that would be able to run anywhere and so on. So I'm not going to read through all of that, but just to see a lot of things have happened. And some of the things we're going to look into, we're going to do now. So one thing to highlight would be the tooling called chip. This is coming from Google, our sponsor, and is a dedicated build mechanism for Java application, but not really Java, but let's say JVM-based application. So Kotlin and Groovy would work as well. And it basically just works as a plugin. So if you have Maven or Gradle as a build mechanism, you just put in that plugin and then you get your application in a container. It's really quick. It's really fast forward. One of the interesting things that you see on that picture is the thing that you don't see. And what I mean is, this doesn't require any kind of container daemon. So if you build a container using a Docker file, you will need a container daemon for the processing of that and always the re-instantiation and the manifestation of the image. So this one just does that on a level of the binary itself and it creates an image, which you then can basically upload to a container registry. You can export it as a tar file. Or if you happen to have a container daemon running, you can also build it against that daemon so it ends up in that local registry, but it doesn't need it to be built. That's basically the important part there. And going back into the thing I showed before about the application being split into layers, this is basically something that that chip does out of the box. So I can also show that real quick again. This always takes a bit. I don't know why. So let me look at my script. So we've done all this that worked well. Now we're going to look at chip. So in the end, this is like your maven command where you just have to add this section here and then your application will be built. There are many different ways. As I say, if you do a docker builds, then it's going to be end up in my local registry. If I only do a build, then it's going to be pushed to a container daemon. Anyway, to a container registry. Anyway, that means I should be locked into that. So it can actually do that. So now it runs and what I could have shown, of course, you might not believe me, but I can stop. I can stop my local daemon. So I hope this won't break all the things because I told you I don't need it and I still did the builds while it was running. Well, it's definitely not running now. Okay, and I can still do that. So you see this totally works fine and it didn't only build the image. It also uploaded it. So if I look at my personal, so this one a few seconds ago, that was just pushed right now. So it's in one step. You build your application, you build your container and you upload it. So of course, the drawback with that one is it this works only with Java. But if you use any JVM build based environment, this is of course something which can definitely improve your process there. And as Google is a company which would probably not going to go away the next few weeks. It might also be a reliable source for the image building mechanisms. I mean, Google has been known to stop certain projects just out of nowhere. But this one has been around for a while. And in case it's not going to be there anymore, you can still use other ones. But I definitely wanted to show this as one alternative. Now next to chip, there is, I'm going to jump in there again. So I'm going to postpone the demo for that. There is also a tool called Co, which is technically speaking chip for Go applications. So it's also a dedicated build mechanisms. If you want to build containers for the Go programming language, then you can invoke it in a simple, basically you install this as a command line tool and just execute Co in the directory where your application is. And it's going to build and package it straight away. And of course also uploads it to your registry if you need to. Question? Can you speak a little louder? Yeah. This is a very good point. Thanks for bringing it up. Yeah. If you run the things all on your developer machine, it's probably not, you don't have many restrictions of what you set up and how you build things. But if you run in a pipeline-based ICD environment, you might have limitations of what you are allowed to run. And I have seen environments where they were Jenkins build nodes where you were not allowed to install a Dock or Demon. And that basically made it impossible to build those images there using that technology. Now using chip or Co, you don't have that problem. You only basically use standard build tools that don't interact with a Demon that requires whatever certain administrative rights. Yeah, good point. I could claim now. I've said it in earlier presentations. I forgot it today, but I give all the credits to you. So thanks for bringing that up. All right. More questions? Yeah. What about caching? That's a good point. Well, I can only answer that for the Java variant. Basically, you execute Maven still locally. So you need a local Maven installation and you need a local Java installation. That means you also have your local Maven cache, no matter if you use chip or not. That's basically running independent of that. But then again, this goes also back to the problem I said before, you build on your local machine. I mean, there's one death you have to die. I mean, you could potentially, I've tried it before, but then, of course, you read the Dock or Demon again. You could run your chip execution inside of a container to make sure it doesn't use any local cache and have it reproducible in that style. But then, of course, running a container means running a Demon. So it's one or the other. Good point, though. All right. So now we're going to talk a little bit about build packs. And I might actually use all of my time. I'm realizing that now. Yeah, thanks for hanging in there. I'm going to come to an end. Now, build packs for those who have never worked with Heroku or Cloud Foundry were basically an initial idea which was very similar to the concept of a container. They originate in the idea of a pass where a pass would claim you as developer, you give us your source code and you don't worry about anything and we care about all of the rest. Sounds pretty good. Doesn't always work that way. But one of the things that worked pretty well was the concept of those build packs. So if a platform claims to be able to handle your code, it needs to build and run it somehow. And the way the build packs worked was like you upload your code and then a mechanism would kick in to say, well, I can detect now this is Java, Ruby, Go or whatsoever. Then I can evaluate if I have a kind of a build and run environment for that. And if I have that, I put it together into a kind of runnable format, which is like one executable that contains everything that it needs, basically very similar to a container image. The difference in that approach was that here the detection, the packaging and the runtime was totally outside of the developer. This has, of course, changed with Docker and Kubernetes becoming so popular because then, of course, the requirement won't only come up with a source code, but also maybe come up with a packaged environment in a way like to go a bit more into that you build it, you run it fashion. Anyway, as those container technologies became so popular, it didn't mean the idea of a build pack would totally be lost. So especially, this is a bit of a timeline here. Well, Heroku started with that as a commercial offering. Then Cloud Foundry adopted that for like an open source implementation. Heroku continued, but they were both basically within their passcode. Now, seeing that containers and Kubernetes is having such like a big adoption, they said, why don't we come up with a build pack idea that doesn't run within the pass, but runs outside of it, but still provides the advantages of developers not having to worry so much about their actual build process. Now, Cloud-native build packs are like a definition of how things should be implemented, and then everybody can build their own builders or build packs. And one of those implementations is Paketo that we're going to look into our samples today. So from a flow perspective, this technology definitely requires a container daemon. Otherwise, it's not going to work. So it's totally based on that concept. So it has an own CLI. So the idea is that you just run, say, pack in the directory where you want to execute your build mechanism. Then the application basically downloads a so-called builder image. This is a very big image that contains the detection, the lifecycle mechanism, and all potentially necessary needed build packs. Downloads all of them at once. Then it puts the code in there, and then it's being processed in there. So there will be a detection mechanism. Yeah, what kind of code is that, how I'm going to build that. And then a certain image is coming out in the end with a structure of a base OS image and various layers that the individual build packs provide. Another interesting feature that those build packs come up with is a so-called rebasing option. Now, in the beginning of the talk, I told you it's probably more important to focus on the application layers, because they might change a lot. Now I want to put the focus more on, could there also be a scenario when you have to exchange a base layer? And the answer is, of course, yes, there can be, because sometimes your application might be built and running and everything is fine. And then there are suddenly things in the news, like a log for J or hard bleed or whatever vulnerability, and you might be very worried that you are affected with your own environment. And this is particularly dangerous with using containers, because as I said before, software developers will probably most know what is going on in there and just happy to have this base layer. But if there is something, some kind of exploit in the base layer that they pulled in via a Docker file, it's really hard basically to make them responsible for that, because they won't be able to scan and do all that. So with this rebase functionality, you can leverage that and say, well, okay, there is a certain problem here. And I'm just going to rebase all my images that use that behavior. And then I know the base image is updated and everything is fine. Where you can see how that works is in one of the implementations. It's called Paketo, which I think is also Greek and stands for package or packet. Now Paketo has a couple of different runtime and language support. And this is exactly where the positive effect of this rebase mechanism kind of kicks in. Like if you run a polyglot application or microservice application in your Kubernetes environment, where you might have your frontend with node, some components with go, some Java components all working together. And each individual application developer team would build their own container images, then they would most likely also always select their own kind of base images. And if such an exploit is being uncovered, I mean, the first thing that people would think about is like, am I affected? And how could I find out where am I affected? I think I've heard one person from the initial cloud foundry creators, they did a survey at a major automotive company to ask how many different operating system levels, like not different versions or vendors, but also patch levels and everything do you have across the company? And they came back with about 300 to 500, not only containers, but VMs and everything. Now imagine you have to go through all the three or 500 to find out, is this a potential risk or is it not? In turn, cloud foundry had only one base image for any kind of application. So that means if a certain thing was happening, then it was immediately clear, okay, all of them are affected or none of them is affected. And this is the same thing you can achieve if you use that for as many things as you have in your environment. I did a similar experiment with my students in the lecture. So I said to them, we're going to build a small micro service application with a front end and a back end and a database. And the only thing we're going to say, there is a certain rest API definition between front end and back end, which always needs to be the same. But you're going to go together in teams of two and everybody writes a front end and the back end. And please pick your own language of choice. And in the end, it needs to be containerized and you're going to send me all the repos and the Docker files. By the end, of course, I had 20 to 30 different base images. I mean, that's just the nature of things. And I didn't even tell them, like, use this one or that one. I mean, they just go out and see what works and they will use it. And of course, this doesn't reflect like professional developer environments. Most hopefully the companies will have scans in their CISD pipelines to go shore, like the images are okay. But it's probably even better if you don't have to worry about it in the first place because you have a mechanism that takes care of it. Of course, it won't work for everything because the build packs have limitations in terms of coverage, what they support. But at least I think that's the right direction to think towards. All right. Now, I think I'm coming to an end here with this slide. So again, for a Java application, it works in the same way. You invoke pack, build packs are being executed. This one also automatically separates into class resource and dependency charts. In case if you have a Springwood application, you can also use this as a Maven or Gradle plug-in. But personally, I would actually say this one is kind of better because then you don't need to execute Maven locally. Everything is be fully covered in your build process. Now, to come back to the question of my colleague that I actually never answered, I said, well, I don't know. And I asked a counter question. It's like that same thing I try to explain to you. Who owns that image? If something goes wrong with that application, who is going to be called? Maybe do you have a certain kind of white list of base images that you can use in your environment, something that the client maintains? Then you're absolutely fine. The developer doesn't need to worry about that. It's just like from the pool and they're outside of that decision process. If they don't have it and every developer is free to change their own base image, that can be definitely risky. Also, who owns the build process? Who would detect if something goes wrong? These are all the things you need to consider. And if you know that, then you have seen a couple of technologies you might be able to place and do that. So, yeah, going back to a few demo steps, I mean, the rest of that is pretty straightforward because I only have to execute one command. So I have stopped with that chip thing. The only thing I'm going to do is I'm also going to build this against my local, so with that docker built target, it kind of builds it to my local image registry and it's not going to upload it directly to the docker hub. So just to show you, sorry. Now, one of the interesting things is that chip image, you will not find in a range of whatever a minute, seconds ago, you will surprisingly find on the bottom and it says 52 years ago. So it's just, yeah, we just traveled in time. Now, what this means is or the reason for that and you will see the same behavior later on when we're going to do the build pack thing is that if you do a docker build, the timestamp of your build step will basically walk into your image ID. So if you do a total identical build at two different points in time independent of each other, you will get a different image ID because the timestamp is a part of that. And the only way to avoid that is basically to set a certain designated timestamp potentially somewhere in the past, which will basically be the same for every image. Of course, it's not very pleasant from this perspective. So in both chip and in build packs, you can turn it off and say use the current timestamp. And then it will show up on the top like all the other image. The price you basically pay is that consistent behavior of having that manifested in your image ID. Now I'm going to do that pack step, the final thing. So it goes now into the detection mechanism. So I have to scroll a little bit up. I mean, one of the things you will see, you get a lot more output with those build packs of what is happening under the cover. So this is basically everything that it found. These are the build packs that are going to be used. And I detected OK, Java 11. It already had downloaded the JDK before. So it goes through that, builds the code and comes out with an image in the end. So if you look into that, this is actually 42 years old. So it's 10 years younger than the one from Google. Anyway, if you look into them real quick, so Docker history, I'm going to look at the chip one first, you will be able to see that here we also have this separation of dependency resources and classes, where the classes are like particularly small dependencies are basically the majority of all that. I think the parqueto one does look a little different. This one is actually not as descriptive in the history, but you can still guess. So this 19.6 I'm pretty surely guessing is the dependency stuff. And the other ones, well, I don't know, which I'll just trust it. Now, this is almost pretty much it. I mean, one of the things that you also might encounter there is when you update. I mean, if you update the Java version, that if you use Docker files, that means you have to find a new base image. And in particular, after many people have hopefully moved from Java 8 to 11, now they can already have 18. There was a shift that some of the images were not continued. And so you had to find new images. It can be solved, but it's time consuming. Now, if I use the build packs, I can export this version and say, I'm going to override that. And then I do to see if that works. So yeah, it's using Java 18. It has picked that up. So again, it will use the same base image, but then it will also use a validated Java build environment and a Java run environment. And yeah, so this is the concept, basically taking as many things away from a developer decision as you can and say, well, I provide you something which has been repeatedly tested and used. So those parquetal build packs are basically the technology which comes out of the build packs of Cloud Foundry. So they're pretty bulletproof. And now just to finalize things, this is like a Go sample application. So with that code command, I would just, I think invoke code build. And so this one would have directly uploaded the image to the Docker Hub. It gave it a bit of an odd name. So I'm pretty sure you can override it. The other option would just say to say, okay, build app go. And in this case, now different build packs will be detected. This is not a Java application anymore. This is a Go application. So that of course also means you can use those, the same invocation and the same basically templated mechanism across a multitude of your applications. All right. I think that's pretty much it. I think I have 10 minutes left. In case you have any questions, well, that's actually me. That wasn't, that used to be me before the COVID times. Now this is the, this lab repo. So if you want to like try this yourself and repeat some of those steps, you can, you can take a screenshot there and yeah. First of all, I would like to say thanks for listening. More questions. Yeah. Can you define your own golden images that you want to use? Yes. Well, I, this is a good question. I can, so with pack, if you just run pack, it has a function which says suggest builders. So the builder image is basically the golden image as you have just mentioned. Um, and you can say, see there are ones from Google, they're one from Heroku, they're one from Paketo. These are like the ones which are officially suggested by, uh, by this tool or like the compliant with the cloud native build pack specification, but nevertheless you can build your own ones on, and you can also embed your own build mechanisms. And I think, I mean, of course, if you have a bigger company and a more variety of environments, then this will definitely help because that will basically then be the source of truth for all the base images that are being used across, uh, across all environments. And if somebody decides not to use them, they will have probably explain why. Okay. Thanks. What, what do I recommend? Um, I mean, yeah, it depends. You would have guessed that now. Um, I mean, in, in general, not only for building container images, um, um, if it comes to cloud, I would always say, say, use the highest cloud abstraction that you can, if you don't, I can have something as a SaaS service, don't rebuild it yourself, then just use it and, and, and buy it if, if the price is okay, of course. Um, and, and, and same thing goes for pretty much everything. If something is a problem that has been already, has already been solved, then try to use that. Um, and in case it doesn't work, then look at the other options. I mean, it's still the same with this one. I don't have any problem with Docker files. And I hope this, I don't make this impression, especially the Docker file is the only option that, that will always work for like every, every programming language, every framework and so on. But in, in turn, it means, well, you need to maintain it and you're the owner from that level of the stack. So, um, whenever I go to clients, of course, I, um, I do a similar presentation and say, I want to introduce that in the end, it's not my decision. It's yours. But I would recommend to start, uh, with a possibility that takes a few problems away. And, and if we figure out, and then we just try, I mean, the easy and the good thing with those technologies, they don't need a big change and turn in your overall, um, architecture. I mean, most of the environments are running, um, in, in pipelines nowadays. So you just define a new pipeline where you have an alternative build mechanism and trying that is not so expensive. And then just iterating it over, over again gives you more confidence. So this would be the approach that I, I would, I would take. And, um, yeah, also if I come and see that people would use Docker files like the one, um, I had in the beginning is like, well, you can do it this way, but these are the risks that you have associated with that. And if they say, well, this is what I'm going to live with, then I can only say it. I mean, does it set enough as an answer? Okay, good. Yes, please. Yeah, just to stay still with Docker files. You use this in your company. I highly recommend you to have a look on KaniCo. Okay. Open source product. You don't need Docker. So, and you can run this in your pipeline with no access. Another one is build that. And if you want to stick with the part, build facts, there is a tool that helps to solve, uh, even not one single issue build, but also to deploy the tool named, uh, waypoint from HashiCorp, which is also good that it can build your, um, software and also deploy this either to nomad or green. Okay. Well, thanks. So to repeat, and I'm not sure if everybody heard it. So it's, it's KaniCo waypoint and buildup as additional open source toolings. So yeah, um, if my arm goes fine again, I have to try to update my presentation to those. Yeah. I mean, this is actually a good point. I mean, I've showed that diagram in the beginning. There are various options, but, um, that there is no, uh, this, this is never going to be complete. There will also be new things evolving and coming up. I just happened to show those that I have most experience with, but that doesn't mean that the ones I did not show are in any case worse. It's just like that. I haven't probably touched him yet, but yeah, thanks for bringing it up. Yep. What about the difference between the builder and the target, example, the multi-stage? Well, the multi-stage. Can you specify it with buildpacks? Well, you can. Um, but that's kind of against the theory of things because, um, I mean, basically the buildpacks, they come with a predefined image that they have checked and validated on their end. So they say, if you download our image or build an image, then you're going to get this as a base image for your containers. And we can give it to you because we have run the tests with it and scan it, which you might have not. But if you exchange it on your own will, then you basically override exactly that, that possibility. But technically, I think you can. And if the, the, the, like the predefined builders don't allow that, I know, for example, that chip and co allow that. So you can specify your, your custom base image now, but I never did because I just used what Google suggests because they have built it and tried it. I don't, I actually, I'm not so keen in working on those base image layers anyway. So if somebody solves that problem for me, I'm pretty happy with it. Um, and if you don't want that is I like the mechanisms, but I want different, um, different contents than there is an API to build your own builder images. And there's, I know from, from people that have done that successfully. So this is not just an option. Okay, no, no, that's a good point. The builder image is not going to be the base image. So, um, so we have here that builder image, which is 1.17 gigabytes. Um, and as you can see the, um, the Go app I just built, or the Java app, they are by far not bigger than 1.17. So this builder image basically contains or contains the reference to the base image that it later on uses. I think you should actually see it somewhere here. There's like, this is, that's there. So yeah, I mean, still you will be able to get your images smaller than that, especially if you use like Go or something, which is just a binary and you can use a distro less container. Um, but that might be better for certain attacks because it doesn't have a shell and things like that. In turn, you won't be able to rebase things like in a way that I showed before. So yeah, it's, it depends on the nature of your environment, which one might be better. All right. But I think it's really time to go to the Guinness storehouse now. So thanks very much. I mean, I, um, we as, I'm the only one here for my company this time. I'm sure next time we'll come back with more people. So we're not sponsoring or having a booth yet, but if you have any more questions after this talk, feel free. You, as I said, you can find me on LinkedIn or Twitter and just ping me. Uh, tomorrow I'm going to be just floating around in case you want to, want to chat if it's just free to reach out to me. All right. Thanks again.