 Okay, everybody, so it's time to continue, let's welcome our next presenter, David Baczovacic. Okay, I will come to you here. So basically I am David Baczovacic and I will speak about how we are duckizing Jebus products, basically it's will describe how we manage to deliver Jebus products, specialy for OpenShift in my team and we will talk about following topics. It will be how we manage the profiles, how we handling layers, how we configuring images, how we testing them, version them and also some notes about documentation because documentation is pretty important for us and with OpenShift it's because images are pretty complex and you need to know a right way to use it. So also it will not be only about processes much more, it will be about our custom tools, our custom stuff what we are doing and what we are developing to manage all of these topics. So, let's start. First, managing the profiles. Basically when we started it was pretty easy, we just use plain profiles and it worked well. But time to time it happens that we get new images and we need to support a lot of stuff. Now we have many products, we are shipping EAP, we are shipping images based on EAP, like decision server, process server and so on. We also shipping Tomcat or enterprise web server and this makes us to think about it more because basically we need also to share stuff between images because for example each of our images contains Java virtual machine. Our images also contain some OpenShift specific stuff and also some of our images are meant to be run standalone with our OpenShift. Basically these are used as a step before porting them to OpenShift. So, as you can see, the inheritance model is described here, basically we have some base image, then we install JVM, then we install all products like EAP or Tomcat as a standalone and then we add custom OpenShift layer which just adds a lot of OpenShift stuff, scrapes and so on. So, basically you can also find this layering model as an inheritance model in upstream for Wildfly and all other J-Boss stuff. So, we even try to use this stuff which we are developing for J-Boss as a product to use in upstream but it takes some time and it will happen in future. So, also for us the biggest issue what was starting to be was inheritance model. Basically in the beginning we have just three, so basically each image has one parent and all the stuff just go nicely, so basically we take base image, created some labels, environmental labels there, inserted some J-Boss user, that was fine. In the next layer we just installed OpenGDK, configured it. It was everything was good, nice, then EAP, then some OpenShift layers. But when we went to the need of shipping for example decision server which is based on EAP or J-Boss data grid, it started to be really messy because we wanted to ship these products as a standalone images which should be based on EAP standalone. Also, we needed OpenShift layer there, but OpenShift layer was already in OpenShift EAP server. So, we basically had two parents, two Docker images, two layers which we needed to match, which we need to reuse the code because if the first approach with Dockerfile just ended in a lot of duplication stuff in Dockerfile, which was pretty hard to maintain. And it ended up with a lot of our time spent only by visiting Dockerfiles, looking if everything is okay. And even stuff like respawning images for security reasons or installing security fixes updates was a pain because it introduced some errors, some typos. So, it was not nice. So, basically, we've decided that we need to do something better. So, we created a tool which is called DOGAN, just Dockerfile generator. It's upstream project, it lives on GitHub, it's even Dockerized so you can just run it from a container. And what is it? It is just simple templating solution for Dockerfiles. It's really easy, it's ginger-based. I will show you how it looks in a few seconds. There is some code snipplet. But, basically, what it tries for us to fix is layering an inheritance code reuse model. Because we will still have a lot of duplication in our Dockerfiles because Dockerfiles are just generated from DOGAN. But, it doesn't care about it because Dockerfiles are just generated, are the products and are not stored in repositories and so on. So, they are not important for us now. They can be important for you as a customer if you want to extend it. But, for us internally, it's not what we versioned. I can show you how it looks in reality. Basically, as you can see, we have some ginger template, which is part and library. By the way, do you know anyone who knows ginger or using some templating solution? OK. So, I will explain a little bit more. So, basically, this framework works like that you have one generic file. We just contain some description, contain some keywords, and you have another file, which is just values for that keyword. So, basically, what DOGAN is doing is that it consumes this, we call it descriptor. This is for our image for installing GDK. And it applies it to this template. So, nothing really hard, nothing much. Also, DOGAN can do more stuff. You can see it can input some scripts and run them in a container in a build. It can just inject some RPMs, because, for example, if we need to respon image or install some security in fixes, basically, we need to wait for them to appear in some repository or some internal snapshot of repository. But in some cases, this takes some time. And we need a way to test if the fix is working. So, basically, if we are responing the image, we are taking some beta releases or pre releases, injecting them to the image, installing them via DOGAN and trying to run our test, put them to the QE and all the stuff. So, this is basically how this stuff works. As I said, you can find it upstream. You can see the repository, you can clone it, you can hack on it. Every effort is welcome. So, what was another problem? It was layering, because each instruction in a Dockerfile it just creates some layer, even the metadata instructions. And with a lot of layers, images just are big and hard to deal with. But also, layering is good, because it enables you to reuse some of your code, make builds quicker and so on. So, basically, we needed to find a solution how to make layering good. So, we don't have a lot of layers, but also we don't have... We don't miss some opportunity to share stuff. So, we had two options. Basically, we could try to combine every command to one command. So, basically, we will have one label command, one environment command, one run command in Dockerfile. It can work, but it's a little bit weird. And sometimes we need to run some script in a Docker image during the build. But we are different user. Typicaly, we are switching to user root and our application user. So, it was not nice. So, another option was to squash layers. Basically, looking at upstream, looking at some Docker issues and Docker approach, they are not interested in having anything about squashing in Docker, even to be able to have some instruction in Dockerfile to mark some blocks that should be in one layer. There is a long discussion, but it was poor request and all the stuff was closed. So, my colleague, Marie Goldman, just go and write some custom tool. It's Docker scripts. It's pretty nice script, which is able to squash layers. It's maintained, it's used, and all of the images produced by Red Hat by our team are squashed by this tool, so it's proven by usage. Also, what's great, it's now even able to squash new Docker image format, which was introduced in version 9.10. Also, this tool is pretty nice. It's just work, basically, the way that is to take the layers from the Dockerdemon, it just look inside, open the tile files, merge them, remove the necessary stuff and just put it back and upload back to the Dockerdemon. Only bad stuff is that it's really, really high demand on I.O., so basically it usually takes some time, and if you are running parallel build, you can run into some Docker box that Docker is unresponsive, and if Docker is unresponsive, it's usually just time-outing your commands without, and you cannot really see what's happening. It's just not good written in this way, so be aware, just not try to squash multiple images at a time. So, next stuff. What we really needed to do is that we need to configure images, and we are configuring really, really a lot of images and a lot of stuff in images. Now, as an example, you can see for EAP OpenShift image. What our configuration stuff is able to do is that it can consume some environment variables you pass to image or OpenShift pass to the image, and for example, create data sources, create connections to some messaging like ActiveMQ. We support STI deployments, inject some assemblescript. Do you know what is STI, STI? OK. This is a pretty nice feature of OpenShift which just enables you to take, for example, EAP OpenShift image, and you have some application you want to run it inside. So, basically in OpenShift you just put URL to your Git repository or any repository, and OpenShift will just use EAP image and run STI for it. It will create another image in which it will build your application via Maven or via build system you specify, collect the artifacts, deploy it to EAP, and run the pod inside OpenShift. So, it just enables you to quickly deploy a lot of other stuff. Also, we support clustering, which some self-discovery, so we need to insert some stuff like our custom clustering providers and way to find another instances. There is really a lot of stuff to do. Also, we have two types of configuration in images. We have build time configuration, which is typically the stuff you run via a local build. So, this is just... It just configures some stuff, which is the same for all. It can inject some logging providers. It injects Geolokia, configures it. It just configures the image the way we want it on the OpenShift, so we are removing or disabling console and all the stuff the image has in base. But also, there is runtime configuration. And runtime configuration is getting to be more and more interesting. Because what is that? It's when you start a container and that's the fancy stuff, like defining the data source or just connecting to active MQ, deploying applications, setting up SSL. You can even inject some secrets. You really, really lot of stuff. And this makes the image usable for a wider audience, because it enables you to use our image with our basing on it. You will just configure it when it's run. So, how we did it? Our first approach was using shell scripts. It was looking great, because everyone was able to write it. Everyone was able to hack it. It was easy to see, easy to extend. But there are some approaches like that. Almost every configuration for Jbos Middleware is in XML file. And editing XML files via shell script is a little bit messy. You can do it. We are doing it for now, but the code is not so nice. Maybe it's a little bit ugly, but it works. Also, we started to think about another way. For example, how to enable users to inject the scripts, its own script and to let them to configure our images. Because we just provide some basic stuff, but maybe someone wants to tweak the EAP configuration a little, but he has two options, basically. He can use our image and base on it and replace, for example, standalone XML file for EAP bar by his own, which will suit him well. But if you do it just in a bad way, you will be unable to use our configure scripts because it will not work on customer specified configuration file. Another option is to enable him to inject the stuff into the image in a runtime and let him to run his code to configure some basic stuff. We started to experiment with Python and created some project. It's just called CCT, which means Containers Configuration Tool. It tries to solve some stuff. Basically, we've moved from Bash to Python because a lot of stuff is easier to do with Python because there is a better API. Also, it's designed to be run as entry point to the image. Do you know the purpose of the entry point? Basically, in the profiles, you can have two things. There is command instruction, which is default command, which will be run, and there is also entry point, which typically is set to bin SH, which is just default interpreter, which is used for the command. Basically, CCT is designed to work as entry point. If you use it in an image, you put it in an entry point, and then it just works in the way that when the container is spawned, the CCT is spawned. It configures the EAP or any stuff in the image. And then it's just execs itself to the desired product, desired process. Basically, from EAP, it's just executing EAP, replacing itself with EAP. Basically, your EAP will be bit one and everything will be nice, and there will be no trace of CCT in your processes. Also, it's Python, so everything is easily possible. It's able to fetch Python code. It's able to just get resources. Basically, if you put any file next to the Python class, you are downloading. It will be just found. It will be used as a configuration snippets. Using Python, all the stuff is really better coded and then shell script is there. There is even better co-chairing because it's using inheritance and all the stuff. I will show you how it looks. So, basically, I will just only scroll a little. I will show this later. So, basically how it works. It has some file which is describing what you need to do. You can use to configure OpenShift EAP image. You can specify some environment variables. You can use it. It's just basic YAML. You can see that there is some stuff which is run on the CCT. Basically, you can see that it's running OpenShift module. It's OpenShift class and the mapping is really easy. Every method in that class is mapped to the name here. Basically, what you specify here is just execute same method from the right module and it passes parameters. Basically, it's really easy. It evens contains it's self-documenting, so it contains some modules. You can even ask CCT for some help. Just a little bit, I will just enlarge it and you can see that CCT describe itself. It can describe modules, which parameter is using, and so on. Basically, this tool is what we are now just considering to use in the images. We are making some proof of concept on it. It's public repo and if you are interested in it, look at it and hack it a little. Next up is testing. Testing is a little bit weird team, a little bit weird topic for containers because not many people are testing them in a way we do. Basically, we are trying to make something like unit test for Docker because we needed to ship Docker files to QE guys, the images, the containers, all the stuff, but we wanted to ship it in a way that it's a little bit tested, so they are not... We can be sure that even basic stuff works, like that the container is able to start, that the containers contains write labels, it generates write configuration files and so on. Basically, there was one effort in Red Hat for this stuff. It's called CTF. It's container testing framework. It's based on behave, which is another Python library. It's using garking grammar. Do you know this stuff, the behave and... Okay. Basically, maybe you know Cucumber, which is similar project. It's just behavior-driven development when you try to write tests in this way. So basically, it tries to use something like native language for your test. So basically, you can use your features, your test files, your scenarios as some description even you can use it as some little analysis of stuff, little task for developers to do. So we started to use this tool and I will show you how it looks in more deep state. I hope it's readable. You can now see some of our tests, which is the features file. Basically, this is the way how behave works. You have some features files, which contain feature and scenarios and some steps. And also you have Python files, which is just normal Python script, which is annotated or decorated in Python. It's better. And you can see if you decorate any function, you can just use it in a test file, in a feature file in a test. So this enables us to really share a lot of stuff, because we created some library of Python functions, which are able to start containers, to look inside containers, grab file in containers, all copy files, put files from host to container and back, start some processes in containers and scripts. We have a lot of stuff and it's parametrized. So you can see that we can really use the stuff in our tests. So basically, it's really, really, really easy. You can see stuff like annotation, which uses parameters. So you can even parametrize your tests. So it enables you to write a pretty generic test suite and use it in a very easy way. And it also enables you to write a new test only by using feature files, not touching the Python. So basically almost anyone in your team, even some managers and other guys are able to run your test or to understand them. Another big advantage of this framework is that you can write the test correctly, the names, scenarios correctly and the steps correctly. If something fails, you can really see what failed and what impact it will have on your image. So that's stuff about CTF. I think that this is a really interesting project and what is really good about it is that it's created about sharing the test and sharing the Python stuff. Because in CTF every Python module, every Python class with test is just git repository and CTF is able to manage them. It's just clone them as a submodule some stuff on GitHub and if you want to use it you can just get our test and write your feature files. It's just almost better to include. So another big part which we needed to do is versioning. Versioning with docker files and docker images, it seemed easy. But it was not that easy in the end. Because if you are publishing some product and it can be some great software you are basically just have version of this software. But with docker it's little bit different because you are not shipping only the software. You are shipping image with that software and you need to version two things. Because you have version of your product inside the image and you have version of the image. So our final solution which is just currently used and you will see it in our Red Hat OpenShift in the middleware J Boss images is this kind of solution. We have just product family. We have a size version of the product. We are not interested in micro versions because we are automatically updating them. They should not have any impact on functionality and they are typically for security and all these stuff. So you want to have them. Also we are saying some type of the image that is designed for OpenShift and we have the image version for us which enables us to really make incompatible changes in EAP64 image. It can happen time to time. We have need for this few months ago when OpenShift was changing requirements to specify user ID to user instruction in the profile instant of username. This was pretty bad for us because we were using usernames there. So the images wasn't able to be run on OpenShift so we needed to bump this number. Also there is another big part of our work. It's documentation. The best documentation is fresh one and actual one and the one with the right stuff inside. But it's hard to do it and we are engineers we don't like to write documentation typically. So we need to find a way how to have a good documentation with not much effort put inside. So we decided because we are using DogM for generating Dockerfiles that we can put some stuff like descriptions some example values and we can generate documentation from the sources. So basically almost all of our documentation is generated it's generated using of a struct and it's just self-maintained so we can see if anything change in the image it's almost instantly changed in documentation. So this is just some summary it enabled for us. If we take all these approaches together we are now able to automatically test all of our images during the builds. We can test pull request, we can see if anything breaks really really fast. Also with every build we have a fresh documentation. So our documentation is always up to date to current images. So we can provide documentation to doc team and they can just consume it. Also another benefit which is pretty important for us is that with templating all of our Dockerfiles look same so we have same order of instruction environmental variables and labels defined in same order. So everything looks really nice it's easy to do and you can really really see the changes quickly if you are looking at Dockerfiles. Also with DogM, CCT the learning stuff we are really really able to share all of our scripts or of our configuration stuff to our images. So we don't have a duplication of code in our repository. So this is really really good for maintenance and this really enables us to spend a lot of time by delivering new stuff not by maintaining testing and doing the boring work. So I want to thank you and do you have any questions? Basically this is part of the documentation and also your relative documentation should be in change lock label of the image because atomic up contains some change lock label define so we are trying to use it but basically what's changed is in the documentation. It's not in the image inside because it's hard to put it somewhere it can be easy to put it in standalone and for example have default commands to list it but it's pretty hard in OpenShift that it's not a good way to show something about the image. All tools I don't think that all are in brew but Docker scripts the squashing one this one I'm not sure if it's in brew but it's part of OpenShift service to use internally and it's a tool you can really use. I think it's part of Fedora now and it's on PiPi but I don't think it's built in brew but the configuration framework will be built in brew because it will be RPM based and installed but we are using this in production and we are maintaining it so you are free to use it and even if you will have any troubles just to put any issue to the upstream and we are really using upstream and we are not patching the versions and we are working on the GitHub repositories so you can reach us there Any other question? In upstream like in this course here on GitHub we still have this layer that you've just described like first install the base then install Java because we have some request to use a different Java oracle Java or use IDM Java inside for JBOS but in a container so is that possible to do with just changing this middle layer? Yeah, but you have to rebuild all the images but basically what you will do you will just change one line and it will work Okay? Have you thought about mounting Java on another container? Yes, we were thinking about it a little but for now we are just for the OpenGDK and because I think that for us it's pretty hard to ship oracle GVM in image or something like that and you know just using just concatenating two images I think that's the stuff the customer needs to do because I don't think for example using oracle Java I don't think that you even can fetch it automatically so you have to have some cookie and accept some Yeah, you can do it We don't have it in documentation and I think this should be part of OpenShift documentation for all images, not about express images because but yeah, that's a good idea and I think that we should document it in upstream for world fly 2 because we are managing that We have to try now Yeah I don't understand that you can basically do all the same things Yeah and basically with CCT it's even easier because you will just inject the RPM with Java it will automatically find it install it and CCT has support for alternatives so you will just specify the version so it's a really one line command for you Okay Yeah, basically that's I think that it's what we are doing and all our tools are integrated and do each part but yeah that was one of approach we considered but you know replacing Dockerfiles and creating custom builders it's hard but I think it's a good idea to do it because there is Racket, there is OpenContaining specification all the stuff just have one tool which can build container for any provider and to get rid of Dockerfile because you know I think that Dockerfile is nice it enables containers to spread but now it's better limiting Okay, any other question? Okay, so thank you 1, 2, 1, 2