 Hello, welcome everybody. I hope you can understand me well. I have a bit of a cold. So my voice is awkward and my hearing as well. So please also for the questions, speak up a bit so that I can hear you. OK, so my presentation will be about throwing together Jenkins, Ansible, Power application deployments. As a starting reference, you're probably here because you know the problems or challenges of deploying applications. And I basically want to guide you along the way how we do this at our company or the place that I work at. Initially, of course, the problem comes to the IT team. There is a new application. We need to run it. And then if you're not already on the Kubernetes train, then it's basically a couple of steps that we'll always repeat. You create a new virtual machine. You put the application on it. And everything is working perfectly fine, hopefully. And I think we figured out a way in the last year how to do this so that we can also trust in our deployments and that we can do these deployments with confidence. Just some background. I'm here with two of my colleagues. We work at a research campus in Vienna. This is a campus of several research institutes. There are roughly 800 people on campus. This is mixed scientists, technicians, and so on. And we are a rather small IT department. Aside from some maybe special things, we do the regular operations of virtual machines, storage, network, and so on. As mentioned before, we are still a Kubernetes-free zone. But also we would hope to get there at some point. Last year, we started to design a system or to build a system that would really support us in deploying containerized application. The focus here is really also on containerized applications. Where did we come from? So there was a setup in place. This was some more or less un-maintained form and installation. We are running all kinds of applications. So there is no strict business use case. But oftentimes, it is also from the scientific side, for example, that they come up and say, we have this application that we run. Or there are vendors that bring in certain applications. And we have no choosing in this. So we are the service provider internally. We have to run it. And also, it's not our application, so we did not develop it. And well, at the beginning, it was oftentimes, if it was dockerized already, then oftentimes what happened is you spin up a host. You copy some pre-built Docker compose file there, run it. And that's maybe good enough. You take a copy of the terminal output, paste it in the wiki for documentation. Works. We do have our own Docker registry, yeah. Exactly. But of course, the problem with this is also it's not really reproducible. If you lose a machine, can you restore this? Maybe. Hopefully, the pasted code to the wiki still works. You try it again. But you cannot be sure, really. So to improve on this, I want to do this basically in two steps because I also said I will be talking about Jenkins and configuring Jenkins. So the first part will be more the using the system side, so the actual application deployment. I want to basically show you step by step what the things are that we do to get the standardized application deployment running. We use Ansible for this. And then we also try to take measures to ensure that Ansible is also itself doing what we expected to do. So our automation tools are fulfilling our expectations. In the second part, I want to go more into detail how Jenkins gets deployed. Because this was also before the situation that you may or may not be familiar with. You log into Jenkins. You get this nice red icon that shows 17. And that's the number of available software updates, plug-in versions, other errors that should be fixed. And then from time to time, somebody goes there, presses the button, update all plugins, and then you hope that it's still OK afterwards. But it's more or less of a lottery. So we will look at how to configure Jenkins container image and basically how to bootstrap all configuration in that image from scratch. And this is both the configuration of Jenkins itself, but also the configuration of the jobs that Jenkins is running. OK, so let's get started with an application deployment. Still some manual steps involved, right? What we do is we put one application, which can be a set of containers. But usually it's one application on one virtual machine, which is then running all those application containers. This works initially so that we go to a satellite. We basically tell it to provision a new virtual machine. This will do this then on our virtualization infrastructure. Since we are deploying Red Hat, we run Kickstart for bootstrapping this machine. And the Kickstart is mostly default, but it prepares the host so that Ansible can immediately come in after the system was deployed initially. This means setting up a service account, allowing the service account to escalate privileges, and then also triggering a callback to Ansible Tower, which is this Ansible execution engine basically. So in Ansible Tower, you can put playbooks in, you can manage your inventory there, and then you can tell it to execute certain playbooks on a given inventory. Ansible Tower does this. And then eventually we have a host that has a minimal base configuration. So at this point, we have an operating system running that has basically packages installed. We have Kickstart that triggered the Ansible run, and then Tower is coming in and applies what we call our baseline playbook. This is, after the initial operating system installation, a minimal set of configuration that we want to do on every host, no matter what this host will do afterwards. This concerns, for example, remote logging, the usual things like our NTP configuration. Usually, we also will have a firewall deconfiguration. For the administrative side, of course, we need to be able to access this host with our user accounts, because, of course, initially the host is provisioned without any IDM, so identity management integration. So we want our administrative accounts to be able to log in there to have pseudo access. And also, if there are more host names, then just the initial host name that we chose in Satellite, we also want to do some setup in DNS. At the very last step, this baseline playbook will put a version marker similar to, basically, LSB release in the Etsy directory, so that we also know this host was last touched by baseline in this version, so that we also have an understanding of how this host was provisioned at which state. Yes. Now, we have admin access. We will repeat this applying of the baseline, not in a very short interval, but, let's say, on a daily basis, we will rerun baseline on all those hosts. This is also to ensure that the config was not changed accidentally or otherwise. And then also, as I will show later, and as also the talk before has actually shown, we run most of these things through Git, so there might be also changes in the configuration that we want to see on the machines. And then it is also important that at some point, these changes will be rolled out. So the baseline, as I said, will deploy more or less regardless on all hosts. But of course, we have the possibility for certain hosts, either individual hosts or grouped hosts, to change how they are configured specifically. For example, I said we bring up syslog for remote logging. But if we have a test system or a development system, we don't necessarily want these logs to end up in the production log, because they might be messy and you might see errors that we are actually not interested in. OK. So now that we have a machine, we can give our application deployment a go. So we're using Ansible, right? We start writing a playbook, deploy my application. But this gets complicated quite soon, because maybe we have a test instance. And we also have the production instance. Do we copy the playbook? OK, there are variables, of course, that we can use and replace them. But really, also having all these playbooks in your repository, you see this gets a long list. And as I said before, also sometimes you have changes in the way you want to handle things, then you'll suddenly have a change to do in many, many playbooks. So actually, maybe we can find a common denominator between those and see if we can do better. So what I said is that we are trying to, and I would almost claim we managed to just run containerized applications. So if you look at the container, the nice thing is the container isolates the application. So there are just certain things that you have to put onto the container from the outside. For example, maybe the use ID inside or the use ID that container is mapped as, you will have to bring in some storage eventually, since most likely not all your applications are fully cloud native, you will still rely on classic, let's say, NFS shares. And then, of course, application configuration itself is also often done through environment variables. These are also things that we need to inject to the container. The other thing is, obviously, many applications are web applications. Then immediately, you will need certificates that have to be with this application. And, of course, looking at databases, for example, or other things, you might need certain maintenance tasks that you also want to be run, and some of them will run on the host. And there is also a mechanism for bringing these on the host. And with this set that we found, we basically can handle most of the applications that we encounter. So as I said in the beginning, we have our baseline playbook that provisions the host initially. And in the same way, we call it App Generic. But it's another playbook that will, again, apply a set of roles. The most important role in this sense is a role that will bring up the container environment. For us, this means basically installing Portman and configuring the containers to run. Actually, there was also a presentation about Portman that now does a part of this by itself, which is setting up a system D service. So in our role, we also create a system D configuration, then register this with system D, and basically configure the services active. So this is the same playbook that will be applied to all application hosts. But the application configuration itself is in a single host vars file, which contains all the configurable and several variables for this. Also, obviously, not all applications will always fit in the container or are already containerized. Sometimes, there are really ugly things that we see. And we have to deal with them as well. And for these cases, we will also have their own specific playbook for this specific application. But still, the total number of playbooks is greatly reduced. What does this deployment descriptor, let's call it, look like? So for this generic application playbook, the role that uses this is the set container role. And it is a configuration file in Yammer in the end. And I have to admit, I'm not that familiar with Kubernetes, but I think there are certain similarities with regards to how to make environment variables or how to provide storage. So this is what we can basically configure on a per container basis. And as you can see also, the dash at the name gives it away. It's a list of containers. So this list of containers will be configured on the host. Each will get their system de-configuration. And it will be ensured that this service is running. So from what we've heard up to this point, we are actually here for something with continuous integration and Jenkins. But this is the process that we do. And already, we can see a couple of things that are actually a fit for continuous integration. First of all, Ansible roles. They basically defined the whole deployment logic of how to well configure any entity in that sense, which could be a container or could be the firewall. But the point is everything that is actual configuration logic, we have packaged in a role. And we don't do it in the playbook directly. So this is also basically an isolated unit in a way. The configuration is more or less all done through the host files and group files in Ansible, which is also good because it's a text file and it's a YAML file. So even though we cannot really check the content, if the meaning of the configuration file is correct, but at least we can make sure that it's correct YAML and that later steps in processing will understand what to do with this. And something that is obviously already established, I guess, is the whole build process for containers. So if the application is containerized, then we know how to deal with it. So first step, let's do something with those Ansible roles. There is the molecule framework. You might be familiar with this. This is basically a testing framework for Ansible, more specifically for Ansible roles. You can define several test execution environments. These are called scenarios. And each scenario has a defined list of steps that will be executed. So there is a step, for example, to validate your YAML syntax. There is a step to prepare the environment that your Ansible role will be executed. This could be a virtual machine, or it can launch locally. It can launch a Docker container into which this role will be applied. What it also offers is the possibility to execute actual unit tests in that sense. For this, it is integrated with the test-infra framework. So the name of the framework is literally test-infra. OK. And this basically is quite a nice feature set that we can use to run tests locally, but then also later on to run these tests in a fully automated way. So of course, there is also a configuration for molecule. Of course, it has to be in YAML. As I have mentioned before, there is a way that you can run those molecule scenarios inside of a container. One thing to note here, this comes from the platform section there. You'll see that we can inject environment variables there as well. In the most simple cases, you wouldn't need this, but this is already thinking ahead. When we run this in Jenkins, for example, there might not be one of these instances around, because we are, for example, testing a pull request where we are testing the head of the feature branch and then the merged situation, just to know that the merge will actually be a good merge. And then we have multiple instances of the same container, and that would be the same container name, which would mess things up. So we can add our own little ID in there to guarantee that the container names on the execution host are unique. Another thing that is also not turned on by default, because for the local machine use case, it's not really required, but you can write the test results as a JUnit XML format, which is otherwise used by software development frameworks or unit testing frameworks. But here, if you want to collect those test results in an automated way, this is also good to have. So more and more building blocks are coming together. Let's put this in Jenkins. If we think about where Jenkins is originally coming from, I would think it's software development. And I would argue that you can also claim to use a software development process for Ansible Role development. It's not a programming language code, but still it's execution instructions. You get results and so on, so the process actually is the same. What we also do use is running multiple molecule scenarios. An example of that would be, for example, let's say you have a role that joins your machine to Active Directory. So you get the user identities from Active Directory. Maybe should this role also be able to un-join you or to leave the Active Directory? Then we can do this in two separate scenarios and test these also separately without one depending on the other directly. As I said also before, for playbooks, we don't do logic tests in that sense because we think this does not really make sense. It's anyways actually just applying roles. But of course, we can make sure that the syntax is correct. And as we always do in Jenkins, we collect the build results or the test results. It was great that the talk before me was actually about GitOps because I think this is also quite fitting. We do this for Ansible Role Development. So we develop new functionality on its own feature branch. Then the pattern that we use, and I think this is also hard to figure out which pattern really is the best one for you, but we then merge all feature branches onto develop after some cleanup. And from develop, we have a code review merging this to master. We use the master branch, the head of master branch, as basically our verification staging environment. And then behind the head, we bring in text that basically mark a release. This is true for our Ansible roles, but this is also true for the other repository, for example, that holds all those Ansible bars and those playbooks. So basically, this is also defining, versioning for our infrastructure configuration. Now, what does this workflow look like in Jenkins? So we have our developer. Developer will push code. This can be a tag, a branch, doesn't really matter. This goes to the Git repository. The way we have set it up is that the Git repository itself will then inform Jenkins that there are new commits visible and Jenkins will know how to handle this. In most cases, for the Ansible role specifically, it will then say, okay, so we test this now. It will get the code for the Ansible role, set up a molecule execution environment, run the defined scenarios, and then collect the test result outputs. Looking at what this looks like in Jenkins, this is an output from, don't laugh at me, we have a role that sets message of the day. So you can see this is actually nice in Jenkins pipeline. You can have those stages they're called. So these are logical groups of pipeline steps. And then you can see exactly, we prepare a container that will run molecule. Then we do the checkered of the Ansible role source code. And then since we also have molecule scenarios, sometimes we have not everywhere, but we have one or two roles where we have like 10 scenarios that we want to test because we are testing different aspects of behavior. And then we can also run those scenarios in parallel because the scenarios are completely independent of each other. So we can save time, run this in parallel, and then in the end go in and collect all the JUnit results. Speaking of test results, you probably know those test result graphs that will show you, okay, this commit introduced some errors, some problem, and then the next commit hopefully removed this. So we can make basically, we can make the mistakes on the development branches, but hopefully on the master branch, this is clean. What you can also see is that this is quite a high number of test cases. This is basically all host vars and all group vars files. For those we create programmatically, we create a test case. So the test case is basically called the same as the host vars file is called. And then we do a syntax check on this because since it's just static content, just configuration data, there is not much else that we can do on this. Okay, so this is the end-able part. For the container, we have a very similar process. This is also probably more established already. The beginning is quite similar. We have a repository that contains the container definition. This is in the minimal case, it will be a Docker file and the Jenkins file. But of course, usually you adapt the upstream container so you will have other modifications in that repository. You push this. Again, the repository triggers Jenkins. And then depending on the Jenkins file that it will find in this repository, it will decide what to do. So basically, it will for sure build the container image. But under certain conditions, it will push it to the registry. And also under certain conditions, it will trigger tower again who knows how to deploy this application. So this would then basically be triggering the aforementioned application playbook with an inventory that is specific to this application which might be just a single host and to apply this new application. In the end, if you're just rebuilding the image, then this will basically just update the container image. Yeah, and then happiness ensues. Our customers are giving us the thumbs up and can continue with the day. Again, looking at this in Jenkins, we again have several steps. We get the repository for the container. We build the container image. We have an optional step where you could verify the container image, the software in the container image. We don't always do this, as you can see. But this is also arguably necessary because you could also argue if the container image builds correctly, then everything is okay. So this might be optional. And then again, separate steps for pushing the container to the container registry and the last step to trigger the tower run. Now that we have more than two applications running, this will of course get often tedious because this is not the shortest pipeline that you can imagine and you are reproducing the same code everywhere. You have several container repositories. You always have the same Jenkins file that contains the same pipeline. So a feature that Jenkins has is those libraries. So you can basically have a shared library of pipelines and then just reuse this in the Jenkins file. Of course, the idea is to reduce the code itself. What would have to be changed if you change anything on your pipeline, you don't have to go to every repository and change the Jenkins file there but you just change this at a central location. The other thing is also, this should be easy to use, right? You shouldn't have to write your own container pipeline every time. Some people are also not really interested in doing this but of course they would like to use the feature. So it's nice if you have a short descriptor that does the right thing for you. So basically this means we will have to write our CI CD library. This is an example for example, the configuration of the Docker container image pipeline looks like. So basically the most important part is this lower part built Docker image. This will just tell it by default it goes for the Docker file so it will build the image from the Docker file that is right in the root directory of the repository. Then it will tell you which branch is to push to the registry and if there is something to do for Tower which in this case you can see on top we do something on Tower in case of changes in testing or developed branch and if we see a new tag. Okay. So this is basically most of the things that we do for application deployments. I also said we have a look at how to configure Jenkins itself. As I said before there is a lot of things that would have to be repeated on a per repository basis but we want to try to avoid this. This is why we go for the libraries. A Jenkins library is basically a bundle of groovy code that exposes functions to your Jenkins file and you can then call this function. What the function has to do eventually for being used in a Jenkins file directly is it will have to return a pipeline because this is what you would anyways put in the Jenkins file, right? You have the pipeline brace. So this is a closure that just defines what the pipeline actually does. Here is another quick example for how we test our molecule. So this is basically the complete Jenkins file for testing an Ansible role because the role is anyways already in the repository. We tell it there are two scenarios that we want you to run, default and remove and we do not want these scenarios to run in parallel. That's it. The rest is being taken care of the library. What it would do then is, again, it would get the Jenkins file. It would check out the library, execute the function that is in the Jenkins file and then this does the whole business of setting up the molecule environment, getting the code for the role, executing the scenarios and collecting the results and publishing them. Okay, this is where I wanted to show the Docker deployment in a bit more detail. So for example, the build Docker image is basically the function call that will return the configured pipeline and the rest is just trying to be clever about defaults and looking up certain configuration items. So for example, here we say, okay, the namespace in the registry is defined. We say we are interested in two branches to get always, if something new is built, to get this pushed to the registry. In this example, we also say that there are multiple Docker files in the repository so you can see from the extra images, this is actually defining two Docker images that will be built and both of them will be pushed the same so they will get the same tag and they will be pushed under the same conditions. Okay, so if you look at Jenkins, Jenkins itself can also run as a container so should be easy enough, right? What we want to try to avoid is trigger a redeployment of Jenkins while it is building itself because this is like shooting yourself in the foot. But the important steps for the Jenkins setup is the initial bootstrapping, so getting the actual Jenkins global configuration in place and we also don't want to add projects manually to Jenkins because it should be as easy as just creating a new repository. We want to auto discover new Jenkins projects and also from an operational perspective, it's also nice, the example from the last talk was also like bringing up your infrastructure again from completely deleting it in 15 minutes. The general idea is here also true. You completely use your Jenkins installation, it is no problem, you just redeploy it again and it will bootstrap itself to the full configuration. So there is an upstream Jenkins image that already has some kind of configuration tunings to it. You can place a plugin.txt file in this image. This plugin.txt file will be picked up during build. I'm not sure if this is in the upstream image or if we did this modification. Regardless, we have a definition of the plugins that we want to install. And these plugins will already be downloaded and put in place while the container image is built. We don't want to do this while we are starting up. This would also be possible, but of course then we would have longer startup times and we would also be messing with the container image itself. So this should be baked into the image. Then there is also this fantastic config as code plugin for Jenkins, which I also just learned about this a couple of months ago. That allows you to basically do all global Jenkins configuration that you would do through the web UI. Again, via YAML files. This covers probably, I would say for us, like 90% of the things that we need to do. And then if you need to go one step further, you can still add more groovy scripts that will run in a privileged context during the Jenkins start. And then you can anyways manipulate basically anything. For jobs, the Jenkins projects, the jobs that Jenkins will run. We also don't want to have any manual work. As I said before, the typical use case would be that somebody can just start working on a new repository and don't have to go back and forth between various systems setting this up. So this should be discovered automatically. We try to group by concern, by folder. So we also have internal customers that also use this system. And then we can say, for example, okay, this department, they get their own folder. The folder is not that of a special feature, but it allows to scope certain things to this folder. Specifically, for example, secrets. You don't want to share all the secrets with everybody so you can have their private secrets in their folder. The other important thing is, as mentioned before, we want to reuse code. So we use shared libraries. And the way this works is that we have a so-called seed job that will basically set up the auto configuration and the auto discovery for those folders. And then these will each by itself take care of the rest. Yes, so this is what I've briefly mentioned before. Tendency, it's probably not really strict. Tendency in that sense that it's super isolated, but I would argue it's good enough, definitely for our use case, we have kind of friendly customer. It's not like one department wants to go to war with the other department and they have to be kept apart. But we can do a good enough separation between this on the basis of folders. Because what we also define on folder level is the access permissions. So we can define certain users or groups that are allowed to trigger certain actions on the folder and then also on all items below the folder. For configuring these tenants, I call them now, I said we do this through the seed job. And the seed job will again read a YAML configuration file if you cannot read it perfectly, that's okay. The point that I wanted to make here is basically how this is concept wise set up. So for the folder, we decide certain base configuration on where it gets its code from. This can be for example, GitHub or our internal source repository. This is what we call the repository provider. Then we have a section of credentials mapping. Credentials handling I think is always a topic, how to deal with this, keep this in git. We use one password as a credential store and what we have here is a mechanism that will basically use the service account, go to one password, extract the secrets that are configured and then add them to this job configuration or to the folder configuration. And in the bottom it's a simple list of basically access group names and the permissions they should have inside of Jenkins. Okay, so at this point, we basically have the global configuration of Jenkins more or less baked into the container image. And we have also all the job and project configuration auto discovered. So I would say this is basically a stateless configuration in the sense of if you lose your Jenkins instance completely, you can bring it up from scratch and it will start working again as before. One fact about this is that our configuration that you would do manually in the web interface will also be overwritten either on application restart. So when the container images, the container is restarted, the global configuration will be brought in exactly as it was configured in the image. And the same is also true for job configurations. So you maybe can override some configuration but as soon as the seed job runs again, it will make it as it was configured. The only thing that we are missing is basically archiving the build results. And by this I don't mean the, for example, the container image, but the actual state of the build so that, I don't know, project Jenkins image. Build number 57 was a good build. This information we lose but certain artifact repositories also allow you to add this information to the build artifacts that you upload. I mean, hint, hint. So artifact, for example, is one of these. So you basically get the information of the build on which build machine it was run and so on to add this to your actual binaries. So looking back at what we have discussed is basically the steps that we do for automated application deployment. I gave you a brief overview of how our integration process for Ansible role works. And this is really something that we rely on because it is really a so much more relaxed kind of work knowing or really trusting in your tooling is having, being able to trust in your tooling is great. Of course, we have seen the shared Jenkins libraries and we can always recover from scratch. So for us, the next steps are basically, as I said, we don't have Kubernetes yet but of course this is also something that we look at. Our hope would be that having these kind of processes, everything in Git, everything as far as possible automated make it easier for us to eventually move everything onto another platform. And one of the learnings for us was also to see that our customers would also actually adapt to this because in a way this is also, I would say, leading by example, you have to show others as well what you think the best practices are as an IT professional and have them maybe adapt one or two of these. So, thank you. This is basically where I'm at the end. I would take questions now or I can also show a little bit source code maybe from the, how we deploy the image. Maybe a quick question first. I was wondering, are you using the same tag or using version of the image tag in some way? So the question is how we handle tags on containers when they are being pushed to the registry and then later on deployed. So the answer to this is yes because we try to follow this somehow, right? So we have, or maybe also the other example would actually be in this year. So what we say here is that for, sorry, for the push branches setting, in these branches we will always push head and we will tag it as the name of the branch. This might be especially interesting when you're developing and you want to iterate fast, right? For tags, this is actually a tenant setting as we have it so this is defined in Jenkins more globally but we define should git tags be handled or not at all. If we decide to handle git tags we will always push the git tags as basically also the container image tag. In Tower it then depends basically also a bit on the application. So for example, when we tag a git tag we will also push this one as latest. So there we have a rolling tag and then depending on the application we can decide okay this is an easy application we always run latest because we notice we do this often, we don't care so much. For other things we might say name your tag explicitly it has to be this version. So it's both. This is actually again configurable because you can also have parameters into the Tower shop. So we could say push this tag or deploy this container image tag on the host. Yes. So the question is for how many roles we have and how long a testing cycle takes. Roles I would say maybe a dozen that we rely on for these regular parts of application deployment. So the number of roles is like 10, 12 roles I would say. Okay, okay. So since we have the roles in the git repository we always test them when they get pushed. We test all the branches, everything that gets pushed we test and then we have the test results for this. And exactly. And then how long does the test take? Sorry, it depends on the role because of course there are certain things like, I don't know, I need to talk to Active Directory to do the AD join with SSSD. This will take a moment, it's not only depending on the role. Certificate handling also everything where you have to talk to foreign APIs it will take longer. From seconds to minutes per scenario it's, I don't think there is a good average number to say that's, no, it depends on what the role does. Yeah, but I mean, okay, let's say minutes. Yes, but the point is you don't really see it. I mean, you could argue, okay, you test locally so you want to have fast, you want to have fast testing usually, right? I mean, that's the point. But then on the other hand you also have it done by somebody else anyways. I mean, it's not always the case that you have to wait immediately for these test results, right? Yes, sorry, how do I deal with the build host? The build logs, yes, exactly. This is exactly the point that I was bringing up. The results of the actual build, they are hard to archive. So we are using artifactory as a registry. There you have the possibility to upload build results as well. And the nice thing if you do it right, you can link them directly to your binary artifacts that you place in there. And a part of this could be the build log. But we don't have a solution for this yet. If there are no other questions, then I say thank you.