 So then, welcome to our talk today, supporting multiple major versions of Bosch releases with Concurz. Before we start, we will shortly introduce ourselves. My name is Stefan. I'm a platform engineer at any Nines. I'm working with CloudForms since almost four years now, mainly working on Bosch, on our Bosch releases for our data service framework, working on the brokers, doing trainings, and also some consultant tasks. Yeah, hi. My name is Michael. I have been working for any Nines for two years. I'm involved in the development of our any Nines data service framework, for example, for our data service broker. And I'm also involved in the development of our Concurz pipelines, so far from my side. So then, let's have a look at the agenda of our talk today. First, we want to explain why it is useful to provide multiple major versions for your customers. Then we also want to show why updates are necessary and what are the challenges dealing with the multiple major versions and the corresponding updates. Then we want to show you how we solved this in our company. And if we have time at the end, we want to show you a short live demo. OK, so then let's have a look why it makes sense to have multiple major versions of Bosch releases. First one is from the pivotal. There is an enterprise readiness criteria that you should support at least two versions of the service, so the current one and the one before. And as we want to offer our services also in the PIFnet, that's one reason why we support multiple versions. Another important point is that you need to grant a transition phase for your customers, because sometimes there are breaking changes and you cannot just switch the version of your database, for example, so the customer need time to change their code when they want. And then it's only possible if you have multiple versions available. And then, of course, there are customers that they want an older version, so you cannot remove it and have to support both versions or multiple versions, depending on how many versions you have. For example, for PostgreSQL, right now we have 9.4, 10, and 11. And depending on the customer, there are three versions supported on some customers, only two. So it depends really what the customer requires and needs. And the last point is that for some versions there are specific customer adaptions, what we added for a customer that is required for one specific version. And that's, again, a reason why they cannot just change or go and work with the latest version because they have some application that requires features from older versions. And that's also one reason why you have to support multiple versions of a data service. Fortunately, there are four major reasons why updates are indispensable. The first one is you want to provide your customers with new features. And then we have the security fixes. Human beings make mistakes no matter how industrious or ambitious we are. We will not always be able to avoid them. Some of them may be negligible. Others may lead to security issues. So for that reason, it's necessary to solve these issues as fast as possible. Then we have the bug fixes that all bugs are security relevant, but they are ugly and impaired user experience. That is why a normal company like any Nines, for example, will fix any bug in minimum time, too. And the last one is the third-party updates, as we do not reinvent the wheel in order to focus on the solution to the given challenges we have. Normally, third-party software is used. Since the third-party software is subject to the same rules as their own, software updates are also essential here. To profit from these updates, it's necessary to update the used third-party version. So now we talked about why it's useful to have multiple versions and that you still need updates. And of course, there are some challenges that occur when you have multiple versions and try to do updates. As what Michael said, security and bug fixes are important and need to be as fast as possible delivered to the customer. So it can be that it's a security fix that needs to be applied to multiple versions. If there is a problem in the Post-Quest Square, for example, in our cluster management, we have to apply it for each version of the data service. So you have to make sure that each service that you provide has this bug fix or security fix. The same goes for the features. It can be that you have to apply the feature for each data service version, or each version of a service. For example, backup and restore or encryption. That's usually some features that should work for each service version. And that's why you can have to apply them to everyone. The last point, of course, if we are using third-party software and they also do updates of their software, they fix security or bugs. And they add new features to their software. So we also need to update these third-party software. That can be only one data service version. We have to do the updates, but it can also affect all services. And of course, you don't want to do it manually. Or at the end, you cannot do it manually. So we right now have seven services in three to four versions, so you have to watch everything. You cannot watch each data service version. You cannot watch each third-party update to check if there's a new update. So you have to automate the work to do. And in our case, we are using concourse. And so in the later, we will explain how we did it. All right. Yeah. Then, of course, NINES has already thought about all these challenges and we come to our own solution. In the following, we want to show you how we adapt the major product versions into corresponding Git branches. Then we want to show you how we provide concourse pipelines for all of them. And finally, we want to show you how we supervise the third-party libraries with our own concourse resource in order to remain up to date with the third-party update software. So let's first have a look at the Git branching model. I guess you already saw this picture. This is from describing the Git workflow. It's from Vincent Riesen. 2010, he wrote or described how he managed versions of the software. Basically, you have a master and a developer branch. Master is for releasing your software. And the developer branch is a mirror from that. You create new feature branches or hotfix branches. And the idea is that you have, in the master branch, only stable versions of your software. So as we also want to provide stable software, we came to this workflow. Unfortunately, it doesn't work like this for our data services. As I said, you have to apply features, bug fixes, security fixes to different versions. So we decided that we put each service or each version of a service in one Git repository. For example, then we have a Git repository for our PostgreSQL HA, HA release. And for each version, we have then their separated branches. So in our case, the master branch is the latest version that is currently out there. So for PostgreSQL, it's 10, 11, sorry. And then we also have branches for PostgreSQL 10, PostgreSQL 9.4. We decided to do that because if you have three different Git repositories or even more if there are new versions, you don't have an overview of what you changed already and whatnot. It's really easy to forget one version. And then you have to do it again. And so if we have to apply a security bug fix feature whatever for one version, it's really easy. So we create a feature branch from the version branch, for example, from PostgreSQL 9.4. We create a feature branch, develop the feature, test it, merge it in PostgreSQL 9.4, and it's done. It can be a little bit tricky if we have to apply these features then also to all versions because now we have different branches and we cannot merge anything across the branches. So what we do now is that we select one version where we want to implement this feature or security fix, test it there, and then cherry pick the changes to the other branches or create on the other versions, version branches, new feature branches, cherry pick the changes there, test it there, because everywhere can happen some changes in the logic. So there's a difference, for example, for the cluster management in PostgreSQL 11 and 9.4. So we have to make sure that we don't destroy our own service by just merging from another version branch. Sorry. That's for the Bosch releases. With this Git flow model, we also write our own broker, our deployer, so there we have only one version that we are supporting, then we can apply this model, but unfortunately not for the Bosch releases for our data services. Of course, we still have some issues, like the cherry picking, it's not as easy. You have to do it sometimes again and see what goes wrong. But for us, right now it's working better, it's working, then have different Git repositories. OK. After we introduced our Git branching model, we now want to show you our conquest pipelines a little bit closer, and one would like to explain how they work together. The pipelines of the various major versions deviate only slightly if at all, and therefore we ask ourselves the question, how do we manage to create a pipeline for a new major version with as little effort as possible? Yeah. In order to solve this efficiently, we choose ERB because Ruby is one of our standard tools. So we looked what are the core components and what are the difference between the pipelines in different major versions. For the similarities, as we can see on the left side, we create an ERB template that we use for all our data services with for loops, if blocks, and what else ERB has to offer. More about individual jobs we see later. As you see on the right side, we outsourced the version-specific configuration in a separate configuration file. Analogous to the pipeline definitions, this configuration file is also in the YAML format. First in our configuration file, we have a general section in which we have stored the configuration settings that are the same for all our versions. Since we maintain our major versions all within one repository, the Git repository, as well as the GitHub paths are stored here. Then in the release section, we have the specific version configurations. This currently includes the name, which we use for the job to prefix them, and also to separate these jobs into groups for better overview. Also, this we will see later. We can also configure the initial version. This was necessary because we already had existing releases whose version numbers were already advanced before we introduced the concourse pipelines. And of course, we also have the option to define our master branch. And under the blobs, we can then determine the third-party software used by that version. This information is necessary in order for us to observe and update the third-party software we use for version. For this, we need on the one-handed repository, which we observe to check if new versions are available, and on the other, the URL to download the blob. As you can see, we use a placeholder here to set the version if we download them, if we download the blob. And an important aspect is still the version constraint. This allows us to determine which version we want to monitor to ensure that, for example, in our MongoDB 3.6 push release, we only use versions of the MongoDB 3.6 series. At the moment, these are all the version-specific details for our data service that we used. But of course, we have the possibility to extend them at any time if necessary. OK, and then now this is our pipeline looked like deployed with the template and the configuration file. In this case, it's for MongoDB. So on the upper left, you see the different versions we support. Yeah, we have some different jobs here on the left. Upper job is the third-party updater job. He uses our own resources that Michael explained a slide before to monitor, in this case, a MongoDB repository. And if there's a new version, this job is automatically triggered, downloads the version to a feature punch, tested it, if it works, updates the blobs, merge it in the version branch, and creates a new release of this version. Beneath, there's the ship job that we are using for creating them, the new release version, again, with testing, updating the blobs. And then at the end, we ship our MongoDB release also to GitHub to create on GitHub a release with tags and with the tar file. And there's also some smaller job that increases the numbers. So if there is a new run, then we don't overwrite our number. Down there on the left side, there's also these tests. So if we implement a new feature, we create a pull request. Then this job is automatically triggered, and unit tests will be executed, or even other tests, depending on the release. You can also then, in GitHub, what we did, set a check that you are not allowed to merge this feature branch into your version branch when the tests are not green. So we have to really make sure that it's working still, even if we updated the feature or the version. If it's merged, then we have to actually click manually on the ship button so that the ship process starts. Right. This is really, it looks, we can see it later in the demo, it looks almost the same for each service. And as you can see, if you have been to other talks or looked in the internet, it's almost the same architectural design of the pipelines. So we didn't came up with a new style of setting up or pipelines or configure them. Of course, one little issue we have with this solution, as I said, the third-party updates is automatic, completely automatically. And if you have a feature that is not released, but already merged into your branch, you get this feature with the new blobs. So this is one thing we are currently looking at and try to solve this issue. But for most times, it's working perfectly. We don't have to, so we can really focus on our solution, on our features we want to add, and don't have to care about the third-party updates again. Okay, now we have some time left, so I want to show a short, there's my browser. I see me all the way. Aha! Yeah. So this are currently a test pipeline that I uploaded. We have here the tabs for the different major releases. And yeah, then we have here the configuration of the pipeline, which I mentioned before. And if I now want to create a new release, it's just to copy and paste this block, for example. Then adapt the corresponding options. For example, we want to create a MongoDB 5.0 version. We want to start with the version number one. Okay, we have to set a new branch. For example, master 5.0, and to adapt here the constraint for the version. And then we only, then we have a little script to automatically upload these after we change this, and then we see what changes are made in Concurz. We accept this and then, as you can see here, we have a new tab for our MongoDB 5.0 version. So this is all what we have to do if you want to create a new major release. No, it's easily fast. And yeah, it's just a little bit copy and paste. That's all. Yeah, okay. That was all. So yeah, if you have any questions, you can ask us right now or even later. We are here at the booth. You can also talk there with us. But if you want, you can ask also right now. Okay, then thank you for your time being here. And have a nice day. Bye. Thank you.