 Welcome, everybody. This is the actually the second image builders special interest group meeting of the Openship Commons. And really thank you all for joining us. And if you have any questions or suggestions for upcoming topics, please send them to the mailing list or type them into the chat room here and we'll have a conversation after today's presentation. So what I've done today is invited Christian Roldan from Protobahn, who has a large multinational deployment of OpenShift out there in the real world in production to tell us a little bit about their enterprise CI CD workflow for doing deployment with containers and building those containers. So I thought what we would do today is let him talk for 20 minutes or as long as it takes him to do his demo and we'll record that and other people can watch it. And then afterwards talk a little bit about with each other. I'll just unmute everybody. And you can ask questions of Christian and his team, a number of whom are on the call. And this is interesting for me from a couple of perspectives is this is definitely an enterprise approach to it. We've talked and we'll be talking with other people who are doing like if you only get one service that you have to continually integrate and build Docker and maintain a Docker image. So this is going to be an interesting talk today to hear about how they're doing it across the enterprise. Christian, I'll let you talk and introduce your team and folks ask questions in the chat and we will read them out at the end or try and answer them while the presentation is going on. So go ahead. Perfect. Hello, my name is Christian and I work at Prodman. So in this presentation, we are going to talk about our continuous integration for Docker images that we use in our past platform. We use OpenSheet version 3.1. So I think that this experience is really interesting to share with the community. So let's go to the next slide. Well, and Christian again, the past project manager here with me is Julio Fernandez, who is our past solution architect. He is the project leader for the continuous integration, the OpenSheet continuous integration solution. So I will try to do a really short introduction and after that we are going to explain more information about this solution. So what is Prodman? Prodman is a global company. We are in nine countries, for instance, Spain, Mexico, Brazil, the UK, United States, Portugal, etc. We are more than 5,000 professionals around the world. So Prodman is an IT company. We are the IT operation company for the Santander Group. So also, it is important to comment that Julián is from, is another Santander company. Isvan is a software company. So in this project that we call Global Pass, we are working together for Prodman and Isvan. So what is the objective for the Global Pass? What is the Global Pass? Well, the idea is that we have to design, deploy and operate a multirectional, private pass solution. This solution, this pass solution is based on containers. We use containers for everything. On this pass solution, we run a financial application. So for this point of view, the pass environment is a real critical environment. So we selected OpenShift Enterprise version version 3.1 for this pass environment. We started to work with OpenShift version 2. We were working with that version during three years, more or less. And we started with containers two years ago. And we decided to migrate to OpenShift version 3.1 more or less one year ago. Let's go to the next slide. Well, as I told you, the Global Pass is a global solution for Prodman. We deployed OpenShift, a different OpenShift cluster around the world. Right now, we have up and running Spain and Mexico regions. Our idea is to have two more regions at the end of the year. We would like to have Brazil and UK. So as I told you, we have two important regions, Spain and Mexico. So from this point of view, for us it's a very big challenge in order to operate and design this pass solution. It's really a big challenge. So what we have solved in this project, why we decided to create a continuous integration solution for local images. We have time to have the same building system for all the corporate docker images. We have a catalog of ten corporate images. We have, for instance, a spring mode, a microservice runtime. We have Tomcat runtime, Nginx, Cassandra, MongoDB, Rabbit and Q, etc. So we decided to design the same building system for all the docker images. So standardization for us is very important. So the standardization in this project is very important. We use the same repository structure for all the images. We have the same folder, the same scripts, the same metadata, etc. So the documentation for the project we use the same format. We use Markdown for the docker image documentation. The tab naming is very important. At the beginning of this project, we were using the latest level. We found a lot of problems. We decided to use another tab naming. Julianne is going to talk about this tab name and how we use this tab name. We also support our corporate images to port binary deployment. Binary deployment is a feature that we have in version 2. We implement some binary deployment in our corporate docker images. Also, we do some pre-processing and deployment of OpenShift template. Every corporate image has an OpenShift template. We use that template to deploy the docker in OpenShift. When we update the image, we also update the OpenShift template. After building the docker image, we deploy a new template in OpenShift. That is a very important topic that Julianne is going to talk about. Also, we push the docker image in our corporate docker registry. We execute different unit tests. Some corporate images are critical. For instance, the Tonka or the microservice images are really critical. If we introduce some bugs, we can affect the production. That is the reason why we decided to include unit tests in every docker image project. Also, when the image is built, if everything works okay, we send a change log to all the past customers. In the next topic, I want to talk about why we rebuild the docker image. The reason why we rebuild is because the operating system is updated. Every corporate docker image is based on red, and we use the base image from red. If we receive an update, automatically, we update all the corporate images. Also, we update when the product is updated. For instance, if we receive a notification that OpenCTK is updated, we also update the docker, the corporate docker image. We also include new features or automation scripts. For that reason, we also rebuild the image. If we need to update the template, we rebuild the docker. In the next slide, Julian is going to talk about the global solution. Also, we try to do our live demo. Okay, thanks, Christian. Hello, this is Julian Fernandez. As Christian has told, I work as cloud solution architect in Isban, and I'm collaborating with the Christian's team in the project ProDubang Global Pass. Today, I'm going to explain to you the continuous integration and continuous delivery cycle that we are running to implement beyond deploy our corporate docker image into OpenCTK. Before to start the presentation, I think it's important to emphasize that when we start working with OpenCT version 3, we decide not to use the sort to image tool for building docker image. How Christian has told before, this decision was taken because we have our own application lifecycle management team, and we are offering to the developers a binary deployment solution using the ALM jobs with our docker image. Okay, so let's go to start the presentation talking about the technical solution stack that make up the global pass application lifecycle management. Also, I'm not going to talk about it. Sorry, in this presentation, as you can see in this slide, we are using Jira and Trello to manage our project planning and to manage the support of incident of our customers. On the other side, ProDubang Global Pass is included into the new architecture of Santander Group called Serenity. Serenity is a group of assets that have a common roadmap and follow the current Santander cloud study. That is the logo of our group, our line, the Serenity Global Pass. Serenity Global Pass is only one of the lines that are part of Serenity. Maybe one of the basic lines of Serenity. Okay, and Serenity is using Confluence as a way to show all the documentation between the projects. So we have the documentation of the docker image in our GD repository and also centralized in Confluence. We are using Jenkins as the continuous integration engine. Everyone knows Jenkins, so I'm not going to explain how it works. I'll try to describe the jobs we are running in Jenkins. Okay, all our source code is stored in a GD repository. In this case, GD Lab. All the images, Christian told before, but all the images have the same standard GD repository structure. So far, most of this code is pre-made, but we hope that in the future we can pull this into a super short. The store binaries we use Nexus and to store and expose the docker image we are using docker registry. So we are going to be great to play or atomic registry, but we still have to make that decision. In order to test the docker image before building them, we use two tools implemented in Ruby called RSpec and server spec. I will dedicate some time to explain how to perform the test later. And finally, as you know, we are using offensive version 3, sorry, version 3 as platform as a service. Now I'm going to describe the workflow we are following for the continuous integration cycle of our docker image. Well, I have to tell you that today I'm going to explain the process of release of the image. The development process of the image follows a different flow. As you can see in this slide, first of all, we tag the docker image in GD, Jenkins job and release it every time a new tag is created on a docker image in GD Lab. So Jenkins check out the tag version into the Jenkins workspace and once the image code is downloaded, unit tests and integration tests are executed. If the tests are passed without failures, docker image is built with the corresponding tags and later is deployed to the registry. The next step is deployed associate template in offensive. Jenkins is responsible for updating the image tagging the template to point to the new tag. Once the template is put in the offensive, we upload the redmi-map-down file, I mean the image documentation to contrast. And finally, Jenkins clean up the workspace and notificates by mail the result of the execution of the job to the user subscribed to the image. Okay, we'll talk about the test. Now I'm going to describe how we're testing the docker image. Our idea is to use the test drive-in-development methodology in the docker image creation process. For this, we are using RSPEC and ServerSpec. Well, RSPEC is a behind-the-world drive-in-development framework for Ruby. This framework can be considered a domain-specific language and resembles a natural language specification. And ServerSpec is a Ruby tool very interesting tool which allows you to write simple tasks to validate that a server is currently configured. With ServerSpec, you can write RSPEC tests for checking your servers. In this case, our cooperated docker images are configured correctly. Okay, here we have two examples. You need tests verified that the docker file contains all the specifications defined. Really, it is a good docker-inspired command and verified that all the specifications that I have declared in the RSPEC file are included in the docker-inspired response. I can verify the participants, the measurement bars, working disk, package installer, users, or a lot of things. And the integration test verified requests to server into docker-image returns a way to respond. For example, in the docker-ad-age image, we are checking that on port 1880, we are exposing a default application that returs the system environment variables. And also, here, we verify that you can access to the docker-ad-age management deployment. Okay, go to the real world. Here, I'm going to show you our engineers' image test. Okay. You can see it. It looks good. Thanks. And we have the unit test. We check that we have a working disk in OPT-s-plus-pro-duban. We have two environment bars. We expose the... Oh, sorry. This is another file. One moment. Test spec unit. Okay. This is the file. Well, the working disk, the environment bars. We expose the port 81. And we have some package installed. Okay. Go down. Okay. The file. I'm going to show you now our integration test. Okay. In the integration test, we test two applications, one in the port 18, and another one in the port 21. And we expect that returs the code 200, the status code A, okay. And some HDD MR. Let's go. Okay. Now go out and go to execute the test. I have one script to do it. Okay. This is very quickly because I have the image pull it in my... In my machine. Okay. Now we have the first test, the second test and one test more. That is the integration test. Okay. Okay. So everything is right. Now I'm going to show you one Docker file. Well, sorry. What are spec files with errors to show you what happened when, when something like this. This is not. Okay. I expect with this one. For example, well, I'm not going to explain all the file, but here we are expected that the port 19, 19 is exposed. Okay. Now I'm going to execute this test. Not a face. That's why we do that. As you can see, you can, there are the errors and, and reduce the failures and explanation of the road. For example, here we want the, to expose the 1990 import, but we only expose the 443, 4, the 18 and the 81. Okay. So go back to the presentation. And once we have checked that Docker image have passed the unit and the integration test, the image is built. We have a build cell script in all the image to make the Docker build process. This script contains all the variables needed to make the build process. And Jenkins take care of tag the image with the same technique that we have previously created, created in our gene repository. The tag naming convention is the classic release convention my chain. Change. This is an example. And later we added the, the latest to this image. Finally, we boost the image to our private, private Docker registry. Okay. Now integrate our Docker image into OpenSeam. OpenSeam template with all the configuration required by the Docker image. Is in the standard repository that the rest of the image Jenkins job handles for setting the value of the Docker image with value of the new tag into a template. It's important to say that we aren't using the latest tag in our templates to avoid problems when we change the version of an image to our developers. Prodvango and past templates are touched as instant up and with the Docker image. Once the template starts to date, is deployed in the admin project, offensive of all available source. Well, as Christian told before, currently we have three production regions. Two in Spain and another one in Mexico. Okay. And to finish, we publish the image documentation. We are using conference as corporate. We kill to serve between us, the documentation and conference exposed an appeal rest to police documentation. Jenkins job, take the red mime down five of the image and pull is it in conference. Once the job is ended, Jenkins clean up the workspace and send an email notification to everyone who is subscribed to that. In this last slide, you can see some pictures of some of the features that I just described. The first one is our Jenkins last wall. Luckily everything is green. If one job is red, the good thing can go home until the job will be green. Well, this is only a job, but maybe this will be the way to do it. And the other features are to demonstrate the deployment of the image to offensive as instant apps here. And the relationship between the template and the document. Okay. And go to the real time. There's now. Oh, hey, hey, we can see the Jenkins as well. We have one red. So, so what will happen today? And here we have an example of one image. In the docker registry. Christian told us about the WordPress image before. And here we have, well, the first bill is not sold. The three release we have. And as you can see, the latest is the same image. It's the same image that the one dot to do dot zero release. And the last one is to show that the engine is template pointing to the one dot five dot one dot release of the image. Okay. Oh, sorry. Christian is told me that I have to maximize. Okay. So that's all. That's pretty awesome. Actually to see that whole workflow and action and the use of confluence and Jenkins. I'm going to open it up to questions now for anyone else who's on the call right now. Folks are interested in that. I'm, I'm very appreciative of the use of Jenkins in this. I'm a big fan of Jenkins. And in the, in the email notification process of all the subscribed users. I think that's a really interesting, interesting aspect of your, your workflow. See, you know, some, some. Images are critical. So it is important to send all the information to our customers. Our customers are developers, teams, teams. So it is important to send the change to all the past users. So this for me, this is, and I understand that you're not using STI because you had all this tooling in place. Do you have any plans to incorporate STI eventually or use it in your developer worlds? That's a really good question. We are, we were waiting STI, but this solution is just for corporate Docker image. Okay. So for applications, we, we have another CIDC solution. Okay. We have a very interesting A-L-N solution. And in order to deploy applications on OpenSheet, we use a corporate A-L-N solutions. Okay. That A-L-N solution uses Jenkins. So we were, as I told you, we were waiting the STI, but we decided to use our corporate A-L-N solution for application deployment and not to get them open image a bit instead of STI. Yes, I told before about the Serenity project and the A-L-N project, which is one of the lines of Serenity. And our developers want to have a binary deployment like the OpenSheet version too, or like how fun we have. The better solution for us to do it is this. Okay. It makes sense. I'm not saying you should. I'm just, I'm just curious if the end developers would use it. Because that's, because that's as an evangelist and a community manager. When I do my demos, I don't have access to all of the things that you have. So STI works quite nicely for me as a developer. One other thing that I was thinking about in terms of, so far, I mean, you said you were using RHEL as a base image. Have you had to do a RHEL base image update across the enterprise yet? And how has that gone? Is it? I'm sorry. Could you repeat your question? So if you're going to update the underlying operating system, so you're using a base image, have you had to do a major upgrade on the operating system level across all of your images yet? Yes. When we deploy OpenSheet, version 3.0, we were using RHEL version 7.0. After that, after 7.0, we received a notification that there were a new base RHEL available and we decided to reveal all the images using RHEL 7.2. Yes. Currently we are running RHEL 7.2. And so you used your CICD workflow to do the update of all the images? Yes. And it worked fine. Good. Yes. Really interesting solution, simple and interesting solution. And it is important to mention that it is not a CICD solution, it's not an ALM corporate solution. It is just a solution for OpenSheet for our past environment. We have another ALM solution for application deployment and build. Yes. It's for our token image. Maybe the next step is to include in this continuous integration cycle our configuration script or fancy ball or puppet, but that will be an interesting one. Yes. Another day. Another day. So let me look again. I'm looking to see if someone who is on the call has a question or wants to, there's a couple in Spanish. So I'm not going to try and read those out. My Spanish is not very good. Well, I think that is the question. I have one more. I have lots of questions. Because this is the image building process is really different across different enterprises and such. But I did have an interesting conversation. I'm here in Texas at Ozcon. And there's a lot of open source project folks here, including the people from WordPress since you used your example. I'm wondering where you got your, if you pull down the source code for WordPress and built your image from your own pull from WordPress.org. Or if you used, if you started with one of the base images in Docker Hub for WordPress. Well, good question. In the case of WordPress, we use the official WordPress that is available in Docker Hub. Well, we made some customization. We used the base image that is available in the Docker Hub, to get some interesting customization. In order to, for instance, we have a big file synchronization software in order to synchronize web content when we have several WordPress instances. We also have some corporate clients and also some hardening techniques. Cool. I think almost everyone who does WordPress has some customization for it on the deployment. So that's sort of what I expected for a response. But I also, I'm curious if people will start to have different versions of WordPress so they'll need to build from the source code from WordPress.org. And you sort of answered my question. I think that the Docker Hub one is good enough as a starting point for building your custom image. Yes, yes. The problem that we are seeing right now in the Docker Hub is not the latest version of WordPress. So probably that's the reason why someone is preferred to build the image from source code. Yeah, and that's what we found. I'm curious to see, and WordPress, I'm not trying to pick on WordPress, but I'm curious to see how those images in Docker Hubs get maintained and when they are upgraded, who are the people who are maintaining and supporting them, whether they're Docker people or WordPress people. And that's kind of the conversation I've been having this week with the WordPress people and trying to figure out who's going to build the latest image and make it official and maintain it on Docker Hub or wherever WordPress wants to host their own registry. So that's, I think you've done a great job because nobody's asking any questions at least as far as I can tell in my bad Spanish translations. So I want to thank you for coming today and doing this presentation. I will pop it up on blog post and on the YouTube channel and into the mailing list for the image building. Going to host another SIG in two weeks' time and I'm wondering if people have suggestions for topics. I have a couple of speakers lined up, one from the Atomic Registry Program to do a deep dive into the registry and ask back to that. And I think I might use that as the next topic, but if people have other suggestions, I'm very open to that. If there's something that Protobahn wants us to drill into and image building, let me know. Sure. Many thanks and I am for this, for this time. So we will just introduce the Powered Planes solution with the people, okay? Many thanks. Perfect, thank you everybody. Have a wonderful weekend or no, it's not quite the weekend yet, but evening. Take care.