 Hello everyone. Thanks for coming out here. I'm Dharmit Shah and I work with CentOS Container Pipeline Team. Today's talk, this talk is about a container pipeline which can be used by both developers as well as the enterprise people. This is actually coming from the feedback that we got in the previous year. They've come 2018, I was here to talk about the same project and we got pretty good feedback. It was really helpful, critical feedback, and based on that feedback, we went back home and re-architected the entire pipeline service, such that it can now be used by developers, which is already hosted on the CentOS Infra, and it can also be used by enterprise or some people who are maybe concerned that, they don't want to push the container images to a public registry. Maybe they want to push the container images only to an internal registry. So yeah, because I had given a talk about the same topic the previous year, I just want to do a quick show off hands if anyone was here in the talk or they took a look at this talk from DevCon 2018. Anyone? Okay. That's great. Anyone who has used the service so far or this is the first time? Okay. Thank you. Anyone who has heard of it before the talk? No. Great. This is not related to my previous year's talk, but anyone over here who maintains more than one container image or as I heard in the talk from Honza that are you a container maintainer, container image maintainer who maintains about five images, 10 images, anything? Okay. Then this talk is, I'm quite sure it's going to interest you. All right. So I have divided this talk into three different parts. One is the features I'm going to talk about. Second is the architecture change that we did, how OpenShift came to the rescue, and finally the main purpose of the talk that why exactly I'm here to talk about the same service second year in a row. So the core features of the pipeline CentOS container pipeline, which is hosted on the public interfaces registry.CentOS.org. You can take a look at all the container images on that URL. The core features of our service include pre-build, which means that you can pre-build an artifact or a binary before you build the container image. Next is the Dockerfile linting. So you use a Dockerfile, so far we use Dockerfile and the dev cons in this year and previous year, we have been trying to find different names, but we want to avoid using the word Docker, I know that. But yeah, Dockerfile, that's the canonical way of making container images right now, and we do the Dockerfile linting using the Dockerfile lint project from Project Atomic. Next, we build the image which is obviously the main meat or the main bread and butter of the service. That's what it's supposed to do. It's supposed to build the container images, and we have some surprise over there. We scan the container image. When I say we scan the container image, I'm not sure how many of you have taken a look but obviously most of you have used Dockerhub or Qua.io or something like that. And if you look at certain pipeline services, what they do is they do provide scanning feature, but it's not free of cost. It's available at a certain price tag and it's only CVE scanning that they provide. They don't provide any of the scanning. But as a part of the CentOS Container Pipeline Service, what we do is we scan your container image for possible RPM updates or possible updates for NPM and GEM package managers. We also scan your container image for RPM data verification, that is RPM minus VA, but we give you the output in a very human-vailable form. And finally, we also scan the container image for possible capabilities that a container is going to come up with. So there is a run label in the Docker file. If you have specified the Docker run command that is supposed to be used for the container image, we check that out and we tell you that, okay, these are the certain set of capabilities that the resulting container is going to come up with. And I'm pretty sure, I'm like 100% sure that no other container pipeline service out there provides these features. Next, we deliver the container image to the registry. At the moment, registry is registry.centos.org. And finally, we notify the users about, sorry, we notify the user whose email is registered for that particular container image. We let them know that your image has been built, whether the image build was successful or failure, what is the URL, and what was exact git commit that caused the build. Next, coming to one of the most important features, which is parent-child relationship. So yeah, of all those people who are maintaining container images, you guys, the container maintainers. There might be a scenario wherein you have 10 container images, and all of them are probably dependent on the same parent image. So you got sentos latest, and then you got container images like HTTP, Nginx, or Node.js, anything which is dependent on the same parent image. This is a very common scenario that we have seen and we see that in our pipeline service at the moment as well. So you got 10 container images, parent image gets updated. So you are maintaining 10 images, all of them based on the same image, and parent image gets updated. So how do you exactly rebuild, how do you update the 10 child images manually by doing Docker build, or maybe a shell script, something like that? What if there's not 10 images, but you are an ops guy, or you are a maintainer who's maintaining a number of images, let's say hundreds of container images, like we have an SCL org. They have around 20 images or something like that, which are building on sentos container pipeline right now. What do you do in that case? Rebuilding all the container images manually is a really tedious and boring task. Obviously no one wants to do that, but what sentos container pipeline does is it automatically rebuilds all this container images, which are dependent on a given container image. So previous year, we had this scenario of Meltdown and Spectre where the container images got updated, the base container image sentos latest got updated, and a number of child images were also updated. What we ended up doing was rebuilding around 500 container images all by ourselves automatically. No developer had to contact us to request a container image rebuild. Everything happened automatically. So a quick summary of the features that our service provides is that we rebuild every container image whenever there is a git push to the repository that's corresponding to that container image. We rebuild the container images upon parent image rebuild as well. And as a result, if you are a container maintainer, then you can be sure that container images coming from our registry are going to be updated pretty much all the time. Weekly scanning of images. So I mentioned about the various scanners like RPM update, NPM gem, the container capabilities, and RPM verification. We do that on weekly basis. Every Saturday night, we have a job which runs and which scans all the container images and it sends out an email to every user. And finally the email notifications. So yeah, email notifications are for every build as well as for every weekly scans. Quickly going through the workflow of the container pipeline service, there's only one time when a user has to interact with the service or when they have to deal with us. It is when you get onboarded with the project. So when you are saying that, okay, this is my open source project and we are building container images on top of CentOS and we want to build our container image on your registry. You open up a PR on github.com slash CentOS slash container index. I'll be sharing the link at the end of the preso. But yeah, that's the only time. You open up a PR, we take a look at it and once we merge it, this is what happens. We first check if a pre-build is required. Pre-build, remember I mentioned that if you want some binaries and artifacts to be included in your container image. So pre-build is the step that takes care of it. And if the pre-build is required, then okay, we go ahead and run the pre-build script which is available in the Git repo or we do the link build scan deliver and we push the image to the registry.centos.org. Finally, we send the email. At the bottom right of the diagram, we see that we do weekly scans. So yeah, that's the same thing that I mentioned earlier. We do that on weekly basis. And this is an example YAML file. This is the kind of YAML file that you are going to write and open up a PR with container index. It's pretty simple because everybody over here works on Kubernetes or Ansible or something like that. So YAML is not really new for anyone. We require the app ID and job ID, which is so based on the app ID, job ID and desired tag combination, what happens is the image getting created out of this YAML file will be registry.centos.org slash luster slash storage sig minus heketi colon latest. The notify email over here is heketi.devils at luster.org and we have the pre-built script as well as the pre-built context. We also have the build context, which means that if you have Docker file in a certain directory and if you want to do Docker build from a different directory, then that's what you do. I mean, that is the very last dot of the Docker build command, right? So that's the build context. And finally, the depends on field, which is going to determine that, which is the parent image of this container image. So every time registry.centos.org slash centos slash centos colon latest, every time that image gets updated, the luster storage sig heketi image is going to get updated as well. All right, so I said I was here in previous DevCon and I'm here again. What changed in a year? To be honest, when I went back to India from DevCon 2018, I went back with really good feedback and one thing was clear from very first day that I'm going to come back to DevCon 2019 and I'm going to talk about things we changed because there were a lot of things from architecture point of view that needed to be changed for the service. And to say it in a very short, maybe in one line, it was that the service was not really stable. We guys were doing firefighting most of the time. I'll come to that in the next slide. But yeah, in DevCon 2018, we had the same service, we had more features, just one more feature. We had emails being sent out to the users that contained logs from all the different stages like pre-build, docker file lending, building the container image, scanning. So all the logs were written into the same email and being sent out to the user that resulted in a pretty long email. We used to do the lending, building, scanning, delivering and automatic rebuilds used to happen upon every Git push, upon every parent image rebuild and upon RPM updates. So till previous year, yeah, the service used to do container image rebuild when any of the RPM in your container image was getting updated. But that is something we don't, we didn't have the bandwidth at the moment when we changed the architecture of the service. So it's not there. We are going to be working on that in near future. In DevCon 2019, what we have is the same service with one last feature, which is lack of RPM update. But we are now sending email with very minimal information. We just send you the link to your repository. And on that link, you can take a look at the logs, you can take a look at logs of all the stages about lending, scanning, building, everything over there. And yeah, rest of the things remain as it is. We still do the same stuff. Okay, so my CFP talks about OpenShift came to the rescue. It says that OpenShift was one of the game changers for the pipeline service. How exactly was it the game changer? Previous year when I was here, the service we had was completely heterogeneous. In the sense, we had workers for every task, like pre-building, like linting, like building the container image, scanning it. For every single thing, we had a Python worker return. And maybe one worker was running as system disservice. Another worker was running as a Docker container, rather a Linux container spun up by Docker run command. Build was running inside OpenShift cluster. And pre-built, that was the craziest thing that was happening on ci.centers.org, which is an altogether different infrastructure. So yeah, things were really messed up. And as a result of this, what used to happen is that deployment cycles were really, really long. Upgrade cycles were also really, really long. As a developer, if I used to make any change, I had to run the Ansible Playbooks once again. And the Ansible Playbook used to take around 30 minutes to deploy the service, even if the service was deployed. So in short, we messed up big time as far as deploying the service is concerned. We had some negative feedback from the ops in the sense that a few teams tried deploying our service and they came back with the feedback that your service is absolutely unmanageable. And we guys were doing firefighting most of the times in the sense all our standard updates are about, okay, today BuildWorker was misbehaving and Dharmit was doing this to make sure that it's getting fixed. The next day we had some other worker misbehaving and the developer was saying that, okay, I was working on fixing this. So in essence, we guys were not getting time to focus on new features or be able to attract more users or go out and tell people that, okay, this is a container pipeline service that does this, this, and this and it does it in a stable way. We had no guts at all to tell anyone that we did things in a stable fashion. That was 2018. Welcome to 2019. And our service is now OpenShift Native. I'll elaborate on that point in the next slide. Our development and deployment cycles are really fast in the sense when a developer makes any change to the code base, all they need to do is or all we need to do is we just do a manual Docker build and we push the container image to our internal development registry and we are done. So instead of 25 to 30 minutes, we are doing that task in about five minutes, tops. Stability, that was the most important reason why we wanted to move away from a heterogeneous architecture to an OpenShift-based architecture and OpenShift did help us achieve stability in the sense we deployed the service in the late September, 2018 and we have not faced any major downtime except once when, I don't know, it was a Friday evening and I was supposed to be out having beer or something but instead of that, I decided that, okay, I want to do a major OpenShift 3.9 to 3.11 upgrade in production and yeah, Friday is not good for production upgrades. So I screwed up big time but that was my mistake. That was not a mistake. That was not a stability issue at all. We ended up having better throughput in the sense the time we used to take to build the container images or do the weekly scans, it has gone down dramatically. Earlier, we used to take a lot of time and we used to use Beanstalk D, okay. I didn't mention the Beanstalk D over here. It's kind of a message queue and we used to use Beanstalk D for communication between different workers. So sometimes what used to happen is that job is there on the Beanstalk D but the worker is not picking it up. This kind of issue is, I mean, it's 2019. We shouldn't be facing this kind of issues anymore, right? So yeah, that doesn't happen anymore. Stability is great, better throughput and weekends are fun in the sense, I can do what things I'm supposed to do on a weekend instead of focusing on okay and not stressing on a Saturday that some worker will go down and my Sunday will be spent troubleshooting what went wrong, okay. So yeah, that is a major benefit that we got from OpenShift and as I said, I'm going to expand on OpenShift native part. So what we do, what we did as a part of moving the entire service to OpenShift is that all the workers are now running as stages in Jenkins Pipeline. They are no longer working as separate workers as a system decervice or as a Linux container or as anything else. Everything is working inside Jenkins Pipeline. How many of you have maybe used OpenShift and Jenkins Pipeline together? Okay, great. So you definitely have some opinion either it's good or it's bad. I'm sure you can't stay in between. You can't say that it's good, it's bad. It's definitely a strong opinion. Adya, that's what we did. We made every worker a stage in the Jenkins Pipeline and what it does is for every build request that comes in whether it's a weekly scan or it's a container image rebuild, any build that comes in for that we spin up a new part for every build request and it happens dynamically. So we don't have to have a separate Jenkins master and different Jenkins slaves running in our environment. We just have Jenkins service running inside OpenShift and dynamic slave parts come up and they just go away once the task is finished. We don't have to use Beanstalk D. That's a major relief and OpenShift with Jenkins orchestrate everything. So no Beanstalk D and just OpenShift and Jenkins they orchestrate every single thing. Switching to Builda, I don't see Dan Walsh over here but I'm sure he would be one of the happiest guys if he had seen the last point. So yeah, this is something I have tested for about 250 images and we were very successfully able to build all the container images using Builda in our service. So it's just about me going back to India from the conference and then focusing on one and only one task which is moving away from doing Docker builds to Builda bird. So yeah, that's what we are going to be doing next. Although we use OpenShift and although it has been really, really helpful it has come up with a certain issues of its own and I don't think you can run away from this. So remember I told about the weekend I had upgrading from OpenShift 3.9 to OpenShift 3.11. One of the major issues that we faced is let me tell you, I have time, I'm going fast but I still have time. So we upgraded from OpenShift 3.9 to OpenShift 3.11 because 3.11 has this cluster monitoring operator which is Prometheus Grafana, everything built in and the moment you spin it up it's supposed to have Prometheus Grafana, everything properly there is going to have the OpenShift routes all that stuff. And we thought that what we will do is we will have OpenShift's built-in rules that it thinks are important, the alerting and monitoring rules and besides that we will add certain rules on top of it. Now that is where I went really wrong. I should have probably done some more RTFM or something but what happened is that the cluster monitoring operator it's read only. It doesn't let you add any rules of your own. So yeah that is where we kind of messed up and we have to kind of live with it at the moment. OpenShift and Jenkins, they don't sync with each other 100% of the time that's our experience but unfortunately we haven't been able to reproduce this cleanly. But quite definitely the OpenShift Jenkins sync plugin it's not working 100% of the time. And Privilege mode, so once again this is coming from Dan Walsh, he doesn't really like people using Privilege mode or earlier he definitely used to hit people who are doing a Selenix permissive but yeah this is the next thing that he doesn't really like. And yeah, so we still need to use Privilege mode to be able to build container images. This is something we would like to move away from. All right, coming to the final and probably the most important section of my talk, at least from my point of view, it is that our pipeline service needs more users. So if you are a container maintainer, building images on top of CentOS or maybe Fedora even, please just take a look at it because this service is now stable and it does pretty much all the things that I am promising over here and we are doing it in a really nice way as far as I can tell you. But yeah, the service is running on CentOS infrastructure for about three years now and there are various upstream projects that rely upon it, but we need more. I mean, one of the matrix, if not, because it's a debatable topic, but one of the most important matrix that determines the success of an open source project is probably that how many people are using it, how many people are benefiting from it. But as far as this service is concerned, it has got two benefits from a 10,000 feet point of view. It is that if you are building container images on top of CentOS, then the service is already hosted on CentOS Infra and you can use it as is. But if you are someone who wants to build their containers in a private infrastructure, you can just go through the documentation. We have made sure that everything is covered. You can bring up an OpenShift environment very easily within your infrastructure. We have even mentioned the steps to do that and you can deploy the service in a private infrastructure. Maybe you can contribute back to the project in that way. And this is something that I don't really talk about a lot or we don't really talk a lot about it, but the service is distro-agnostic in the sense that it probably takes less time than I have been speaking with you guys. So I have been speaking since 20 minutes, but it would take less than 20 minutes to make sure that you change a few commands and the same service can give you useful information for container images that are being built for maybe Ubuntu or REL or Fedora, anything. Anything where OpenShift works, right? So wherever OpenShift works, you can just go ahead and deploy the service. You can build container images for that. Distribution do anything for that. I'll end the talk with a few quick links. So if you want to take a look at the containers that we are building right now, it's registry.centers.org. If you want to take a look at, I mean, if you want to get onboarded with the service, you want to build your container images through our service. It's github.com.centerscontainerindex. If you want to take a look at our code base, which is much more beautiful than how it used to be earlier, it's github.centerscontainerpipelineservice. And if you have any queries, just shoot us an email on centers.table at centers.org or on the free node. Questions, feedback. Feedback is the thing you have at the, under dev.com.sked.com, right? So yeah, I would love if you can share some feedback. If you have any questions for me, then I would be happy to answer. Right, that's it for me, yeah. You see, as I said, I haven't really tested building real images, but I don't see any reason whatsoever why you should not be able to do it, because everything is running on OpenShift. We deploy OpenShift using OpenShift Ansible. And I think it's really, it should really be possible. It's just that I have not tested it, but I'm 100% sure it will work. Ah, okay, so the base image, say you, sorry. But you don't want to, so the question is that you don't want to automatically update? Yeah, can you repeat the question, maybe? Me, fine. Sorry, I'm sorry. So that's what I was doing, actually, that. I think the question, as far as I have understood, is that you want to be able to disable the feature of automatic updates for the base image. Okay, but as far as Git push is concerned, that's not going to happen. So if you push something to the Git repository, image is getting updated. But if you don't want your image to get updated when a parent image gets updated, then probably the best way right now is that don't specify a parent image for it. At the moment, you just leave that YAML entry empty. The last one, okay, depends on none. That's, I mean, that's how Centhouse base image itself works. It depends on none, it's the parent image. I hope that answers the question. Okay, so the question is that, can you synchronize the images with Docker hub? No, that's not possible. I mean, we guys push to registry.centers.org and synchronizing with Docker hub is really not something we have considered. You can always modify the, I mean, change the locations where you have specified Docker.io slash my awesome image. Instead of that, just make, move from Docker.io to registry.centers.org. Yeah, I get it. Any more questions? Yes, okay, I'll try. I'm sorry. Okay, sorry. I'll take the questions outside if anyone has one. Thank you.