 Hello, we're just about ready to start Kubernetes-styled CICD with Harshad, and I had to close all my other tabs because they were very noisy. So, Harshad, I'm going to start it now. If anybody has any problems with the audio, just let me know in chat. Also, I'll be pinning the link to the video. If you do have any issues, you can go and watch it direct from that link in the chat. All right, I'm going to hit start and mute myself. Hello, everyone. Welcome to DevCop US 2020. Hope everyone is doing well and staying safe. I'm very excited to be presenting here. I'll be presenting Kubernetes-styled CICD system. The following would be the agenda for today's talk. We'll be going through the whole design and implementation of the CICD system with some knowledge about decked-on stools. And then I'll be telling you more about my challenges, the challenges which I face and some insights on the whole development process. Before starting, let me introduce myself. I'm Harshad Reddy-Nuller. I'm a software engineer working in AI Center of Excellence, Red Hat currently stationed in Boston. I'm a part of AI DevSecOps team. We work on a project called DOT. It's an AI stack recommendation system. It recommends Python back into modules based upon performance and security. I'm currently focusing on the AICOECI. That's a Kubernetes-style CICD, which is this whole dot-based spawn. I'll be talking more about it, how we designed and implemented it, and also showing a demo about that. So without any further ado, let's jump right into it. So the goal for this talk was to share my findings and learnings when I started to build this CICD tool with decked-on pipelines and triggers. So before going into the design and implementation, let me ask you why do we care about the CICD? The CICD is a very integral part of the DevOps team, or the DevOps architecture, or the DevOps lifecycle, right? When a developer or a contributor contributes to a source code, they make some new changes. They send these new changes for review. While the reviewer's job is to review the new changes for the source code, he or she doesn't have to ensure on the syntax, styling, or linting, or any of the pitest or checks on the testing or the end-to-end testing of this new changes. This all things can be taken care of by the continuous integrations system. The reviewer can focus on reviewing the actual core code of the new changes, and once he or she feels like they are okay to be merged, the CI system can more than code in. Once this code gets into the source code, it needs to be delivered to the production environment or the development environment. So there the delivery part comes in. So today we'll be talking about the continuous integration delivery system that the continuous deployment, we are not considering that right now. We are just focusing on the integration and delivery aspect of it. As we wanted to build the system, we wanted something which can run on premise of OpenShift. As all our core workloads runs on OpenShift, so we wanted to concentrate on those aspects. While researching about it, we got to know that there's no one CI which fits all the application. The team or the maintainers have to decide on their CI CD tool based on their applications. So it's true that no one size fits all. As the applications are moving towards microservices from monolithical architecture. Similarly, the deployment infrastructure is also moving from virtual machines to containers. As the applications are moving so fast towards the Kubernetes environment, we would like to have a CI CD tool which is also running its workload on a container as an environment. So thus the whole focus on building a CI CD tool through with a Kubernetes style. So while trying to develop something like this, we came across Tecton. Tecton is an open framework which allows the user or the developer to build CI CD systems with its tool. It was created and contributed by Red Hat and Google contributors. It's now a part of CI CD foundation. Before learning about Tecton, we would have to learn about Tecton terminologies. We can't just go right into the Tecton designing and implementing a Tecton pipeline without knowing it terminologies, right? So let me take you through the journey of learning like what these terminologies are and how they are used while developing the pipeline for the CI CD tool. So first is a step. Step is the smallest working piece of code in the pipeline. Step would be references the container image which would execute a specific script. A task would be running this series of steps and a task would be something which will initiate this whole task. For example, take building an image. So building an image would be a task. So what would be the steps involved in it? The first step would be to gather all the information to build that image. Second task would be to use a Docker or a container orchestration tool such as build an apartment to build them. The third task would be to publish them into an image stream registry. So this would be the following steps in your task. And if we execute this task onto the cloud native environment, that becomes a task one. Moving further, there is pipeline and pipeline resources. Pipeline resources is a place where you will define all the basic information such as images or get resources. So get resources could be anything such as if you are describing your whole Git repository, where to clone the repository from, or to gather the pull request information, this would be the place where you would initialize that. Pull request pipeline resources also gather this information from the output of the step of a task. And pipeline would be pipeline runs a series of tasks. And pipeline run is the initiator which will initiate the pipeline execution. So let me explain you well more better with this example diagram. So if you see on your left side with the shaded blocks, those are the pipeline resources such as GitHub repository and the image. Here the image is a UPI image. Git repository could be my repository. So the following other blocks are describing the task like a code quality check, accept environment, build image of a publish or deploy them. The whole, this whole figure describes a pipeline. And once we run this pipeline with some inputs, those that becomes a pipeline run. If you see that smaller chunk of code where it's written task run. So if when the GitHub repository resources are provided to the task and the task is executed, that becomes a task run. So a pipeline constitutes several task runs. That's how it can be defined. As we learn about the terminologies of the pipeline, let's learn some more terminologies of the tecton trigger. A tecton trigger takes the event payload from our webhook and provides that towards a resource template. So what are the core components used here? There's an event listener which will be continuously using to the HTTP request from the webhook from any of the GitHub or GitLab. Or it can also take direct webhook payloads. So the JSON payload from this event listener is passed to the trigger binding. Trigger binding maps this content of the JSON payload to the required parameters. These required parameters are provided to a trigger template and trigger template passes those parameters to the pipeline run. So this describes the whole concepts of like basic concepts of tecton triggers and tecton pipelines. We use this component so we can build the whole CI CD system. Let's jump into like how what design constraints or what design requirements we had and then I'll be explaining you with all these components how we built a CI CD too. So as most of our projects are AI ML oriented, we wanted to focus on that. Most of us all the projects are Python based. So we wanted to have some statistics checks and learning checks and styling checks. Static checks would be like by test running some tests by test end to end test. Learning checks would be running some by doc styling or some of the flake it flake it checks. Once we've done with the checks on the pull request, we also wanted to release this applications. So a Python application can be either a module or it could be a running publication. A Python module has to be released to PyPI and then you can use it as a package. If it's an application, you can continue as them and then apply it and it won't bunch of an arm. So we wanted a CI system CI CD system which can enable us to do some code checks while a pull request on a pull request and also enable us to do some releases on the source code. So we focused on two aspects here. One was on pull request and the other was the tag release. So as a contributor makes a pull, creates a pull request, the CI services will trigger a pipeline pull request pipeline. A pull request pipeline will run some status checks such as py test, learning checks and then also check for the build checks. If the whole application can be built into a container image or not. Once this process is done, the checks will return the validation. If the validations are all true, the reviewer can review the source code and if the reviewer accepts the new changes, then the new changes can be merged into the source code. Once all the pull requests are merged into the source code and the maintainer decides to release a tag. The tag release pipeline will be triggered from the CI. The tag release pipeline runs an image build task or based upon the application. If its application is in Portugal, then it runs a pipeline release task. Once this task has run, the task based upon the application will release the image to the pipeline index. Or else it creates the container image and passes it to image registry and then updates that tag into the customized manifest so that it can be deployed into the communities orchestration open shirt. So this is what we focused on and this is what was the requirement and how we designed it. So let me show you the architecture diagram and explain you through that and then I'll explain you the implementation from the source code and then I'll show you some live demo. So this is architecture which we follow. We have so on the left side you have source code from the GitHub repository and also from some of the GitLab repositories. So both of them are triggers, some webhooks on pull request or the merge request. So this webhooks triggers to the tecton pipeline running in our environment. It starts a tecton pipeline for the pull request run. So the pull request pipeline runs some certain tasks such as lend checks, unit checks and syntax checks. So once this task completes, it sends out the results to the status check on GitHub. So these status tips could be based upon the results of the task. If the task has completed successfully, they will send out a true status to the GitHub. And once these status are corrected, as you know, reviewer will review the code and merge and click on merge and merge with the new changes will go into the source code. Another aspect was, like I said, was the direct release. On each direct release, the GitHub will send out a webhook to the tecton pipeline. The tecton pipeline will start on the direct release pipeline. The direct release pipeline has a task called bill image. The image will be built and then there's a delivery task which will deliver that image to the image registry. And as of now we're using the Red Hat Quake to take that. Let me explain you, let's dig into the implementation demo. So I'll show you some of the source code, explain you through each component, how it's been designed and what is the purpose of it and how they will be, how they are a part of the whole big pipeline of the CI CT tool chain. And then I'll show you the live demo, like how it works in actual. So let's go through it. So the whole pipeline is the application is called ASO ACI. It's been developed based on the tecton pipelines and the open shift pipelines. And it is available on GitHub. There's also a GitHub application so you can download it into your own repository and use it based on that. So before going into the development of the pipeline, first we have to install the tecton pipeline at tecton progress. There is a setup script in the GitHub repository of the ASO ACI, or else you can use the actual tecton pipeline repositories to install based upon the instructions. You can read through them and install the tecton pipeline and a tecton trigger. Apart from these two, we also installed tecton dashboard. Tecton dashboard enables you to, it gives you a UI where you can visualize all this pipeline runs. So I have already set it up for today's demo. And let me show you that in the open shift environment have deployed tecton dashboard, tecton pipelines, tecton triggers. So all this is running. There's a VEPOC listener and there's a tecton VEPOC listener as well. So once you have done this, we have to, as I said, first we have to design an event listener. So let me show you some source code and let's talk through that. So first is the event listener. As you can see, this is the event listener manifest. So in the event listener manifest, we are concentrating on two components as of now. So one is the event called pull request and another is called the tag release. So in GitHub, whenever pull request is triggered, there is an event type added to it. It's called pull request to that payload. So as a pull request is generated into GitHub, GitHub sends a VEPOC to the event listener. Event listener listens to that, takes the JSON payload and passes that to GitHub trigger bindings. So trigger bindings can be defined as this. And once the trigger bindings have been passed, the trigger bindings takes out the information from the JSON payload. So the JSON payload takes that information. So from the JSON payload, the trigger binding takes the information such as event actions, PR, pull request number, the repository to which pull request was raised, repository full name, pull request URL, and repo URL. All this information is then passed to the Tecton trigger template. Tecton trigger template is a resource template which basically runs a Tecton pipeline run. So it describes all the parameters which are passed through the Tecton trigger binding and then pass those trigger bindings to resource template. So those resource template is a pipeline run in this case. So the pipeline run takes all this environment and it passes this information to a pipeline. Here the pipeline is described as Tecton CR pipeline. That's the name we have given here. And it has when we are provided all the parameters. With all these parameters, along with all these parameters, we also pass the resources. These are the pipeline resources, which we can describe in an additional manifest file or we can also describe them in line. So here some of the resources are being described in line with the resource spec. The resource spec defines that you're describing in line. You're saying that this type of resource is a good type. You're using the repository URL. So based upon this, it will throw in the repository, whichever repository URL is passed to this. So this URL is a parameter. This parameter comes from the JSON payload as we know. So as one protocol has been triggered, the whole value travels from the event listener, JSON payload, trigger binding, trigger template, and do this resource. Apart from this, you can, as I said, you can describe the resources into a manifest file. Here you see there's a S2I dot image. You could be at image. We have taken a reference. So these are being described in the resource file. So let's go through that. So here we have described the S2I dot image. So the image which we're using is the S2I dot base image, which is developed by the dot project team. And then there's a UBI image. So we are passing those images as also resources, which can be used in tecton task. And then the pull request itself. So this pull request type pull request in this, if you provide the URL. It captures the pull request information into a JSON, which is available for each task. One thing to remember here is that each request from a, from a public repository can be pulled directly, but from a private repository to be pulled from a private repository requires some secrets. So authentication secrets. So you can provide that authentication token SSH keys to the secret to directly to the resource. As we, as we have gone through the trigger template and pipeline run. Now we know that once this trigger template is triggered, this pipeline would be run. So let's go through the pipeline. The pipeline describes about in this pack, we describe the parameter and the resources which we have gathered from the pipeline run. And then we describe some task references. So all these tasks references corresponds to each desk responses corresponds to a task one here. So let's look through one task one. So there's one task run, which is pre-commit check. In this task run, we are describing that this is the task which we need to run. While you're doing that, we want to do some pre-checks before the task has run as, as, as in ticked on what happens is if you run a task and if the task fails, the whole pipeline fails. There could be some, some steps in which the pipeline, the task can result into failure, right? So before doing that, it would be better if we do some, some condition checks. For that, we're, Tecton provides something called Tecton conditions. So Tecton conditions, partner conditions are also a sub-task, but their failure doesn't result into pipeline failure. Their failure just results into a check failure. So we are just checking if the actual pull request has an action open. If it's closed, we don't want to run these checks. We are also checking some other components. And once it's done, if these conditions are true, then, then only the pipeline will run. Also, all pipeline will also assert this task will be run. The task will consider some of the parameters and the resources from the pipeline and run after basically states that until unless this task have been run, please do not run the pre-commit check. So as we learned about the task, let's go through the task, the pre-commit check task. So in the task, we have described the parameters which we gathered from the pipeline, resources which we get from the pipeline. Now we are describing the steps. The first step is taking the information from the pull request and gathering all the information or the changes which were created in the pull request. So we got that changes from the pull request. We checked out a branch. And now we are running some tests on it. We already have a base image for this task. Pre-commit checks are basically series of checks like mypy checks, Python, Python, PyTest checks, Flake8 checks, Pepe8 checks. So all these are already containerized into an image and we would use this image in the step to run a simple script which is pre-commit check run on all files. So we are running this run pre-commit check on the all files in that pull request. So the changes are being checked. And once the checks have been run, we update the pull request with information as a status run. We are saying if it was a successful check or not. Once it's done, the pipeline completes. And let's look through this pipeline in a live execution. So let's go to the GitHub. I have already set up a GitHub repository for today's demonstration. I have already opened a pull request with some changes to the repository. I made a single change that I had a print statement into my app.py stating that, oh, I'm very excited to present in Defconn 2020. So when the pull request started running, there was the CICD system started running. With the CICD system running, it did two checks. One is build check and a pre-commit check. As we see, the pre-commit check has failed because there was some issues with the PyDoc, but the build check has finished. So let's see how this results into a tecton pipeline, right? So in the tecton dashboard, you can go to the pipeline run. In the section of the ASUE pipeline run, we see the bunch of tests running. We see the pypre-commit check running. As we can see, first we gather the pull request information. Okay, so let me explain to you first what happened. There was a webhook which triggered the event list. The event listener triggered the trigger binding. And the trigger binding passed really that trigger template. The trigger template started a pipeline run. The pipeline run is called ASUE pipeline run. This has started the pipeline, which had series of task runs. The task runs have started to execute. Now we are investigating on one task. The one task was pre-commit check one. There was one step to gather the information from the pull request. We gathered some information from the pull request. We run some tests while the task completed. We send those information through a webhook again to GitHub. As you can see, the step was failed. So we returned a failure. And you can see it's a failure here. And now similarly, we have a tag release pipeline. So the tag release pipeline on each creation takes some of the information, like version of the tag, what type of tag is it, and name of the URL. Similarly, it's passing to a tag template template. It runs its own pipeline. It's called tag release pipeline. Let's go to the tag release pipeline. In the tag release pipeline, it's basically checking what configurations do you have? Should I release into PyPI or should I build to make it as a container image? So let's go and see what happens when you are building into a container image. You're taking, gathering all this information. The first is to check if it can be pushed to our image stream or image registry. Yes, we can push it to image registry. So we gather the information of a task in the next step. We take S2I build. We are building with the information. As you can see, it's an added a script which checks if it needs to be built with Docker file as to why from a source to image or from a Docker or from a container. Based upon the application, if you see here from the source code, it is, it is considered containing a Docker file. So this tecton pipeline of task release should run the Docker file. And as you can see, it's gathered the Docker file information and it has been based on the Docker file. So if you consider the changes in the Docker file, it's basically take Python 3.6 CentOS 7. It's the upward favor file and there's a command. So it has taken the Python 3.6 CentOS file, taken the app and then, and continue as I did image. And now this image is pushed to a query repository. We have pushed the query repository from the second step. So once we have built, we have pushed it. So while pushing it, we also provide the configuration for it to push it to a query repository. As this step is executed, the image is pushed to the query repository as you can see here. So this is how the whole tecton pipeline executes. Let us go back to the demo. Let us go back to the presentation. I hope you understood how the pipeline works. There were some few challenges while designing all this implementation. First, as I explained, while interacting, the CI CD needs to know how to interact with private repositories. I showed you how you can set up a secret and attach that to the SSH connection. Another challenge was that you need to pass one result on another task. There's a tecton result resource, which you can read up on the tecton pipeline review. And you can use that to store results and pass those as a parameter. I use that to store the results into a file and then use this file to use as a parameter to pass them to the next task. Apart from this, there are some of the steps which you need to follow while maintaining the connection between running pipelines and a GitHub status. So while you're doing that, when you're executing a step, you can always pass those information to GitHub as a status check. As I showed you, it shows you if a pipeline task has failed or not. So these were the few common challenges which we faced and we figured out a way around through them and we worked to overcome them. Next, let me show, tell you some of the insights while developing it. We realized that all these tecton pipelines can be directly plugged into other CI to select proof. So you don't need to run it as an individual CI. If you want, you can add it into your pre-existing CI pipeline. If you have something like Pro, which is also a Kubernetes-based CI tool. We also figured out that you can also add this, which also works. The CI-CD system works very well along with additional CI tools such as Zooler, Arco, which can be used for gating. And then you can use the CI-CD systems to run some checks. They work in very well with the synergy because all of them uses Jamal arguments and manifest files. So they work very well together. One of the biggest thing which we looked into was while developing, we were using OpenShift 3.11 and we realized that tecton pipeline version 10.2 was the last version which supported that version. So please be careful while you're developing to check which tecton pipeline matches which Kubernetes version and if you can implement that in your OpenShift run-on. While deploying your tecton pipelines into the cluster, you need to admit privileges. So these are some things which you need to keep in mind while deploying and developing such tools. Hope all this information helps you to build the CI-CD tool. And if you need more information, you can add some of the references here. The first two references are of the application which we developed. I have added some of the read-bies. Maybe I haven't gone through each and every detail. I wanted to just cover a lot of topics in this short duration. So a few of the topics, a few of the things you can check through from the read-me. If you have more questions, you can open an issue with that. If you want to contribute, please feel free to contribute if you would like others to contribute to this project. Apart from this, I've also added some of the references from the upstream community tecton and also the OpenShift pipeline. OpenShift pipeline is the downstream of the tecton pipeline which is currently being contributed by Red Hat contributors. So please feel free to have a look at it as well. Thank you so much for attending my talk. I hope you learned a little bit about how to develop a CI-CD with a phase-final Kubernetes style. If you have more questions, please reach, you can reach me. You can reach out to me on the following links. And for more information about Doth, I have also provided some links about that. You can reach out to them. Thank you so much. Have a really good day in the DevConf while looking through all the DevConf presentations. Thank you. Stay safe. Thank you so much for that. That was a great present. Oh, I am mute. Sorry, hold on. Just a second. Next video playing in YouTube on my end. So I don't see any questions. I am really interested in this topic. I work on OpenShift. Do you also work on OpenShift? Not exactly on OpenShift, but most of our applications run on OpenShift. So we have to relate to it. Oh, I see. Yeah. The OpenShift CI-CD, we use Prow, Kubernetes Prow. And it is insane and complicated and brings together over a hundred different repositories that all need to stay in sync and not break each other. You know, the complete OpenShift release payload is made up of over a hundred different repositories, I believe, at this point. And it's complicated and absolutely necessary to keep things rolling. Yeah. I think it's a dependable and working CI-CD, yeah. Yes, definitely. I actually looked into Prow and that was a, like that was the first step when we were searching for all the CI-CD tools. That's the one which we came across because we work very close with OpenShift even we know like they provide us the details. But then we went on to looking more and then there was, there was a lot going on. We just thought, baby, why not? Let's try it out again. And then we can just edit in Prow because Prow has this feature where you can include it into ProveJobs. So you're sending out the GitHub or GitLab web works, but it's going to reach in Prove, but Prove is triggering the Tecton pipelines. So it's not running its own jobs, but it's running pipelines. Oh, I see. So you can do this. So all this is plug and play. So you don't need to change anything. You just need to remove the Tecton trigger. Just put it into Prove. It works exactly the same. Oh, wow. That's really interesting. So we do have the breakout room. Yes. If you have any other questions. Otherwise, thank you so much. And you can watch Harshad's link. I pinned it above and it will be updated in the schedule when I get to it. Maybe a few minutes. Thank you so much. Yep. Have a great night. Thank you everyone. Enjoy the rest of the talks.