 I was talking on you. So here's our next presentation. Open source tools and OpenShift CI. And I'll be presenting. And in this case, I will be doing live presentation because the original slide deck, which I recorded, was a couple of things which I fixed later. So to me, it's better to share the latest and greatest. So in this case, let me just share my screen. And actually, let me first share my slides in case you want to follow. So slide deck I'm going to use and let me share my screen. All right. So first of all, about a little bit about myself, I work as a senior software engineer for Office of CQA in Red Hat. And we sometimes ask to help other teams in some cases, which require some help, external help. So in my case, it was OpenShift bare metal installer component. And there was originally a pretty difficult situation with CI. Most test cases and test speeds were failing over time. And that's how I actually got into this interesting project and got familiar with OpenShift CI in terms. So the disclaimer is that most of this presentation will be provided from a single component perspective, which is bare metal installer. And I'm not going to probably give you a complete picture of OpenShift CI universe because one of previous presenter shared it, so it's huge. Let's move on. So today, I want to bring you my moving parts of OpenShift CI brawl and CI operator. And I'm going to look deeper into bare metal IPI and run test. So in this case, we can go through components of this test, look at the step registry, look at the hardware platform we use to run these tests. And we also look at another open source tool we actually using in this CI flow, which is Ansible. And obviously, some Q&A in the end of this presentation. So we'll start from OpenShift CI overview. And the main component of OpenShift CI is a brawl, which is Kubernetes-based CI-CD system native to Kubernetes running, Kubernetes or in our case, on OpenShift cluster. And the strongest part that it's designed to scale with high number of jobs, with very detailed policy enforcements, and has a seamless GitHub integration. Like we will see it a bit later. It's very flexible, accepting different commands which you can issue from your PR comments field. It does govern automatic PR emerging. So PR is sponsored to external events, to execute jobs, which can be some Chrome event or can be some command from GitHub and remain types of jobs. You can see inside brawl is pre-submit, which you see that gets executed every time you have any code changes inside PR before merge. Post-submit after the code is merged and periodic which runs on repositories on a scheduled basis. If you look at inside brawl architecture in connection to our OpenShift CI, it's some event from GitHub is triggering a web hook and there's a hook component inside brawl which actually responds to such a request. We also interact with end user sharing. The final results, there is a deck component inside brawl and you can see the brawl dashboard as a result. Blank is responsible to starting multi-engine execution for CI components. Tide is something which you will use for merging the code for final merge. If it's successful, Sinkr does use some stale jobs and make sure everything executes without any garbage left behind. And on this background, you can see end-to-end test, our usual design for OpenShift, which gets executed every time in order to have successful merge with the code. The next slide is something which I borrowed from the external source as a reference below. And this one is something which gives you a very live picture of what you can see, what components inside brawl cluster. And by the way, brawl is a leading edge of the ship. That's why you see some marine picture here with a ship with some other components like Sinkr. Cleans up their jobs like I mentioned, launching a container for each job. So brawl is just one of the components. It's a bigger picture, so to speak. And CI operator is something which we interact most inside OpenShift CI system if we need to create any tests, for example. So CI operator is an OpenShift native tool which automates the process of building and testing OpenShift components. And almost every OpenShift CI job is supposed to run an instance of CI operator. It's basically when you look inside under hood, every component git repository has a CI operator config file. And it translates automatically into a brawl job to execute a certain set of tests. CI operator both builds images required and run tests with these images. CI operator config file, this is something which is required to create any tests in OpenShift CI. It defines many things. Which images will be built around the tests? Which tests to be executed? And which images will be promoted in case of success, successful execution for tests? As I mentioned before, brawl jobs are automatically generated from this config file. And it's done usually by running the following commands, make CI operator config, and make jobs prior to checking in your files in the repository. And the repository itself, I provided link here. It's under OpenShift release CI operator. That's where we store content files. I provided a few references here without waiting till the end of presentation. So in this case, we can just see the CI cluster used to run OpenShift CI brawl dashboard, which was actually mentioned before in one of the slides. I'll just quickly show how it looks on the bigger scale. So here we have a number of fresh jobs without any filtering, but if there's all kinds of filtering, you can apply in an upper part of this dashboard. We'll look into more details into the tree and CI operator configs in the demo part of the presentation. And by the way, another useful link here is OpenShift release status. Also presented one of the pro components. And last but not least, CI search engine. It's not technically part of it brawl, but it does allow you to search on OpenShift results. So if there is a certain failure, you can use Regux search partner here and find it within, say, a certain number of days and a certain number of jobs that you want to renew. It's very useful for debugging purposes. All right, now off to our bare metal IPI end-to-end test. And first of all, what is it about? This is one of our end-to-end tests executed by many at this point. That's probably 10, 15, maybe more OpenShift components and repositories. It uses a standard dev screens deployment, which is essentially OpenShift installation on virtual machines, which can be deployed on single physical server. It does not require any additional components. But you thought it does free masters to workers deployment. And it's a bare metal IPI test runs subset of OpenShift conformance tests. Currently, it's, if I remember right, around 170 test cases. Give or take. Originally, it started from very low amount, maybe like five test cases. But, obviously, it's not all of them, because the whole amount will be over 2,000. That isn't why it's not all of them, because there are certain limitations running OpenShift tests in this virtual environment. So we choose mostly a core and critical test for testing cluster functionality. Here is how our test workflow looks. We have this M10 bare metal IPI test in OpenShift CI and OpenShift K&A test image stream for DevSkates. And we have that packet cloud, which is basically infrastructure as a service, a cloud where you can borrow the server with hardware configuration, whatever you want. And it will be running operating system whatever you choose. In our case, it's CentOS 8. So for the server, first thing we acquire, at least provisioned a packet server. Using, by the way, AnsibleScape, like we see our few steps ahead. We'll test your latest DevSkates image from DevSkates OpenShift repository and copy resources to the server. We run DevSkates installation, which is actually just pure make command with maybe some tweaks in the configuration by end-to-end metal IPI test. And then we run our conformist subset of test cases against the cluster. And in the end, we copy, browse logs and test results back to the system to OpenShift cluster running CI. And we do provision like a server using another AnsibleScape. And another thing to consider in this design, we run our VIST tests and end-to-end metal IPI is written as a multi-stage test, which was introduced literally, I would say about six months ago by our DPTP team. And the main idea is it has a modular test design, which allows user to create new tests combining smaller steps. So in this case, step is the smallest executable unit. And then we'll have chains. And then we'll have workflows as the biggest element of this multi-stage test cases. So all steps are stored in so-called the step registry. And here's the URL. And the OpenShift release CI, the other step registry. And the good thing is it now includes a pretty extensive help. The help is provided here on this URL and it goes up into any case. It does include pretty detailed explanation of every way you can use step registry and OpenShift CI in general. With all possible examples of CI operator config files, environment variables, et cetera, et cetera. So for multi-stage steps, this slide just explains how we pass our information between steps and feed some information from OpenShift CI into this step. In our case, it's all governed by environment variables. So cluster profile directory, it's something where we store so secrets for a bare metal idea test. And we use a shared directory to share our information between steps. If we have more than one step in a room in an environment, usually the final step writes its information or intermediate step may be also writing information in hard drive directory to store our logs and later share it with the route and store it, share it with the customers of the space where we started this test. Enter and metal IPA, steps and workflow. As I mentioned, we design a specific workflow for this. And based on the test step definition, we have three steps, steps designed to set up the environment. I have the test step, which effectively executes test steps. And post, it's something for cleanup and something which always executed, even if any of the previous stages fail. So in this case, we get a clean system and we'll not have any garbage left behind. And here, I just wanted to spend maybe a minute looking at packet.net, this is our, so this is the interface we use. In our case, we have a dedicated project to run bare metal tests. I'll just make you sure that I'm very proud of them. You can see there are a few servers already running with certain configuration. Most of them in the same data center. And we use certain set of tags to actually better understand what our job actually is in the server. So in this case, we're able to debug it using the test. So now, a few things about Ansible, where it is in this picture. So I will not probably spend too much time telling you what Ansible is, because it's pretty widely known and some of the previous presentation went integrative about Ansible Tower. So in our case, Ansible is another open source tool, which is convenient to fit open-shoot CI environment. And namely, we use Ansible in order to preserve and prepare packet.net server for the test and clean up and reduce. So in this case, we'll proceed to demo part of this presentation. And I have some set up command to reserve this system. And here, actually, how some of our steps in step register look like. As you can see, it's under this operator step register. And in this case, it's set up for packet. The path kind of explains what it does. And naming convention is to follow this path. And essentially, every step of the show script. But in our case, it does have embedded Ansible, which is using packet server model, which is supported by packet.net. And it gets updated whenever they change any of the APIs. So in this case, we are fully isolated from any possible breakage in this case. And it goes reserving the server and reporting the failure. If something goes wrong, we have a notification sent straight to the Slack channel for bare metal install. And similar, we have bare metal DS packet teardown commands. Again, single structure show script with Ansible inside, which does teardown for server once we're done with CI execution with bare metal DS testful. And again, in case of any headers, we'll get notification sent in the Slack channel. So let's see what else I wanted to highlight here. Oh, and another interesting item in this presentation, the PR test flow, how it looks like from, say, test developer or CI developer perspective, how you interact with these tools. So in this case, originally, I have this upper PR, but right now it was already merged. So I'll just leave the life one in this presentation. And in this case, we have it right here. And I created this sample PR, just a showcase of day-to-day work you may need to perform inside of the components here. So in this case, we may need to tweak a number of test cases. And let me just keep off this push. So we'll see some action GitHub and from browser. So in this case, I made some change. Then one of my teammates made another change in the list, and I have to rebase and resolve some components for this PR. But for our purpose, it's pretty much irrelevant, which I can just see what's going on. And all I did was just pushing the updated version of my code. And in this case, we'll see the automated CI test execution for some new tests running here. And hopefully, we'll see some results reiterated in a few seconds. Because depending on the test case, it may take from, say, several seconds still to, in this case, of an end-to-end metal idea. It will probably take an hour and a half. So we will obviously not wait till the end of this execution. But we'll see how it starts. As you can see, there's some successful tests already marked as such. But this is basically checks by format of the files, say CI-formers, et cetera. So the interesting part will be, actually, some of the end-to-end metal idea tests, which is still in pending state. Let's see how it looks like. So we don't see yet any logs here. But as we go for execution, it will be here. You may be interested to look in artifacts directory in the end. Or for the purpose of this presentation, I can just look at the logs. And instead of this pending, look at some previous execution, which is one hour 40 minutes. So if we wait all the way to the end, we'll see it's something like this. There's always one flaggy test, but otherwise most of the test cases, which are supposed to run here past, from my can tell it's a green brand, so this will be successful. And artifacts directory in this case have all kinds of logs. And in our case, if we look at the test case logs, it's under end-to-end metal IPA. And every step of the test flow produces its own directory in this case. Like a set up, like a tear down. By the way, if we look at the answer, how it looks like from logs perspective, it's not probably anything interesting because in the interest of securing our environment, we don't really share too much information from unskilled in the logs. But for the test steps, it's more detailed, the actual test execution. So you can see all the test cases started fast. And if you scroll down, it's probably more details. So this is very much what I wanted to share for today. And in the end of this presentation, you can see more fundamental set of references. Again, the link to CI steps documentation, which we saw recently, environmental IPA steps in the repository. And very detailed open shift CI doc is written by our team. The NID deployment for very much IPA. It's our code survival guide for CI. And that's how one of the sub nodes tells us. And at this point, we are done with my presentation. And I'm open to any questions. Please feel free, type your questions in the chat window. Thanks, Narin. I was actually, I did some dry run of this presentation before in our tech talk. And in this case, it's kind of trying to already know what probably will be asked in some cases. All right. So in this case, you have my presentation slide deck, which I probably will still keep up to date in case of any new changes in OpenShift CI. It's very dynamic system to a degree. And some new stuff is literally getting introduced, if not daily, but monthly basis. So I highly recommend again the link, which I shared on the step register because now OpenShift CI team is trying to keep it up to date. And if there is anything new, it's usually it's getting documented here. So thank you very much for your attention.