 Hi, my name is James Blair, and I'm here to talk to you about Zool. Zool is a project gating system that we originally developed for the OpenStack project, but it's now being used by an increasing number of other projects. A project gating system is like a CI or CD system, but with a few main differences. It's designed from the ground up to support projects with the microservices architecture. And it's designed to help teams develop code faster by testing each patch as it's being merged, along with any other in-flight patches to related projects. Let's take a look at the system in action. This is the Zool that's running in the OpenDev Collaboratory that's serving the OpenStack project. You can see it's testing a large number of patches to different projects simultaneously. Zool is a highly scalable system. This particular Zool serves several independent tenants and is running on a control plane with about 30 virtual machines. At peak, it runs jobs on around 1,000 test nodes. Today, I'm going to be showing you how to set up a Zool system that is simple enough to run in a few containers on your laptop. But it's running the same container images that we use in production for OpenDev. So it's easy to start with a small system like this and grow it as needed. The instructions and files for the tutorial are available on the Zool website. So you can reference them at any point in the future. Just scroll down and click the Get Started button. Now, let's get started. The first thing we need to do is to clone the Zool Git repository. That's at opendev.org slash Zool slash Zool. That's going to contain all of the files that we need for the tutorial. They're in the doc source examples directory. In there, you can see a number of files including the docker compose file, which we're going to use now to bring up the system. So we can just run docker compose up, and it's going to start a number of containers. It's going to start a ZooKeeper system, which is what Zool uses to store its state information and share that between all of its different components. It'll start an input launcher, which is the component of Zool that interfaces to whatever is providing the test resources. That could be an OpenStack Cloud, AWS, or statically defined bare metal servers. It'll start the Zool scheduler, which decides what jobs to run. It'll start a web server and the Zool executor, which is the scalable component of Zool that actually executes the jobs. And finally, it'll start an instance of Garrett, which is the code review system that we're going to use for this tutorial. Don't worry if you use something else. What we learn here applies to GitHub and others as well. So this is our Garrett that Zool has created for us. It's running in a developer mode so that we can actually log in and create accounts without having to deal with passwords. So we're just going to go ahead and create a new account and set that up. So this is going to be an end-user account. I'm going to enter in my username and my full name here. Save that. And then I'm going to put in an email address since it's a little awkward to use Garrett without email addresses. It likes to associate changes with authors based on that. And finally, I'm going to add an SSH key because that's how we're going to upload changes to Garrett. So I'll just open up a terminal and copy in my SSH public key here. So with that out of the way, we've set up a new Garrett account. We can go ahead and reload the page and we should see the new settings take effect. I'm going to show you a few projects that the Bootstrap script has created for us in Garrett. You can see three of them here, test one, test two, and Zool config. Those are Git repositories that have already been created in our Garrett. I'll show them to you over on the Zool side as well. Here's our running Zool instance. You can see right now on the status page, it hasn't been configured yet. It has no pipelines defined. On the projects tab, you can see that test one, test two, and Zool config. Zool already knows about them as well. Zool knows about two different types of projects. We call them untrusted or config projects. An untrusted project is just the project under test. It's the code that you're actually developing, the thing that you want Zool to test. Test one and test two are untrusted projects. You can perform some amount of Zool configuration in an untrusted project. You can define jobs. You can say to Zool what jobs you want it to run for this project. But other parts of Zool that require elevated access such as configuring pipelines, that sort of thing, those are done in a config project. So we'll be modifying the Zool config project in a minute to do that. You can see a fourth project on here, the Zool jobs repository. That isn't hosted locally. That's on the opendev.org server. We're going to use that later to take advantage of some already defined roles and jobs that the community has developed that we can go ahead and use locally. So next we're going to go ahead and clone the Zool config repository from our local Garrett. And if we see the end of that, you can see that if we see the end of that, you can see that it is an empty repository right now. The bootstrap script just performed a minimal initialization. We're going to make a directory to hold our configuration files in. Zool can either read its config from a single file or a directory of files. In this case, we're going to be configuring a few different things. So we'll go ahead and make a directory called Zool.D to hold these different files. I'm just going to copy the first one from our examples documentation in Zool. I'm going to be copying the pipelines configuration file into place. And let's go ahead and take a look at that. Zool configuration files are, they consist of a list of YAML objects. In this case, you can see that we're defining pipelines. So the first item in the list is a pipeline that defines what we call the check pipeline. This is what happens when somebody creates a new change and uploads it for code review. You can see that it has a couple of different triggers. There's a patch set created trigger or you can leave a comment that says recheck. And any of those actions in Garrett will trigger this pipeline. If the jobs succeed, it'll leave a vote in Garrett and the verified approval column of plus one. And if it fails, then it'll leave a minus one vote. So this is how we configure the check pipeline. The next item in the list is the gate pipeline. That's similar to check except that instead of being triggered when somebody uploads a patch, it is triggered when a reviewer decides that the patch is ready to be merged. The way they do that is they leave a workflow vote on the change and Zool picks up on that and encues it into the gate pipeline. You can configure these pipelines however you want. This is just a recommendation and how we configure it inside of Garrett. Similarly to check, when the jobs succeed, it leaves a verified plus two vote in Garrett. If they fail, it leaves a verified minus two vote. And also, if they succeed, it tells Garrett to submit the change. Submit is Garrett speak for go ahead and merge the change into the give repository. So in this way, Zool acts as the gating system. It's in control of what change gets merged and when. And that's how we make sure that changes merge only once they pass their tests. All right, now we're going to copy in some more configuration from the examples. We're going to copy over an initial project config. The project configuration is how you tell Zool what jobs to run for what get repositories. You can see two configuration items in this file. One is a project configuration that is going to match every project in the system. The reason we do this is we go ahead and attach every project to the check and gate pipelines. This just makes it a little bit easier to add new projects later. It means they can bootstrap themselves into these pipelines on an initial change. And the second item is the Zool config project itself. So in this case, we're going to tell Zool to run the know up job in both the check and gate pipelines. The know up job is simply an internal job that succeeds immediately. It's really useful for bootstrapping a system when we don't actually have any jobs ready to run yet, but we can still go ahead and run changes through the gating system with this. Finally, we're going to copy one more file over and that's our initial job definitions. Zool job configuration has an inheritance hierarchy. So every job has an inherent apparent job all the way up to what we call a base job, which is the top of the hierarchy. It has no parent. This is how you can build up jobs into more and more complex jobs. So you can start with a simple job and then add new features onto that, increasing complexity to build up from there. Zool needs a base job in the system, so we need to go ahead and define that. And we do that just by creating a job entry where we set the parent to null. We don't do much else here yet other than say that this job is, by default, going to run on an Ubuntu Bionic node set. All right, with those configuration files in place, we can go ahead and add them to a git commit and then go ahead and push that git commit up for review in Garrett. Now, if we switch over to our Garrett server, we should be able to see that change has been uploaded and is ready for review. Now, since we haven't finished bootstrapping Zool yet, we can't actually use Zool to gate this change. We're going to need to force merge the change and bypass the gating system. Our user doesn't actually have permissions to do that, so we're going to have to sign out and then sign back in as the administrative user. We can skip the introduction here and then go find that change again. And now, as the administrative user, we can go ahead and leave some votes on this change. The code review vote and the workflow vote are normal. The verified vote we normally wouldn't have access to, but since we're the administrator, we can go ahead and set that. So now with those votes in place, we can submit it, which again tells Garrett to actually go ahead and merge the change. Now that it's been merged, if we switch back over to Zool, we can take a look at the status page and see that we now have a check and gate pipeline. Zool watches the event stream from Garrett and notices when changes are merged to its configuration repositories. So it can go ahead and reload that on its own. So now let us switch back to a terminal and we're going to go ahead and clone one of our untrusted repositories. We'll make a copy of test one and we'll go ahead and add some, an initial job to that repository. Now this time, in addition to configuring the job, we're actually going to have to tell it to do something. So the way you have jobs do things in Zool is by writing an Ansible playbook. So we'll write a simple playbook called testjob.yaml and we're going to run this play on all of the hosts. That means that it should just run on that Ubuntu Bionic node that we requested in our base job earlier. And then we'll just add a task here that's a debug task which basically just prints out this message hello world. And this is a pretty simple Ansible playbook. It should be pretty self-explanatory. You can do quite a lot with Ansible playbooks. If they're not your cup of tea, you can always just write a playbook that executes the shell script or something similar. Now we'll create the Zool configuration for this job. So we just create a new yaml object in a list called job. We'll name the job test job and then we tell Zool to run the playbooks slash testjob.yaml playbook. And that is it. That's how you define a job. Next, since we need to tell Zool when to run this job, so we'll go ahead and set up a project entry for this test one project. And we'll tell it to run the test job in both the check and gate pipelines. And now that should be it for configuring this job in this repository. So we can add the Zool config file, add the playbook, and create a new git commit with those and then push it up for review and garret. Now that it's up for review, we can switch back to garret and look for our changes. And there's the change that we just pushed up. If we switch back over to the Zool status page, you can see that it's now running our new job. And it succeeded. Once again, back in garret, if we reload the page, we should see some feedback from Zool on the change. You can see that it left a plus one verified vote and it left a pretty incomprehensible finger URL in the bottom. The reason that it did this is because we haven't actually configured Zool to save any logs yet. So what it normally would do there is leave a link to the logs, but since we haven't done that, the best it could do is come up with a finger URL. So since in order to correct this, we're going to need to update our base job and make it a little more sophisticated. So to do that, we're going to switch over to our terminal and modify the Zool config project. Again, we're going to tell Zool to actually do something this time, so we'll make a playbooks directory and edit a file called pre.yaml. Zool can execute a number of different types of playbooks. We call them pre-run, post-run, and run playbooks. A pre-run playbook happens before the start of a job, a post-run after, and a run playbook is the job itself. So in this case, what we're doing is we're creating a pre-playbook and we're adding the build SSH key and prepare workspace roles. These roles are defined in the Zool jobs repository, which I mentioned earlier. If you go to ZoolCI.org, you can find the documentation for the Zool jobs repo. It contains a number of jobs and roles that are generally useful. Next, we're going to make a post-ssh.yaml file. In this file, what we're going to do is tell Zool to run the remove build SSH key role. That's the inverse of the add build SSH key role that we added earlier. Finally, we're going to add one last post-run playbook called post-logs.yaml where we're going to tell Zool to actually save the log files. This time, instead of running on the remote nodes, we're going to run on localhost, which is the Zool executor itself. What we're going to do is we're going to invoke two roles. The first is the generate Zool manifest role, which just has Zool create some metadata about the logs that it's about to upload. Finally, we're going to run the upload logs role, which is going to copy the logs to our static log server. We're going to tell Zool the URL of that log server is running on port 8000 on localhost. This is another container that is started by the Docker compose file. Now, we're going to edit the job definition for our base job and tell it to run these playbooks. We're going to add the pre-run playbook, which is the pre.yaml that we created earlier. Then we're going to add the post-run playbooks. Now, there are two of these. Actually, we can switch this to a list syntax instead of a single. The reason why we made this as two playbooks is because Zool will run them independently, and if one fails, it'll keep going and save the logs. Finally, we need to tell Zool to add the Zool jobs repo to the checkout so that it can actually access those roles. For good measure, we'll just go ahead and add a default timeout value of 1800 seconds. This timeout will just be available to any jobs that we add from now on. They can override them if they want. We'll add the new playbooks and the updated configuration file to a git commit and push that git commit up to Garrett. If we switch back to Garrett, we can go and look for this change to the Zool config repo. It's there at the top of the screen. Now, since we've bootstrapped our system, we no longer need to be logged in as the administrator. We can sign out and sign back in as our regular user, and this time we can actually see the Zool gating system in action. See, it's run the noop job that we configured earlier, and now we can go ahead and leave our code review vote and our workflow vote to tell Zool to go ahead and enqueue this change into the gate pipeline. Now that we've done that, we should be able to reload the change and see that Zool has run the noop job again in the gate pipeline and merged the change since it succeeded. Now, if we go back to our new test job, what we want to do is run it again so we can add that recheck comment that we mentioned earlier and that will re-enqueue it into the check pipeline. So if we go back over to the status page, we should see it show up here shortly, and there it is, it's started running it, and if we click on the link, we can actually watch the console for the job scroll by in real time. So you can see the add build SSH key role is running now, and I see the hello world debug task there, and now it's uploading the logs. So at this point, it should have reported back to Garrett, so we can reload the change there and see that this time it left a link to our log server. If we click on that, we can see the logs that it uploaded. Now, if we switch over to Zool, it's worth noting that under the builds tab, you can access information about previously run builds. So if you click on the link here, you can see there's a summary of the tasks that are run. There's also the logs that we just saw. They're presented here as well, as well as the Ansible output. That's it for this tutorial. Thank you for watching.