 Good. All right. Hello. Welcome. This is advanced tool features as used by OpenDev. My name is Clark Boylan. I'm an employee of the Open Infra Foundation, but I'm also a Zool maintainer and an OpenDev admin. Before we dive in, there's a quick little disclaimer, which is that Zool has a lot of features. I'm only going to be able to talk about a few of them today. So don't think this is exhaustive. If I had more time, I could potentially get into more detail into the topics I'm going to be covering or other features. But it's short, so we're going to focus on what I think is interesting and relevant to some of the stuff that we're doing in OpenDev today. And hopefully, interesting and relevant to you if you deploy a Zool and have a Zool and want to do more with it. So some background on what Zool and OpenDev are so that we can understand what we're doing with these features. If you're not familiar with Zool already, it's an open source CI-CD system. One of the key things that it does is it gates changes with speculative future states. So it's testing the future before anything merges. You're confident that your code is going to work and function when you do merge it. And another kind of key detail with Zool is that it wants to integrate a bunch of different code repositories out there on the internet or within your company or within open source projects. It grew out of OpenStack, which has a lot of repositories that need to be integrated and tested together, and so it does that really well. OpenDev also grew out of OpenStack, and it is a software development collaboration tooling system. We host Git repositories, code review for those repositories, continuous integration for those source codes in that development. And importantly to the stock today is that we eat our own dog food. We use OpenDev to deploy OpenDev. So the Garret code review system is deployed using Garret code review. We review the changes that modify Garret. And similarly with Zool, we use Zool to CI the changes that modify Zool and that deploy Zool. And I think that makes us good Zool users because we're familiar with how to use it. We're not traditional software developers in that sense. But we're systems operators that are using DevOps or whatever you want to call it to deploy into production. All right, so let's start talking about these features. Zool can integrate source code from many locations. This is, I think, an underappreciated feature of Zool. We in our Zool today talk to our OpenDev Garret. We talk to GitHub to get changes. We talk to Garret's Garret. We talk to ROG Git repos on the Internet. And our Zool is pulling in changes and events from all of those sources and able to take action on them. And whether that's running a CI check or building container images and eventually deploying to production, Zool is able to, regardless of where these events are happening, kind of combine them all into a way that makes sense for our deployment in our systems and take appropriate action for them. Zool is also able to express rich dependencies between changes. And it uses this using a depends on construct that you add into commit messages. So within a commit message, when you push to Garret or GitHub or GitLab or whatever system that you're tied into with Zool, you're able to express that change A depends on change B. And this is important because while Git has a directed acyclic graph for its internal dependencies within a code repository, that graph only lives within the code repository. And so that means you can't really express using Git alone that one change over here and this code repository depends on another change in this code repository over there. And so this is a real world example I've got that Jeremy helped author with me. And what it is is we had a bug in Garret. And I went and fixed the bug in Garret. Jeremy wrote a change to consume that bug fix by updating our Garret configuration. And even though these commits live in completely different code review systems operated by completely different people, we were able to combine them in our Zool installation using this depends on construct. And I don't know how readable that is, but it's one change indicating it depends on the other through its commit message. And I think we'll talk a little bit more about this scenario towards the end of this quick talk. It also comes with built in container image or artifact management. There's kind of batteries included roles that act like a library to manage your container images and we use them ourselves. And there's kind of two aspects of this. There's the management of the artifact or the image within a build set. And the build set is all of the jobs that are operating for a single event. And then there's using those artifacts across build sets. So within a build set, one of the things we do is we pause a job. It runs before everything else and that job will run a container registry. Then we can have subsequent jobs, build container images, upload them into the registry and then other jobs pull from that registry. And they can use the container image and test that that container image is functional and that it does what we want it to do. It doesn't break anything or there's new features is there and as expected. And then if this is happening in a merge context, those container images from that registry can be uploaded to a more permanent location like say Docker Hub and then you might pull from production or pull from there into your production environments. Like the speculative git states that I briefly mentioned earlier, Zool can manage this sort of thing with its artifacts as well. And it does that by recording in an earlier build set, you'll record an artifact and Zool makes note of that in this database. And then subsequent build sets can query for those artifacts and incorporate them into their jobs. And what this allows us to do is successful image builds in previous build sets can be fetched in as base layers in subsequent builds. So we have a base Python image that we use to build some of our container images that you can be built upon in subsequent things. Like if you're updating a tool like our Garrett bot is built on this. It can depend on that container layer and it's all built together. And when you publish, you're actually publishing those layers together as well. There's not like an extra rebuild step. This topic alone is probably worth hours and hours. So I've dropped some quick links in there. If you want to learn more, feel free to pull that up or reach out to us and we're happy to talk about it. But I think this has been a really powerful part of our move to containers and deployment systems. Another kind of underrated feature is that Zool has native support for multi-node jobs, hopefully that's readable. But this is an example of our system config run Zool job, which is how we test that we're able to both deploy a Zool installation and that that Zool installation is functional after fact. To do this, we deploy a bastion house called bridge, a Zool keeper node, a Zool merger, a Zool executor and a Zool scheduler. And what that does is it very closely mimics what our production environment looks like. In production, we've got multiple Zool keeper nodes, multiple mergers, multiple executors, multiple schedulers. So this is kind of a distilled form. But all of the key components are there when we go to deploy in CI to ensure that both the deployment and the Zool software itself are functional. One aspect of this that came up yesterday in our discussion at the Meet the Projects is whether or not you could mix sources of these nodes from different providers. And unfortunately you can't do that today, but we've had some early discussions on how that might be possible. So right now these are all VMs. We can't say have a VM and a Kubernetes pod mixed together. But I think one day you might be able to do that. All right, so what does all of that allow us to do? When we combine all that, we like to call this testing like production. An example might help. So going back to that depends on example. We had one change depend on another. And those two changes were part of a bigger set of changes where Jeremy and I were kind of trying to replace the old Git web browser for Garrett with GitE, which is something that we've been running for a while. We just never tied Garrett and GitE together. So there was this initial change that was pushed up. And all this did was remove the old stale Git web functionality in our Docker image because it wasn't needed anymore. It actually wasn't used at all. Garrett had bundled this in by then. On top of that, there was a change push to normalize our Garrett config. And the reason that that was done was to make it easier to debug if anything went wrong and subsequent changes. You don't have the mixture of all this config editing involved in the actual setting of what you're interested in. And then finally, there's two changes that we looked at earlier where we had a commit that updates our Garrett config that depends on a Garrett change to fix the issue that prevented that config from working in the first place. When each of these changes was pushed, it was tested before it merged. It was then tested using multiple nodes, using a Bastion host, and the actual Garrett, well, actual in this case, it is a test Garrett server. And then our CD deployment system ran against that test Bastion and Garrett server, ensured that the deployment of the Garrett worked, and then we can ensure that Garrett is functional on top of that. So we're testing both the deployment aspects and that the service itself functions. So the first two changes go in, and what we're looking for there is that we actually want things to know up. So we're looking, we don't want change in that system. And that is often just as important as ensuring that the change happens as expected. And then finally, we expected change. We want to get web links to be replaced with Getty links. And in the early development of this, we found that it didn't work as expected because there was a bug in Garrett. I went and fixed the bug in Garrett. And then still, before anything had merged, we're able to tie those two changes together and ensure that the bug fix upstream actually solves our problem in our deployments. And in fact, there was an issue in the bug fix, and so we had to go through this twice, and that got through because we didn't go through this iterative process of ensuring that the bug fix actually fixed the issue in our system. So that'll be a lesson, I guess, that when you've got the testing, you should use it. Let's see here. So one of the key things I think that comes out of this is that it encourages us to collaborate with our upstreams and fix bugs as close to the actual problem as possible so that everyone else is benefiting from this. I don't know that anyone else out there is interested in combining Garrett and Getty in the way that we are, but if someone is, they can now do that. The bug is fixed. We could have fixed it locally by running a local patch and that would have kind of maybe gone through a little quicker because we wouldn't have gone through upstream code review and all of that. But I think at the end of the day, it's beneficial that we did it in this way. And it actually encourages us to test more because we've got a system that can test the deployment and the actual functional aspects of our services. We're doing it a lot more. In the past, this was really difficult when we were using other tools. And by removing that difficulty, anytime we're adding a new service or we're modifying service that doesn't have testing yet, we make a good push to actually add that testing and ensure it's working. And I think since then, we no longer need to run staging servers. I don't know that we might have one or two, but yeah, for the most part, we don't run staging servers anymore. Everything is tested in CI. And if we really are worried about something, we'll hold CI nodes and interact with them directly, but there's no longer dedicated staging servers. So this kind of cuts down on our footprint and gives us a lot more confidence, which is the next thing. It really does give you confidence that your changes are functional. They're going to do what you want. And it gives you confidence in your deployment system. I think if you've ever done kind of operations half the time, the problems are not in the service itself, but you're fighting issues in the deployment system. And we're kind of catching both of those problems early before they end up in production where they might cause problems. If we had more time, there's definitely a lot more detail that I could get into and there's a lot more features that are very interesting in Zool. A big one is secrets, Zool will encrypt secrets and manage them so that you can do interesting things like upload to Docker Hub with credentials and things like that. There's semaphores. This was, I think, mentioned earlier today in a keynote that allows you to manage external resources that Zool is interacting with as well as other things. We use semaphores to avoid multiple deployments in production trampling over each other so we can lock out the production deployment and ensure that each one completes sequentially. There's queue windows. If you need to manage load on your test environments, you don't want a bunch of changes pushed up by developers to trample your back end resources. You can window things and ensure that that doesn't happen. There's a ton of node pool drivers to integrate with a bunch of different resource back ends and there's a lot more. I think I have about a minute for any questions. If anyone has them, there's a mic up here, but otherwise that was the speed run of talking about some of Zool's advanced features. Well, thank you and I guess enjoy lunch.