 Hello everybody. I think it's about time to get started. I guess we've got a few people coming in. I can vamp for a minute while people come into a little song and dance, come up with other ways to work the word potato into talks. My name is James Blair. I'm one of the maintainers of Zool. I'm the current project lead, and I helped start the project 10 years ago. I think it's 10 years as of last week. And yeah, I guess maybe we could close the doors. That would be great. There's a very high energy summit going on outside. So as I was saying, my name is James Blair. I'm one of the Zool maintainers, and I started a company called Acme Gating, which is the world's first project gating company. It's focused on supporting people that want to use Zool in the enterprise. And I'm here to give sort of an overview of what Zool is and what project gating is. Zool was originally developed for the OpenStack project. The problem that we were presented with is that OpenStack is built out of a lot of different components. There are actually a lot of projects. They all have to work together, but they're all built by different teams. So, you know, these teams go off and do their own thing in their own project, and then all of these projects get combined together into a larger system. That might sound familiar to folks who work in the enterprise. And so it turns out that all of this work that we did to support Zool for OpenStack can be leveraged by others. And it actually ended up being a lot more generally applicable than we thought. We thought we were dealing with the unicorn at the time. We thought, like, nobody else builds software like this. But it turns out everybody does. Everybody has these problems. And so Zool is now being used by countless other projects and companies. As I said, I helped start and maintain the Zool project. And I now spend all of my time helping companies use Zool more effectively. So why is Zool interesting? A lot of things that we thought were really important when we were developing OpenStack have kind of made their way into Zool and really define its character. One of the first things is that it's git driven. So in OpenStack, we intentionally developed a very strong code review culture. And the idea is that anybody should be able to propose changes to any of the OpenStack projects. And then the maintainers of those projects would review the changes and approve them. And get them merged. One of the key aspects of this is that we tried to get humans out of the loop as much as possible and automate as many things as possible. And so a result of that is that a lot of our processes in OpenStack and then later on in Zool and in the open infrared community in general are git driven. Because the more that we can put into git and the more that we can put into git, the more things are available for anybody to go in and contribute directly. So Zool does not have a kind of web interface where an administrator goes and click, click, click and defines jobs and things like that. To the greatest degree possible, everything is defined in git repositories which can be contributed to and reviewed by people. By putting all of this information in git repositories, we've now given ourselves actually a mechanism to alter how control of a CI or CD system is performed. So again, going back to the idea of an administrator using a web interface to define jobs, actually that person has that ability and how they define how that ability is scoped is sort of defined by what's implemented in the web interface. Whereas by putting things in git, we have this ability to have both centralized and decentralized control of the system at the same time. And really it's kind of a sliding scale as we do that. So developers can put things in a central repository or in a sort of edge repository, a repo that's being used for development of software itself. Zool's configuration constructs can be put in any of these things. And the combination of how you tell Zool, like, read your configuration from this repo in this case, read it from this in another case, sort of lets you find the sweet spot for having both centralized or decentralized control. Zool supports cross project collaboration. Again, going back to the idea of Git repos. Actually, I'm going to go into more detail while I'll test a little bit later. It has this idea of speculative execution so that Zool, rather than testing things as written, Zool is going to test things as they will be when they're merged in a potential future state. And finally, all of this is leading up to the idea of project gating. So I'm going to go into these a little bit more detail. I feel like I've actually talked about Git quite a bit, but the key aspect here is that code and infrastructure changes can be committed together. So when you're working on a software project and you want to change how the software functions, you can change how it's tested and how it's deployed all in the same commits, all in the same series of commits, and Zool can handle that holistically. So it's really bringing the ideas of continuous integration and continuous deployment into the realm of development and doing all of those things together. Like I said, you have the choice of central or distributed localized control. A lot of that in Zool comes from the idea of Zool's tenants. Zool is a multi-tenant application so you can, as an enterprise, say you could define a tenant for one part of your organization, another tenant for a different part of your organization, and those groups can operate the system effectively completely separate from each other. So this lets you run a much more efficient system. You can run a single Zool for your entire enterprise and have as many tenants as you need to give local groups control there. And like I said, because things are in Git repos, you can share those repos between tenants whenever you do want them to collaborate. Zool has the idea of cross-repo dependencies, and this is something that really hasn't shown up in any system other than Zool in any significant way at this point. Zool supports a number of different Git backends. It supports, Garrett was the first one, and then we added support for GitHub and GitLab and Pagure. And what Zool does is it actually doesn't care where the code comes from or where the changes come from. It treats all of these systems completely equally, and you can write a change to a repository in Garrett, and you can tell Zool that that change depends on a change to a different repository in Garrett. And Zool will make sure those changes are tested together. That doesn't stop at the Garrett boundary, though. If you have some developers using Garrett and some developers using GitHub or some developers using GitLab, you can hook the same Zool system up to all of those and actually have a change to a Garrett repository that depends on a change in a GitLab repository. By expressing dependencies between these two, you can actually bridge systems which otherwise wouldn't be able to talk to each other in any reasonable manner. So you can have jobs that check out unmerged changes from projects in multiple code review systems, and they all get checked out into the same workspace in the same job and run and test it together. So the next key thing to know about Zool is its idea of speculative execution. And the way that I like to think about this is kind of the evolution of CI systems. So when the first CI systems came out, CI standing for continuous integration, the idea was that after you merged a change to your Java project, you would then integrate the project and see if it still worked. And you would do that after every change or maybe even just on a fixed schedule. And you would be notified of the breakage as soon as possible after it happened. So basically, what these systems were doing is testing the past. At some point, people realized, well, you can actually run these tests on changes before they're merged, and so we ended up with this idea of pre-merge checking. So somebody opens up a pull request on GitHub, pushes a change up to Garrett. A CI system checks that out and runs tests on it and says this either does or does not work as written. That's testing the present. Zool goes a step beyond that, and we call this project gating. And the idea is that at every point, you're trying to test the future. So it's not just the change as it was written by the developer. It's the change as it will eventually be merged into the Git repository. And there's a couple of nuances about that one. It means that if between the time of the developer writing the change and it's eventual merge, if the repository itself has moved on, because it turns out there's other developers working on the same thing at the same time and they get their changes merged, if the repository has moved on, you need to run the tests again, and Zool is going to do that. So essentially what Zool does is it prepares, right when a change is ready to merge, it prepares the repository as it is at that moment and then merges the change that it's, it speculatively merges the change that's under test into that. And if there are any dependencies, it's going to merge those changes in as well. And so what we're doing is we're testing the proposed future state. And if the tests pass, then we sort of atomically make that future state reality. So the guarantee here is not that the change worked when the developer wrote it. The guarantee is that when this change merged, it worked. And being able to do that, plus being able to combine it with other potentially unmerged changes is extremely powerful, and it actually changes the way developers work. So to sort of take this idea from another tack, I would define the word gating as saying that every change proposed to a repository is tested before it merges. That's the simplest way of looking at this. It's literally a gate. It either passes these tests and gets merged or it doesn't. You can extend that idea with co-gating where if there are changes to a set of repositories, again, think about the OpenStack use case, where all of this came from. OpenStack has Nova and Swift and Keystone and all these other projects. In this case, you're looking at changes to a set of repositories merged monotonically, such that each change is tested with the current state of all the other related repositories before it merges. So the advancement here is the idea that we're not just gating changes to a single repository. We're gating changes to all of these repositories together at the same time. And then finally, Zool goes overboard here and implements this idea of parallel co-gating so that to do this more efficiently, we actually put all of these changes into a virtual serialized queue and test every stage of that queue with all of the changes ahead of it under the assumption that they're all going to pass. And by doing that, we're able to merge things a little more quickly. So you can imagine that a naive implementation of project gating might just test a change and then merge it and then test another change and merge it. And if it takes you an hour to merge a change, you can't merge more than 24 changes to all of your projects in a day. And nobody I know wants to work that slowly. So by having Zool start all of these jobs in parallel, it's able to increase the throughput of the whole system. So here's sort of a graphic view of that. When Zool is setting up its virtual serialized queue, you might have a couple of changes that are already merged in the repository. So start with that. And then Zool is somebody approves the change and Zool starts testing that change. Somebody approves two more changes after that and now you've got this queue of three changes that are all lined up to be merged. Zool starts the test for all of those in parallel and if any one of them, for instance this second one from the bottom, starts failing its tests, then Zool will pull that out of the queue, finish running the test so that we give the results back to the developers as quickly as possible. But by pulling it out of the queue and resequencing the changes behind it, Zool is able to improve the throughput of the whole process and still potentially merge these changes if not at the same time, nearly at the same time. So the overall throughput of the system can be quite high even though we're being extremely rigorous about making sure that every change is tested. So as an example of how you can use this idea of speculative execution across different repositories, you could imagine a change to your infrastructure, the backend of a software system and you have to implement that first. Then once that's implemented, a front-end developer writes some functionality to use what is newly available in the backend. Sorry, I've actually got this backwards. You might need to write something in a library which then is used by something in the front-end which then infrastructure deploys. So by having, being able to create this dependency graph, you can actually have Zool test all of these changes together before any of them have been merged. This is actually something that Clark talked about a little bit in his talk earlier. It's extremely powerful because you can, in a complex system where changes are touching multiple components, you can actually see that all of them completely function as designed before you merge any of them into any repositories. So by doing this, you're doing the testing upfront. You're avoiding having to revert out changes because you thought you fixed a bug or you implemented something correctly and it didn't actually work right. You're essentially front-loading all of the testing. So this is going back to my previous illustration of speculative execution. If you apply this to multiple repositories, then again, say you've got some library changes, those have to merge first, then a front-end change and then your infrastructure changes can be all lined up behind that. So in this way, Zool is enforcing the sequence of events as well. That front-end change isn't going to merge until the library changes merge ahead of it. Likewise, the infrastructure change isn't going to merge at all in this case because it's failing, but you get the idea that Zool is helping you test all of these things together and make sure that they are merged in the correct sequence. Zool is, again, because of its birth in the OpenStack ecosystem, it definitely has a bias towards clouds. Generally, when you're using Zool, you're going to want to hook it up to a cloud provider to give it ephemeral test nodes. So you can hook it up to OpenStack, AWS, Google, Azure, or the IBM cloud. These are all basically interchangeable node providers that Zool can use when it's starting tests. So the idea is that it gets a virtual machine from one of these cloud providers, spins it up, starts running the test on it, and destroys the VM at the end. So this is a way of making sure that all of the builds are reproducible. They all start from a clean slate. So as a developer collaborating with other developers, you know that anybody starting on a project, they're going to get the same results starting from a clean slate. Zool also works with Kubernetes and OpenShift. So you can run jobs on pods or get Kubernetes namespaces to deploy systems in there. And finally, static nodes are also supported. So if you have some real hardware that you're testing against, either bare metal or some kind of specialized hardware rig that you've got hooked up to a bare metal machine, you can enroll these static nodes and node pool can manage them and Zool will use them in the same way that it uses cloud resources. One of the key things to know about this is that all of these things are interchangeable. So in a way, the failure domain for Zool isn't just at the cloud level. It's sort of at a multi-cloud level. So you can have redundant clouds providing resources to Zool. And if an entire public cloud has an outage, which does happen from time to time, that's fine. Zool will just use resources from a different cloud instead. Zool has native support for multi-node jobs. So if your job needs more than just a single node to run on, for instance, if you have a system where you need to set up a controller node and then you have multiple compute nodes or something like that that you're testing out, you can tell Zool to request multiple nodes at the same time and your job can use all of those resources together. So it's a very expressive way of creating jobs which is quite difficult to achieve in other systems, but it's very natural in Zool. And part of the way it's so natural is that Zool's job language is actually ansible. So when you get to the point of writing what a job is going to do, it's, you know, we thought about it and we're like, should we create yet another job definition, or sorry, job control language? But we realize there's already one out there. There's already one out there that lets you say, run this thing on node A and run this other thing on node B and that's called ansible. So when you write Zool jobs, you can write ansible playbooks where you could, as an example here, have roles that run just on the controller, have roles that just run on compute nodes, and that's the way that you can request these multiple different types of nodes from Zool and then orchestrate what happens on them in the job. Zool is also highly scalable. So itself is, it's a sort of microservices-based application, and this is a list of all the components of Zool. And at this point, as of early this year in Zool version 5, every single one of these components is both scalable and redundant. So you can run multiple versions, sorry, multiple copies of every single one of these components and by doing that, your Zool system can be fault tolerant. It can have zero downtime upgrades because you can do a rolling restart of the entire system and it can be scalable in that as you run more and more jobs, add more tenants, add more projects, you can scale up these components to meet your needs. So that's a quick overview of what Zool is and some of the key concepts behind it. If you'd like to learn more about Zool, go to ZoolCI.org. And there's information, there's a tutorial there to get your own Zool up and running. There's plenty of documentation and other resources there. There's information about how to get involved in the community. We have online chat via the matrix system, mailing lists, that sort of thing. And if you'd like to find out more about the services I offer, you can go to actnegating.com. And we do have some time for questions. So if anybody has a question, there's actually a microphone over there and we can all hear your question. Can you speak to the relationship between explicit dependencies that are potentially cross-project that you express with depends on and their relationship to the implicit dependencies that we get out of the DAG that we have in a Git repo and how that works across multiple repos? Yes. And did everybody here in the room hear that? Great. So I sort of used the word dependencies and glossed over the fact that there are actually a lot of different kind of dependencies. And it is worth being aware of all of them and how to use them in Zool. And then it's also worth being aware that once Zool recognizes a dependency, it behaves exactly the same way no matter where it came from. So this is sort of, you know, earlier when I was talking about how you can incorporate changes from different code review systems, the reason that works is because by the time they show up in Zool, Zool doesn't care where they came from anymore or what kind of dependency there are. So yes, there are a couple of different kinds. There's the dependencies that come from Git itself. So if you have multiple Git commits, a sequence of Git commits sort of implicitly depends, you know, the one on the end depends on the one ahead of it and so forth. In some code review systems, that's not a big deal because you're kind of like in GitHub, you make a branch and you put a bunch of commits on that branch and you open a pull request for that branch and the pull request is what Zool sees. And so Zool is going to, it's going to enqueue that pull request into its pipeline and that's the thing that's going to change and that's implicitly going to have all of those commits bundled up into it as a unit. In Garrett, on the other hand, if you make a bunch of commits and then upload them, each one of those commits shows up as a distinct change in Garrett and so Zool recognizes that and it treats each of those changes because in Garrett, you can individually merge each of those changes, right? Obviously in GitHub, if you merge a pull request, you've merged all of the commits. In Garrett, even if you have a bunch of commits, you can still merge them one at a time. So Zool implicitly treats those as dependencies between them and so if you have a series of commits ABC, Zool is going to enqueue ABC in that order and both assume and enforce that dependency series. Going back, I guess, stepping up a little, there's the idea of cross-project dependencies which obviously can't be expressed as Git commits nor can they be expressed as pull requests in GitHub because you can't have a branch that has more than one project in it. So to do this in Zool, what we do is we add a depends on footer to the commit message in Garrett or to the pull request message in GitHub and you can say that this change, pull request, whatever, depends on something else and what you do is you say depends on colon and then you give the URL of the something else. So that could be the URL of a change in Garrett, the URL of another pull request in GitHub. And again, this is where you start to do the cross-project or cross-source thing. It's almost always going to be a change in a different project but that different project can be in the same code review system or a different code review system. Finally, we actually just added one more kind of dependency to Zool which is the topic dependencies in Garrett. There's actually this behavior in Garrett that some Garrets use and some Garrets don't. We don't in OpenStack, which is why this wasn't a priority but other Garrett Inzool users do use this and that's the idea of pushing up changes to different projects within a single Garrett and having them under the same topic. That's a way that you indicate to Garrett that all of these changes should be merged simultaneously. So what we do with that in Zool is we implicitly treat them as cross-project circular dependencies which we're starting to get into like mind-bending things here but that means that every change in the dependency cycle so all of these changes that are under the same topic will be tested together and not merged in sequence but merged simultaneously. So if you have a dependency in project, if project A depends on a change in B and that same change in B depends on something in A. In other words, you want to simultaneously change two projects at the same time atomically. You can do that in Zool and still have them tested together. So that's a quick overview of the many types of dependencies that Zool handles. Any other questions? All right, I think we're actually exactly at time anyway. So thank you all for coming and... Yeah.