 Welcome back and thank you for joining us. Once again, it's open and for live, the open infrastructure foundations weekly, hour long interactive content series, where we share production case studies, open source demos, industry conversations, as well as the latest updates from the global open infrastructure community. This is our seventh episode and we're focusing today on open source CICD. We have some amazing content coming up over the next few weeks. So please join us every Thursday at 1400 UTC for more open info. I'm Jimmy MacArthur and I'll be your host for today. We're streaming live on YouTube, LinkedIn, Facebook, and Twitch and probably even Myspace. We'll be answering questions live throughout the show. We will also save time at the end. So feel free to drop questions in the chat regardless of the platform you're on. We'll do our best to answer as many as we can at the end of the program. Today, we're here to talk about Zool, a free open source CICD solution. I'd like to welcome our guests. First, some call him the godfather of Zool, it's James Blair. He's the Zool project lead and CEO of Acme Gating, offering expert support and consulting for Zool and the one and only Mohammed Nasser, CEO of Exos where users can spin up their own fully managed Zool. We'll have real world business cases to discuss plus a live demo, dangerous, showing how to manage cross project dependencies in production. James, I'm gonna pass it over to you to get the ball rolling. Thank you, Jimmy. Hi, yeah, so I'm James Blair. I'm one of the original authors of Zool and the current project lead. And let's start with, what is Zool? Well, Zool is a project gating system and I can hear you saying, great, what's a project gating system? Well, the way I look at it is, it's kind of the next step beyond CI and CD. So if you look at the history of continuous integration, if you go back to the first CI systems, CI stands for continuous integration, what were they doing? It was when you merged a change to say your Java project, you would run the build and you would see if it broke. And that was, I mean, that was great. That was better than say, finding out that it broke when you were releasing. But ultimately that was focused on the past. We call that post merge testing. And it's still a very relevant thing to do, but it's not really the state of the art in CI and CD at this point. So fortunately, since then, we've moved on to testing the present. And by present, I mean, this is what a lot of CI systems do now. They perform pre-merge testing. So this is, you write a patch, you push it up for review and your pass, sorry, your patch gets tested as it was written. And you can find out if your patch is good or not. So we call that pre-merge testing and it's like I said, it's focused on the present. For gating, for what Zulu does, it's really focused on the future. And by that, I mean, not only are we testing your patch before it merges, we're going to test your patch as it will be when it merges. So if the project has moved on since you wrote your patch, other developers are merging changes, that sort of thing, we're gonna merge your patch into a copy of the branch as it exists at that point. If there are other changes that are being merged around the same time, we're gonna sequence those changes and we're gonna make sure that your patch works with all of the other patches that are in flight. So if you're in a relatively busy shop and you've got a lot of developers working on different things, you might have not thought to test all of those things together. But a project gating system needs to make sure that all of those things are tested together before it merges anything. And finally, an extension of this is something that we think is really neat. We call it cross-project dependencies. And that's the idea that you can indicate that your patch depends on some other patch and the system will test those together. And that's actually the focus of our demo coming up later. So we're gonna talk about that a little bit more as we go on. So you might be wondering what prompted us to think that we needed to create such a complicated system. And to explain that, I'm gonna talk a little bit about Zool's development history. So it was originally developed for the OpenStack project. And this project, this slide here, it has a bunch of logos that you may or may not recognize. These are actually logos of individual OpenStack projects. While we sort of put on this face for the world of, there is a thing called OpenStack and you can download and run it. That thing is actually composed of a rather large number of individual semi-autonomous projects. They have their own identity. They have their own logos. They have their own communities which are all part of this larger community. So if that sounds like, frankly, almost any enterprise, right? Every application these days has some form of this. We're not all just working on necessarily one monolithic thing that every application sort of has tendrils that go out and little sub-projects and things like that. Anyway, we were presented with a situation of having all of these projects that needed to develop somewhat independently but still work together at the end of the day. And so we needed to build a CI system that could handle that. And so that's sort of how Zool was born with this idea of cross-project dependencies and gating. Since those days, those early days of developing Zool for the OpenStack project, we've actually, our community has grown. We have quite a large number of users now. These people are all using Zool and there are many others as well. One thing that I think a lot of these folks have in common is that they are doing, they have pretty large development efforts and those efforts, once again, span large numbers of projects. So Zool does scale down. You can totally run it down for a single project. You can run it on, you can run the software on a laptop and it can manage a single project just fine but it also scales up really, really well. And so if you want to run a CI CD project gating system for a community of hundreds or thousands of projects that are running at cloud scale, Zool can handle that very well. And we actually have a video from one of these Zool users from Volvo. So let's actually hear from them about how they're using Zool. How are we running Zool? Well, we have a small core team continuous integration architecture that was the Zool backend and infrastructure. And we support around 111 developers who consist of many smaller teams and they have at least one DevOps you can add nodes, pipelines and repositories. And whenever a new team wants to use Zool continuous integration architecture trains one of these people to become a DevOps. So why are we using Zool? Well, one of our developers is in a really good way and he believes that premium cars need premium tools and Zool is a premium tool. And it forces us to have a green master and in that way we increase our development speed. For example, the one pedal driver Polestar 2 was built with Zool. So that's great. It's really exciting to see how Zool is being used in the real world with companies like Volvo. So one of, I think one of the reasons why it's popular in organizations like that is this idea of multi-tenancy that is built into Zool. Again, when you have lots of teams sometimes working together, sometimes working in isolation it becomes important to sort of establish how they're going to work together. How does all of this happen? How do all of the teams have complete individual control over the CI system? Or do you sort of establish some parameters centrally and maintain central control? And with Zool, we actually have a lot of knobs that we can turn to adjust that to exactly what the situation requires. So if your organization has really strict standardization, compliance, auditing requirements, you can turn that knob towards central control where all of the jobs and the workflows are managed centrally. Or if your developers need more freedom you can turn that the other way and let developers in their own individual projects define their own jobs, pipelines, things like that. So there's actually a lot of flexibility in Zool for dealing with that. And beyond that, we not only have this idea of having dependencies between projects, we also have the idea of dependencies between connections. We call those cross connection dependencies. So Zool supports several code review systems. Garrett, GitHub, GitLab, Pagier and more are in the works. And this isn't the case of just, oh, you can start up Zool and point it at whichever one you're using. You don't actually have to pick one. Zool will work with all of these systems at the same time. So if you're in an organization where one group of developers is using Garrett and another group of developers is using GitHub, that's not a problem for Zool. Zool can talk to both of those systems at the same time. And those developers can actually express dependencies between their projects across those good review systems. And I've found in talking to large development shops that this is surprisingly common. So once you get a certain number of developers, you also get a certain number of opinions about the best code review system to use and editor, of course. And frankly, our goal with Zool is to accommodate all of those. And finally, continuing in that vein, Zool supports all of the different major cloud systems out there. We've got support for Amazon, Azure, Google, Kubernetes, OpenShift, OpenStack, and more are in the works. So not only can Zool support talking to these different clouds, once again, it can support talking to these different clouds at the same time. And Zool jobs are inherently multi-node. So Zool isn't the kind of system where you just get a container that can run your unit tests. You can do full stack deployment testing with Zool. So if in order to deploy your system, you need to set up a Kubernetes from the ground up, you can ask one of these clouds for Kubernetes and deploy your system on it. If you need, say, three virtual machines, then ask these clouds for three virtual machines, you'll get them. And in Zool jobs, you can individually address each of those machines and run your job on all of them. So this is great for deployment testing. So it kind of facilitates our vision of bringing continuous integration, continuous deployment, GitOps, DevOps, all under the same umbrella, all with the same software using the same processes. We'd like to say that we want to make testing more like production rather than making production more like testing. So with all of this in mind, I want to talk about speculative execution next, which is really Zools. We think that this is one of Zools' key features. And there's two aspects to this. First is project gating in general, and that is just this, if you look at the diagram on the right, when you're merging changes into a project, the idea is that we sequence those changes and we test every change in that sequence with the changes ahead of it. And if those changes pass their tests, that's great, they merge. If they fail their test, we move them out of the way. We reschedule the jobs behind them and retest it as if that change wasn't in the sequence anymore. A natural outgrowth of that is this idea of cross-project dependencies. And so if you look at the diagram on the left, that's this idea that you can have a front-end UI change. So say that's in one Git repository and that depends on a change to a back-end server. And perhaps that back-end server in turn depends on some library changes. So what Zool does is it allows you to express the relationship between all of these changes and tells Zool to test all of these changes together as a unit and you can verify that sequences of changes that cross project boundaries, that cross organizational boundaries are all tested together and work before they're merged. So this is a really key part of how Zool behaves and a lot of how people use Zool kind of develops out of this behavior. So I'm gonna turn it over to Mohamed Nao, who is going to walk through a demo of this concept. Thanks, Jim. Great. So yeah, what I'm going to do is I'm gonna go into a little demo here where I'm gonna go through some very basic code. So I'm gonna give some context here is that we have in our case here a both an application and a library. So taking into example that we have an application for our little store that we have that greets individuals when they come to the store and such that a good developers that we are, we've decided that we are going to build a library to greet individuals so that we can reuse that library in the future and maybe be helpful for other people that want to use a library like this. So we're gonna very easily look at the code quickly. So I mean, all you see is we've got a function that has to welcome to store and we're using our little hello library to greet this individual that comes to the store. Now, at the same time here, we've got some tests because we're good developers and we've got tests to make sure that, I mean, if a customer walks in and we know their name, we're expected to say hello, Billy. And if we get a customer without their name, we're expected to get a simple hello. Our very simple library that when we kind of go into it here, all we've got is a function that is say hello and given a name, it'll give the name and given no name, it'll just say pure hello. So it's very, very straightforward here. It's not exactly the most complex of scenarios that we're trying to publish, but the most important thing is just to get an idea of what we're going to accomplish here. So let's operate with the assumption that a bunch of people got together in a meeting, got into way too many meetings and decided that from now on, we need to ask for the individual's names when they walk into the store. And finally, you're tasked to do this work. So as a developer, you tell yourself, well, I know that I've got this library that does this, so I'm gonna go ahead and make a change to the library. So I'm gonna quickly hop in here to make this change. And say, you know what, instead of saying hello, I'm gonna say hello, what is your name? And at this point, it did my work, it was a good demo. Let me run my tests as well. As my tests are running, they fail. So I say, okay, wow, my tests have failed, right, because I changed the behavior when I'm saying hello to no name, therefore my tests are currently failing. So it probably makes sense to fix those tests as well, considering that we've changed the behavior. So that's what we'll do. So hello, what is your name? And we'll run the test again. So our tests are all passing now. So as a developer, we're probably pretty happy. Probably do the right thing and be on a separate branch and I'll call that just demo reading and I will push this up to the top. So no, we just did a very simple thing, copy this, we'll go into here and I think I forgot to commit. So let's just do that, fix greeting and then we'll push much easier to not do any commits if only life was that easy. Okay, so we'll create this pull request. And so now what's gonna happen is, we're gonna look at our files changed and so we see here that we've made those two changes and Zul in the background have started to do its job and it's running the check job. So can view more details and see that it's actually starting to run all these jobs that are queued right now. So those will take a couple of minutes so that we don't have to wait through it. We'll do the cooking show trick and we're going to jump right into a pull request that has the same exact change and look at what the check results are. So we see that the test have actually failed and why did they fail here? So this job specifically, the Hello App Talks Pi 3.6 job is actually running the tests of the application using the library change that we just did that hasn't even merged yet. And so what this just signaled to us is while the library test itself are passing we've actually just broken our application by making this change and we can click on this and go into the Zoologs and notice that it's going to tell us that well, there's no problem here because the applications tests here are expecting that we're getting Hello but we got Hello, what is your name? Which is problematic and you know. So in a normal scenario if you're using a normal CI tool this is the sort of change that would not be caught because you're probably running tests for the project itself and very rarely is it giving you the ability to run tests for projects that are not actually the ones that are affected by this change. Now, the other kind of thing is, you know you tell yourself, okay, well I've got to find the solution for this. The solution for this is another alternative. So, you know, you say, okay, well so that I don't break the API I'm going to add an extra flag saying ask and default that to false and only if ask is set to true then we are going to say Hello what is your name? And then we're going to also add a test for that. So again, any expensive time and we look at the checks and we see that it actually passed with this change and that's mainly because this is defaulting to false so it's just going to work naturally out of the box. And so finally, we go back to now now we know that we've got a change that works in our library and if we merge this we're not going to break our application but the thing is we don't know if our application is really going to work once we flip this switch over because we don't know that yet because we still are defaulting to ask equals false. So that's one of the really nice things with Zool is the cross-project dependencies. So this is another pull request that I made where what I did was actually add the ask equals true and change the test to actually expect hello, what is your name? And if you look at the conversation here I've set up something unique here in the pull request which I've set up a depends on and then put the PR of the change that included the library fix. And what Zool does automatically when it notices this is it automatically checks out the library change that is in this pull request even if it hasn't merged yet add it to the application tests and then run all the tests of this application with that change pulled in. And so in this case, if in a normal CI system if you make this change, it would also break. The only way that this would not break is if you somehow merge the first change or did some sort of release and then tested it after which is again, testing is going to test not pre-mer like it's not going to test the future you're testing after the fact which is after it's already broken. And if we look at the checks here we'll see that the tests actually passed because what Zool did was actually check out a version of the library that we had installed. So if we quickly look at the console here and we see here we'll see that it actually installed the hello library from the checked out local copy. And so because of that, it's really, that's kind of like the power here is now you've got this change that tests out a library change that has even merged yet. So what's even more powerful at this point is because Zool does gating this change that failed will never be allowed to merge because it's failing. And so that way your application is never gonna break because this change can't merge. This change that is with the ask flag can merge because we know it doesn't break the application but what's also even more interesting is because we've expressed the dependency here with Zool this change is not going to merge as long as the dependent change has not merged yet because if you think about it if we merge this change in some way this ask equals true doesn't actually exist because this other change has not merged yet. But because of this express dependency Zool is never gonna let this change merge so as long as this other change has not merged yet. And so you kind of have this locking mechanism where you can make sure that nothing is going to be merged that's gonna end up breaking code and you can have things merge in a sequential matter that it's always going to maintain that all the code is always passing all the tests and nothing breaks. So that's essentially an overview of what this sort of power that Zool gives you to do and I'll hand it over back to Jim to chat a little bit about how to get started with Zool and I hope you enjoyed this demo. I think just before we get to that and thank you, Muhammad, that was amazing. We've got John from Texas asking, can Zool test against projects you aren't an owner or maintainer of like upstream libraries? That is a great question. And I also see that Clint has responded to John a little bit that yes, you can pull in upstream or external, whatever you wanna call them, projects as dependencies, but it gets complicated without triggering if you wanna test all changes, that's what Clint says. And I think that's spot on and what this really goes to another one of those continuums, right? And that is sort of the continuum between how isolated you are as a project that depends on another one versus really interacting and collaborating without other project. And so again, Zool has several tools here depending on where you fall into that spectrum. If you really have no other relationship with that project then you're probably just going to be able to pull it in as a dependency and test your code with it. So that way you can make sure that as your project changes and as the other one changes, you'll at least always know that your project is either working or they've done something to break you. You'll find out early on in that case. If you have a closer relationship with that other project then you may be able to trigger runs of your jobs on changes to that other project. And so that's when the other project would get notification that they're going, they're about to do something to break you. And that's a great relationship to have with another project. And we found that, I guess in my own experience over the years, I found that I found myself in various places on the spectrum. Sometimes my use of another project is minor enough that it doesn't really warrant establishing that relationship with the other project. But other times, you start talking to the maintainers of the other project and you say, hey, I really rely on you and I wanna make sure that you don't do something that breaks me and they'll say, that's great because we don't wanna merge a change that breaks you either. And you can quite often establish those relationships between projects. So I think really the exciting part about all of this is not only is it a cool technical problem that you can solve with something like Zool but it leads to greater cooperation between projects. So it's a real great human factors thing. Awesome. We have some more questions coming in but I think we're gonna get back to Jim's get started with Zool piece and then we will address those questions in the end. So keep them coming. Great, so if you're interested in getting started with Zool, there are a number of great options here. If you, we have in our documentations something that we call a quick start. Basically, you can run a Docker compose setup and get a complete Zool system up and running in the matter of a couple of seconds. With that up and running, you can sort of start to evolve it to change it to meet your requirements add your own code review system start developing your own jobs, that sort of thing. So it's a great way to get started with Zool and start to evaluate it. And we've got the URL right here on the screen here. If you just go to ZoolCI org slash start that'll take you right there. There's once you've got a Zool up and running there's a standard library of jobs that are developed by people all over the world in various organizations. There is no, there is no central you know there's no one company that's driving Zool. It's a real community driven open source project. And as people write jobs that scratch their own itches they share those jobs with other folks in this sort of standard library of jobs that we have. So you can take advantage of that. That might sort of help you get ahead really quickly. And then finally, there are some commercial offerings here I have a company called Acme Yeating which focuses on Zool development and support. So if you're, if you wanna run a Zool on-prem and wanna like some help setting that up and support for it, talk to me. And Mohamed, I don't know if you wanna talk about your offerings from Vex host. Yeah, so I guess on another end of the spectrum if you're kind of more of a company that relies on something that's fully hosted and fully managed you kind of want to run somewhere else and consume Zool as a pure service. That's something that we offer as well. So can be running, it usually is running on our cloud and can consume resources on our cloud or on any other cloud that you wanted to run on. So we have that available as well if you're interested and all of these are available on the get started link that we've just shared as well. Excellent. Well, thank you Mohamed and James. I think we're gonna go to a couple more questions here. We've got one from Antoine. He asks, is the depends on recognized and commit messages slash PR slash pull request summary or does it have to be added manually as a comment? And I think this is specifically around GitHub. Yeah, I can comment on this because I thought this was a really good question. Personally, I have a lot more experience with Garrett and haven't done that much with GitHub. So I was like, let's go and look it up while the stream is still running. And I actually found the documentation Brazil explicitly mentions this. So I've actually posted it in the chat for Antoine and anybody else that's interested. So it actually talks about because of for GitHub it has to be in the PR description. That's what the documentation says. Maybe Jim will say, well, we've changed that. But as far as I know, the dependencies happen on a PR level because the mergeable, you could think like the object that gets merged is a PR in the context of GitHub. And another thing like Garrett, the context of the blob that gets merged is usually a single commit. And so therefore it works in the footer of a commit message in Garrett. And so to answer your question in GitHub it has to be in the pull request description whereas in Garrett it would be in individual commits individual commits message. Excellent. So another kind of common question we have circular dependencies. So what if you have two changes and they both depend on each other? How does that work? This is a really exciting one because for quite some time Zool did not support this. Our, as Zool was being developed for the OpenStack project we kind of had this view that we never wanted to have two changes that depended on each other at the same time. And that's because of the upgrade strategy that we chose for the OpenStack project. Each of those different components needed to be able to upgrade on its own. And so there should like always be this way to progress through an upgrade component by component without having to take the entire system down. So we kind of enforced that in the CI system by not supporting circular dependencies. You always had to be able to say you can get from point eight to point B one change at a time in sequence. As the Zool project grew to support other environments we found that other people didn't have that constraint. In fact, they had the opposite constraint. They said, you know what? We're not upgrading cloud software. We're upgrading embedded firmware inside of an automobile. So this is not something where we actually want to upgrade one thing at a time. We want to ship the entire blob over with all of those changes at once. We need to be able to make those codependent changes together. So Zool has grown the ability to support circular dependencies. And so basically any kind of graph you can come up with you can encode that into Zool. So it's not just two changes that depend on each other. You can grow that to whatever you can manage to fit inside of your head, really. So yeah, that's a great question. And Zool supports both and it does so as an option. So we still actually, you know, we set the option in OpenStack to disallow that so that we can't accidentally get into a situation where we have circular dependencies and then people who need it, they set the option the other way and developers go crazy. So Zool basically prevents me from hurting myself. That's the idea. Perfect, I love it. And so we've got someone else, John from Texas again, this guy, what are the downsides of gating? Why isn't everyone just doing it this way already? Well, I think the answer is because they haven't watched this live stream yet, but hopefully after this, once they know this exists, they will. So I think there certainly are trade-offs, right? And one of the things that happens when you start gating is it becomes difficult to merge code that is broken. I think that's a feature. And I think that's kind of key to accelerating development within teams and organizations is, you know, there's this idea of like move fast and break things, right? With Zool, we actually, we think that you can move fast without breaking things. And the way to do that is to make sure that everybody on the team can perform their work without stepping on the toes of anybody else, right? And that's what happens when you merge broken code. You merge code that you think works and it slows somebody else down because suddenly they can't get their work done. So by keeping things green all the time, Zool helps speed everything up. At least that's been my experience. When if you're not used to this, that can be a hard thing to wrap your head around. You might say like, oh, well, what if I just need to merge something, right? And honestly, at the end of the day, that's kind of an organizational question. Like under what conditions is it okay to just merge something? And I would say that normally you shouldn't be merging things that are broken. Maybe your project gets broken by an external dependency and it's outside of your control. What do you do then? Well, I mean, there's always escape valves, right? So if that happens and you need to merge a quick fix, sure, there are ways to bypass gating to get something into production really quickly. That's something that you can do. So it's kind of about like, what are the norms for the project, that sort of thing. If it's not an emergency and some external dependency breaks my project and that prevents me from merging code, that's okay from my perspective. That's actually, that's good information for me to know. I have learned that the project is broken. It's unfortunate that it happened that it couldn't be prevented. It may have been able to be prevented if you established those relationships that we were talking about with John's question earlier. But stuff happens and in that case, what do you do? And from my perspective, I wanna know that it's broken. And usually what I do is the next change that I merge is the change that fixes that again. So yeah, that's kind of how I see that. I don't know if you have anything to add Mohamed from your experience. Yeah, so I think another thing is, I feel like it's probably gating is one of those things that is kind of like a big change and shift in how we do development. It's something that maybe if you're involved in some of the OpenStack communities, it's really, it's something we're used to. But I think if you think about it, many, many years ago, there was no continuous integration. It was everybody pushing to a branch and then one day a bunch of QA people would go in manually, you know, run some stuff and run through some lists of tests and say that everything is okay. And then you would get that thumbs up. And then eventually with time, that transition to machines and computers giving you that thumbs up. And so I feel like one of the things is it's just not something that is so widespread yet. So there is kind of a lack of exposure. And I think a lot of people feel concerned or you know, it's probably like a fear of the unknown, like not really sure about this and never really seen this happen a lot. And it's like this weird feeling of like control over your project, like I want to be able to merge something when I want to do it. And when I want to do it, I want to do it now. But that potentially risks slowing things down even more down the line. And yeah, I think also Antoine's comment is really good. I mean, yeah, if there's dependencies that are breaking, fixing the dependencies generally will allow you to keep moving again. Oh yeah, probably would be nice if you could just from the get go, never have these breakages. But you know, you're gonna find out about these breakages whether you're gating or not. It's just a matter of will you merge code while your stuff is broken or will you merge code all the time? Sounds like gating is the future basically. I think so. So how do I deploy as well? What are the different ways I can do that? So there's, like I mentioned earlier, there's the, we've got the quick start which will get you a Zool deployed and Docker compose. So that's a great way to get started. If you can of course, so Zool is written in Python, you can install that on some virtual machines just like you would any other Python program. For some of the Zools that I work on, we actually, we deploy them in containers on virtual machines. That's kind of a best of both worlds situation in that we get the atomicity of container upgrades and that sort of thing in an environment where we don't necessarily have access to a Kubernetes or that sort of thing. If you do have a Kubernetes or OpenShift, there are, you can play it that way. There are Helm charts out there. We have a Zool operator that's a work in progress right now. But if you're really into early adopting and you love Kubernetes, you can jump in and help us with that. So I think those are all great ways of deploying Zool. Awesome. So Albin asks us any comments about running reproducible build systems with Zool like Nix or Bazel? I have a little bit of experience with Bazel. We're actually doing some work with the Garrett project for a Zool that helps test changes to Garrett plug-ins and things like that. And they use Bazel a lot for their build system. And it's really interesting. And I think it's a really good fit. I mean, the key concept in these systems is sort of what is effectively the cache, right? So once you've built something can you put that thing in the cache and then make sure that you get the same value? You can pull it out of the cache and save time. You can be assured that builds started from the same starting point and things like that. Exploring how that interacts with Zool and its speculative execution is actually really exciting. It's sort of like what is a reproducible build system and a build cache mean when in a world where you're constantly doing throwaway builds of things that you may not end up merging in the long run. And so far we're actually coming up with some pretty good answers for that. It's all about identifying is this build speculative or is it final, right? And so what we're doing with Bazel is saying, well, when we complete one of these speculative builds we'll use the cache to help build that but we're not going to put the result in the cache until we're sure it's final. But in the gating process, when the underlying change merges we at that point know that it's safe to put the results into the cache. Effectively the build is finalized and we can put that in there and then builds after that can use it. We actually take the same approach with Docker containers and things like that. So we're always building the speculative Docker containers and sometimes we even push them up to the registry but we don't tag them in a finalized way yet until the change merges. So really what I've done is I've twisted this question into how does gating apply to other systems? And the answer is every system we've worked with so far we've found a way to integrate it and it makes sense and it's really exciting. That is really exciting. So one question that I've gotten a lot is what differentiates Zool from just your run of the mill CI? I know you've covered lots today on gating and speculative execution but aside from those things, what's different? Why Zool? I'm going to ask if Mohamed wants to answer that maybe put a different perspective on things. Yeah, I think so for me, I think the biggest points are first of all the idea that you have a gating for your code. Gating helps merge, you don't merge any broken code. So you don't have any problems with your main branches breaking. As a matter of fact, I always thought it was funny for a while. I think still the case that the, you know how CI tools usually have a cute little icon that says passing or failing and the Zool one is a static file because if you've got your jobs doing right, it's just always passing. There's no need for us to dynamically determine if it's passing or not, because it's passing. And so that's something to kind of keep in mind of how what Zool kind of provides to you. The second thing I think is the cross project dependencies is a super powerful tool that allows you to iterate and do work across multiple different repositories to accomplish what you need. And this goes for so many different things, right? You've got scenarios where if you have a repository that contains the configuration of all your systems and you've got independent repositories containing Ansible roles that you use, well, you can do speculative testing by changing, making a change in a role and then running the tests on the repository that's running your playbooks. We saw an example of a Python library and application. There's other things you can do such as maybe using something like Terraform and building modules that in modules that are all separate and all fully contained on their own and then consuming them later. There's also some more complex and advanced things that you can do with speculative things like speculative containers. So building containers that are just not even built yet. Like you make a change in a Docker file, you don't need to publish that somewhere for you to actually make a change for their Docker file that depends on this. It'll automatically go and take care of all of that. So there's a lot of kind of power in that in being able to allow developers to be more productive without actually having to merge anything. And I think that's another powerful thing because if you think about it, let's assume in a normal scenario where you've got some more senior people that are doing code review and other folks that are writing code. Well, if you can't start your application change unless your library changes landed, now you're kind of having this bottleneck where you constantly need to have a human that goes in and merges that code for you so that you can continue with the rest of your development, which could be pretty distracting for whoever's looking at the code. Whereas with Zool, you can actually start making the library change, make the application change to a whole bunch of other changes and then present that as a single unit to be reviewed and know that all the tests for all of these different components have all passed together without you having to wait for code approval or merging something before it can get done. So I think that's also really powerful. And the one last thing that I'll add up very quickly is the huge library of roles and existing jobs. So because Zool's been around for a while, there's jobs for, we just talked about, there's jobs for Bazel, there's jobs for Python, jobs for Terraform, jobs for Docker. So there's a lot of pre-existing job libraries that are very well tested and maintained by the community. So at any time you can get the project up and going really quickly by using those. I think that's mainly the main things. That's awesome, that's a great answer. So I think we just have a few minutes left here. One thing I did wanna find out is what's next with Zool? Well, yeah, that's a great question. We have a lot of work in progress right now. One of the big things that we're focused on at the moment is scaling. Zool is already one of the, it's running CI for the world's third largest open source project. It scales pretty well already, but having said that, we still do have one single point of failure and we're gonna get rid of that. So that'll actually make every component of Zool fault tolerant and scalable. So we can grow even larger and be even more robust. We've got a project to, so Zool has a built-in secret storage system that you can use on its own or you can integrate with other secret storage systems. We wanna make sure that all of that sensitive data is stored encrypted at rest. And we're working on improving our metrics and logging. So basically better integration with various monitoring systems and that sort of thing. So a lot of operational stuff right now, we just came off of a release where we did a bunch of new in-user visible features. And so we're trying to kind of back that up with making the system itself more robust and reliable. That's exciting. You always have to do after your big release, you always have to go for the stability. That's great. So listen, thanks to both of you for coming out today and for joining us. It's been really interesting. If you're like James and Muhammad and you're supporting an open infrastructure project, we would love to have you join the foundation to continue supporting this community. If you're interested, hit us up at openinfra.dev slash join or email me directly at jimmy at openinfra.dev. Next week, we are super excited to host our open infrastructure foundation chair, Allison Randall. She will be joined by Jonathan Bryce and Mark Collier to answer your toughest questions around hyperscale, public clouds, security, software supply chains. So think hard, make them tough. Don't go easy on them. Share your questions with us in advance on all our social media channels, Twitter, Facebook, LinkedIn, Myspace and definitely join us live to see it all go down. And don't forget, if you have ideas for future episodes, we want to hear from you. Mark your calendars and I hope you're able to join us next Thursday at 1400 UTC. Thank you again to James and Muhammad for the great conversation today and we'll see you all on the next openinfra live.