 Hi everyone, welcome to Open Infra Live, the online webinar hosted by the Open Infra Foundation talking about all things open source infrastructure. Today we have our first episode of 2024 and I'm excited to be here. My name is Allison Price and I will be your host for today. We are streaming live on YouTube and LinkedIn and we'll be answering your questions throughout the entire show. So feel free to drop questions in the comment section and we'll answer as many as we can. Today we are joined by the leaders of OpenDev, a collaboratory for open source software and development at scale. They'll be talking about how you can leverage and contribute back to the OpenDev community. Please join me in welcoming Jeremy Stanley, Jim Blair, and Clark Boylan. Hello everyone. All right, I'm gonna kick it off to y'all so y'all can talk about what OpenDev is and the history of how we got to where we are today. I guess I can jump in there. So OpenDev really the origins of it lie with the OpenStack project I guess what probably late 2010. I know Jim was around at the very beginning. I and Clark did not come on until I guess 2012 but basically the OpenStack project was very popular, very active new open source project and needed a fair amount of collaboration infrastructure to really help the developers who are working on it coordinate to be able to produce the software there at the beginning. And so people with experience in other open source projects were being brought in to kind of help shepherd this newly forming project provide some context and open source ways of working and so on. And so they applied that to the infrastructure that was being used to collaborate as well. And in particular, I know some of the early people working on the infrastructure came from an open source project called Drizzle, which was a database relational database server. And they had an egalitarian approach to code review and testing and basically the idea that whatever merges to the revision control for that project should be fully working rather than testing things after they land and finding out that stuff's broken and then having to figure out who broke it and go back and so on. So that was basically from the beginnings of the OpenStack project. The idea was that all of the changes that were being proposed to the various components of OpenStack would go through a battery of code review and testing before it was allowed to merge the repositories. So that required a fair amount of infrastructure. We were I guess at the very beginning using Hudson, which later forked to become Jenkins, created the Zool project pretty soon into that to start coordinating the activity that was a little bit too intelligent for Jenkins on its own to manage. Over time, we kind of needed to scale some of that out. We were also running mailing lists, a meat bot for the projects to be able to have automated minutes of their meetings that they held in IRC and so on. So that was sort of the very beginnings. I don't know if anybody has anything to add to that. Yeah, I think, oh, go ahead, Jim. I was going to mention that I think one of the other important ideas that were brought along in the early days is that open source tools were important for what the project was doing. So very early on at that time, the CI system was open source at Hudson and Jenkins and the code review system Garrett was an open source tool and the mailing lists were built on Mailman open source tools. And so there was a kind of from a very early beginning start of the project infrastructure for OpenStack, there was this strong idea that we should rely on open source tools as much as possible to get the work done within the OpenStack project. And then at the same time, we were relying on infrastructure donations from donors for like cloud hosting companies. At the time, Rackspace and HP Cloud, then that kind of also continues today where we're working with different cloud providers and cloud systems to run and build these services. How many clouds is open to have up to now? I don't know if it counts providers or regions, but it's around six, I think. Six different providers. Yeah, I think we've got if you count individual regions, it's probably close to double that. Yeah, so just to kind of jump ahead a little bit, it's from an operational point of view, it's kind of neat that we treat the entire service as a multi-cloud application. And that directly comes from the way that this was started as relying on these donated cloud resources, because that's constantly changing. New folks show up and add resources, resources disappear or become less suitable for us for some reason and are removed. And we just roll with it. But going back to what Jeremy said, I think a lot of the character of Open Dev and the nature of the services that we run really goes back to that one idea that we call project gating. And that was what Jeremy described as every change that merged to the Drizzle project had to pass tests before it was merged. They put that in as a hard rule for that project. And when those folks came over and started working on OpenStack right at the beginning, they said, let's start with this rule in place. And really OpenStack, the development process and community of OpenStack developed around that. It's a really great way to work. And if you can get that rule in place early on in a project, that's great, the earlier the better. You can always retrofit that later on. You can always say, well, starting from now, we're going to make sure that nothing merges unless it passes tests. But it's always a little bit of an uplift. And the reason why is the reason for the rule in the first place, which is that if you don't do that, your tests are inevitably broken in some way, right? And that's what slows down velocity on larger projects. That's, you know, if somebody merged a change because they knew it worked well enough, right? And so what if a couple of tests are broken, right? Well, those couple of tests are preventing somebody else from doing their work at the same time. So if you have a rule like that in place, this project gating rule, then it means that you can develop at full speed and everybody else can develop at full speed at the same time. And that's really what was needed as OpenStack was growing. Because the OpenStack project started as two independent projects that sort of got put in a room together and said, hey, guys, you're OpenStack now. But in that room, everyone saw that the future was going to be more and more projects taking on more and more of these cloud services. And so it was very much, how are we going to grow the development process and the community together to do all of this? And that kind of thinking that we need to keep everybody working at high velocity and working independently yet towards shared goals really drove a lot of those decisions. I mean, the ideas behind project gating are actually not all of that surprising to people today. But certainly 14, 15 years ago, this was extremely novel. And I think it really goes to show how effective that approach was that at the time when it was virtually unheard of in an open source project to work that way, it's now how a lot of large open source projects operate. And we did a large part of the proving that it was possible. We also talked about it at a lot of conferences over the past decade. So we've been out there sort of touting the benefits of this for a while. So I think it's been great to share that experience and help promote the idea out in the world. I think if you're an open source project, this is even more important for you because most of your developers probably aren't all sitting in one office together or in the same time zone. And so they can't walk down the hallway and be like, hey, this test broke. What happened? Oh, I checked in this thing. Sorry, and work it out. It's a lot more asynchronous. And the time delays involved for things being broken are larger. And so it's important to keep everything working so that the global community involved in these projects can kind of keep rolling forward, which is probably a good lead into how open dev came to be. We've talked about how there's this open stack tooling and development process and structure. And essentially, what happened is that there was a realization, and I think we knew kind of early on that this is useful beyond open stack. And we would like other people to be able to take advantage of these tools and use them for their projects, whether they're open stack or not. And so very early on, there was a concept that we called Stack Forge, which was essentially kind of like the very early beginnings of open dev where the idea was we can provide these tools to projects that aren't open stack. And that, for a number of reasons, kind of petered out. The hosting for that was a little odd. And things had to get migrated and moved around. And eventually we realized, no, we should be focusing on that and making it possible for Stack Forge to live beyond where it had been. And that's where I think we ended up with open dev is a bigger concept of open dev hosts open stack rather than the open stack infrastructure hosting other projects. Back to Clark's earlier point too about the importance of open source tooling, I think it's been really one of the biggest drivers in our design choices. That's the very beginning open stack development was happening to a large extent with Bizarre and Launchpad. And the people involved in the project saw that Git was really the future of revision control. I mean, it had its flaws, but it was gaining in popularity so rapidly that it was obvious that the project was going to need to migrate to Git. And I know GitHub was still pretty new at the time and it was looked at, but really a large part of why we ended up using Garrett was because Garrett is open source and GitHub simply is not. There were also technical reasons. They're actually, at the time, we would not have been able to implement the project gating rules and the code review process that we were already using on Launchpad with Hudson. We looked at doing that with GitHub and it wouldn't have been possible at the time. They've since added features that do make that possible, which is switching hats here for a moment. So I wear a lot of different hats. I maintain the open dev infrastructure, but I'm most focused on the Zool project, which is the project gating system that we've spun out of all of this effort. So putting on my Zool hat for a moment, one of the things that we've done with Zool since the early days of all of this is we've expanded the code review systems that we support. So Zool itself supports not only Garrett, but also GitHub and GitLab and Pag Gear. So that's been good for folks that want to take the code review and project gating methodology to these other systems with their own self-hosted Zool inside of their companies or whatever. But so it's been good to see that, I guess, that way of working gained enough acceptance that basically you can do it with any of the code review systems now. But just to be clear, OpenDev is still very much focused on just the the resources that OpenDev provides. So OpenDev runs a Zool and it runs a Garrett and basically the Zool that we run is in service of the projects on the Garrett that we run inside of OpenDev. It also says a lot that the advanced workflows that we were relying on didn't become possible in GitHub until a decade later. But we were able to do that with 100% open source long before that was possible on their proprietary platform. Right, and I think that's an important thing to note is that as this kind of infrastructure we've built kind of more from the OpenSec infrastructure to OpenDev, we've continued to keep a lot of those strong principles. We want everyone to do project gating on the hosting that we provide. But we also want to continue using open source tools to run these services and we're still using Garrett because it's open source. We're using Zool, we build Zool with the Zool maintainers and so on. So while we've changed kind of the branding around what this is to be hopefully more welcoming and inclusive to other projects because not everyone wants an OpenSec banner over the top of their project, which is totally understandable. But we haven't left those kind of principles behind. This is the development process that we believe works well for open source projects and we also believe that open source projects should be using open source tools and so we continue to build and run and deploy those tools for open source projects. So we actually had a question that I think is really interesting and the blog may go into it, but I think it would be fun for y'all to answer live. So Song Soo, one of our Open Infra Korea user group leaders, he said he remembers that Jenkins was a CI for OpenStack a very long time ago and wondered what inspired y'all to create Zool for the CI for OpenStack and other open source projects? Well, why don't I go into that a little bit and yes, that blog post is a great resource for some of this information and I definitely encourage folks to sort of follow up and read that. But I'll give my sort of oral history of that. So we, and just to be clear, we're actually, I'll speak for myself and I think at least some of the other maintainers, we're big fans of Jenkins. We really like Jenkins and we consider them our open source friends in the CI space. Some of us have actually contributed pretty significantly to Jenkins in the past. And basically what was going on is we were using Jenkins to implement this project gating process. And as we began to do that more and more, it became clear that the structure inside of Jenkins wasn't really suitable for that, especially once you start using multiple projects. So as the OpenStack project grew and we added more than just those initial two projects, what we were faced with was this struggle of, well, we want to make sure that nobody breaks, we want to make sure that every change is tested before it merges. Okay, that's not too hard to do with Jenkins, right? But then we had two projects that interacted with each other and we wanted to make sure that if we merged a change to one of the projects, it wasn't going to break a different one. That was a little bit harder to do with Jenkins. We managed to find a way to do it, but it was starting to stretch Jenkins capabilities a little bit. And then we added more projects and more developers and the rate of changes increased. And at one point, it was taking like an hour to run the tests for the entire OpenStack system. And there are 24 hours in a day, which means we could across all of the projects that we had, all we could ever do was merge 24 changes in a day and we kept adding more projects and more developers. And the math is pretty clear on that. We weren't going to be able to sustain it. So we started to have, we basically needed to move beyond Jenkins internal scheduling. But we were really synacious and we really liked Jenkins. So we actually, we did kind of a crazy thing. The first version of Zool was actually just an external scheduling engine for Jenkins. And what it would do is instead of letting Jenkins decide when to run jobs, the first version of Zool would just tell Jenkins to run jobs and then use the results of those to make its own internal decisions about gating. And it would decide when to merge changes and whatnot. So we kept using Jenkins to run these jobs, but the decisions about when to run them and how to run them and what changes to include in them were all moved out into Zool. So Zool became the gating system, Jenkins became the job execution engine. We did even more crazy things as we continued to scale. I think we probably made the world's first multi-master Jenkins system by hooking out to the Gehrman job scheduling system. So what we would do is we would have Zool put jobs on this Gehrman queue and then the various Jenkins servers that we had would pick up the jobs from them. And so we actually had like a fleet of eight Jenkins servers at some point, all basically top-level Jenkins masters. Each of those had a hundred different nodes that they would run jobs on. So at some point we were like, okay, well, we've really, really pushed Jenkins as far as we can. We're not using the features in Jenkins for this anymore. And so we took the plunge and replaced Jenkins as the job execution engine with Ansible. And so now at this point, Zool is completely separate from Jenkins. It's its own project gating system that uses Ansible to execute its workloads. So that's kind of how we grew out of Jenkins both in terms of scale and our workload needs and ended up making Zool and then making Zool its own standalone project. Yeah, also I would say a big part of that late push into putting more logic into Zool was because when you started having multiple Jenkins servers, you could no longer rely on the Jenkins dashboard because it was spread across all of these different servers and you had to figure out which server had run which build. And same thing with log hosting, like now you needed to aggregate logs from somewhere. So basically we needed a separate dashboard that provided an overview of everything that would have been on all the individual Jenkins dashboards and log hosting that was effectively disconnected from the Jenkins servers themselves and aggregated all of that. And so really once you've taken out the web interface and the log hosting and everything else, there wasn't a whole lot of Jenkins left in the picture. I guess one thing that Jim didn't touch on in kind of the history of Zool is how it, so it was a scheduling engine for Jenkins, but from day one it was a scheduling engine that understood how to do project gating and that was modeled after CPU speculative execution, right? So there was some outside influence into the design. I think it seemed to recall there was like an IRC message one day where Jim's like, look at this cool thing I did. Yeah, there was kind of going back to that thing where we're like, oh, there's 24 hours in a day and it takes an hour to run tests. How are we going to solve that problem? And yeah, what Clark was saying is we took a page from the flagship feature of the Pentium processor, right, which was speculative execution. And we said, well, what if we were optimistic and assumed that all of these tests are going to pass? We could, what we could do is just stack up a series of changes that we're going to merge together, run tests for all of them in parallel. And if they all pass, then we can just merge them all at the same time. So imagine taking 10 changes and you run jobs for all 10 of them at the same time, they take one hour, at the end of the hour, you merge all 10 changes. So that's sort of how we increased our throughput without actually reducing the fidelity of the information that we provided back to developers. We're still running tests on every individual change so that we can, if there are any problems later, we don't have to bisect them or anything like that. We know that each individual change works alone, which is, from a development standpoint, that's really nice, because at any point, you can revert any single change if you run into problems later. It's not like that batch of 10 changes has to be reverted together or bisected to figure out what's going wrong. You know that every change to the project is a valid change that stands on its own. But so we were able to still provide that guarantee to users, but increase the throughput at the same time. Then, of course, if one of those jobs does fail, then we've lost some time. We have to restart the jobs. We reordered the changes to remove the failing change, that sort of thing, and restart. But that's still going to be less, you know, the worst case for that algorithm just ends up being the base case for the algorithm that we had before on Jenkins, which was run every change serially. Well, I was thinking, so I know we've gone through history and a lot of the different components of how we got to where we are, but I wanted to get talking about the nitty gritty. How do you actually operate these services? What goes into it from a community perspective? Yeah. So I think a very important part of what we do in running the services is that we actually dog food the services. So what I mean by that is we use Garrett and we use Zool to deploy Garrett and Zool. And sometimes this can create problems where, you know, you've got dependencies that are annoying because, you know, how do you restart Zool if Zool is responsible for restarting Zool is just kind of the classic. But what that means is we're using these same tools. We know how they work. We want to make them better, not just for you, but for ourselves. And we're doing everything in the open on top of those tools because Garrett is doing code review for open source projects like OpenStack. When we do code review for OpenDev in Garrett, it's out in the open. Anyone can review it just like you can for say OpenStack or Zool or StarlinkX. And what that means is, you know, you don't have to be an OpenDev root administrator to get involved and say fix a bug in a system or, you know, improve some feature in one of the services we deploy. You can go out and make those changes, propose them just like you would to any other project we're hosting. And then the configuration management will take care of it once things are approved and merged. I don't know, Jeremy, do you want to talk a bit about the configuration management? Well, I would probably jump in first and say I think this is an excellent demonstration, maybe even the flagship demonstration of the complex capabilities of Zool as a CI and CD engine because we are effectively able to propose changes for our configuration management and orchestration, see them work in mock-ups effectively of the services that they would normally deploy. And then if they work and if we're satisfied with them from a review perspective, then Zool itself is deploying those same changes automatically once they get approved. So it's really end to end management of the services themselves where we can test every single component to make sure that we're not going to break them before we emerge a configuration change or an automation change or whatever. So it's from a systems administration perspective, it takes a little bit of wrapping your head around, but it's really freeing to have this sort of automated, I mean like in traditional systems administration environment, you would have say a test environment and you would deploy your proposed changes to your test environment first and then exercise it and make sure that everything worked and once you're satisfied that it didn't break the test environment, then you deploy it to your production environment. And then the next time you wanted to test something, you'd have to go reset your whole test environment and start all over. And so with Zool, we basically get all of that automatically because it's creating ephemeral test environments on the fly whenever we want to test a proposed change and we get the interdependency support. So if we are proposing a change for let's say our meat pad infrastructure, for example, we've got systems that we host that are integration of Jitsi meat video conferencing and ether pad collective collaborative documentation editing. And we can test whether a change that we're considering making to Jitsi meat is going to break the integration between Jitsi meat and ether pad. So even though those are being run as distinct services, the integration between them is still testable and we can even order changes that we're making to both systems and make sure that that sequence of changes to both Jitsi meat and ether pad configuration collectively aren't interfering with the integration between those. From a nuts and bolts perspective, again, we've got similar choices to to what Zool itself uses internally. We're using Ansible to orchestrate all of the configuration and restarting of services and so on, which again provides a nice integration from a Zool perspective because we can reuse those same deployment playbooks as basically job execution payloads for testing the services in the same way that they'll be deployed. I think that's probably a key principle of the modern design we have for OpenDev is that we test things the way that they will be deployed, not I mean to the extent possible. Obviously, we have some placeholder credentials and we may create self-signed certs for some things and whatever when things are being tested, but but as much as is feasible, the same automation that will do the production deployment is used in the testing of those configurations and automations. I'm back which means we have a question from the audience and I think this one fits in nicely with what y'all are talking about right now. So is Zool gating plus Garrett the first in the sense of automated battery of tests plus code review at large scale? So I think there's I guess in our conception of the development process we actually, I would say it's kind of review centric, right? So we see the code review system as the nexus for ensuring that a change is suitable for emerging. So when somebody uploads a change to Garrett, a lot of things can start to happen at the same time. Other developers who are working on the project can look at that change in Garrett and start performing code review so they can start making decisions about whether that change is a good change for the project, whether it fits the overall direction, whether it's a good technical implementation, things like that, right? All these sort of human decisions about whether a change should be merged or not. And then at the same time, Zool starts running the automated test suite for the project and potentially integrating it with any other changes that it depends on and things like that. So for most of the projects hosted in OpenDev, they're pretty significant in terms of size and complexity. So that might take an hour or more to get those initial test results back. When those are reported back, they're reported back to Garrett as well. And so what we're trying to do is we're trying to present all of the information about a change to the developers who are maintaining a project inside of the Garrett code review system. So that's kind of how they work together and how they feed this process of deciding whether or not to merge a change. Once those automated tests look good and once the humans have looked at the change and decided it's a good idea, then at that point they approve the change and that's the signal for Zool to go out and actually try to merge it. So it runs tests again. And just in case anything has changed in the mean time, so that at that point it knows that that last set of tests that it runs are going to test the change exactly as it's going to merge. And if they pass, then Zool will cause the change to be merged inside of Garrett and it goes into the Git repo. So they really work together that way all in service of the developers who are maintaining the project. If the question was more about the history and whether our use of Garrett and Zool together is the earliest example of combining code review and automated testing at a large scale, it's probably one of. Like we mentioned earlier, the Drizzle project had something similar set up at the beginning before OpenStack existed. To some extent, the popular Linux distributions, Ubuntu for example, had something not too far off from the concept which is why Launchpad kind of existed and had integrated automated package building and so on and had its own revision control built in. But I think from as far as what really kind of made this combination we were using somewhat novel was that it was again very focused on code review whereas a lot of the previous test automation approaches did not have nearly as much of a code review centric approach. I think among open source projects it probably was the first large one. I'm not aware of any... Firefox had something not quite the same but fairly larger scale at the kind of concurrently but they didn't do project gating. That's why I say it's not quite the same but they did run lots of tests and it was kind of code review centric but not quite integrated in the same way with project gating. We tended to see a lot of open source projects doing one or the other but not necessarily combining them the way that we did. Garrett came out of the Android project and their need for extensive code review but they didn't really have their automated testing tied into it quite so quite so thoroughly as what we ended up with. And occasionally you might hear about companies that have some of these features internally in their own systems. I can think of a few sort of famous tech companies that are rumored to have something fairly sophisticated in terms of how they do their testing and development. I think one of the things that we've heard from folks working in the industry is that this combination of well especially Zuul but Zuul with any of these code review systems is as close as you can get in the open source world to the sort of fabled internal testing suites in some of the large tech companies. Perhaps it even surpasses it in some ways. It's a little hard to know. Some of these companies are pretty tight lipped about it but the fact that as Jeremy described earlier the fact that in OpenDev our process for making a change to our production system is to in fact have an automated system spin up an entire ephemeral copy of that infrastructure to test that change in an automated way destroy it at the end of the test report the report the results back to the user and give the folks working on the system basically complete confidence in deploying a change before it happens because our test system is so exactly like production and we have an infinite number of test systems that anybody can can can cause to be created just by writing their own change. That's extremely powerful and I think if it doesn't surpass it's it equals what is available in in you know some of these large tech companies what they're doing privately so I think having an open source project that can do that and then having something like OpenDev as a public demonstration of that is wonderful and one of the most important features to me that that enables is that anyone who wants to help us with the management of these systems is able to do that through public submission of proposed changes code review they can help review configuration changes that are proposed by other users and so on and so it again allows us to sort of crowdsource the the management of all of these services by engaging the community who are relying on them to to help with their management. Going back to the the actual kind of nuts and bolts of how we're deploying things Jeremy mentioned we use Ansible so Ansible is the kind of configuration management system that we use and if you're familiar with say OpenStack Kala we're using Ansible to deploy container images of our services and this ties back to what we were just discussing of how you know we have these ephemeral environments and and so on. We also have ephemeral container images where you can make changes to the container image a new container image will be built and tested in those ephemeral environments to ensure that you know things are working the way you expect it to before anything merges and so you don't have to make a change to your container image first push that up to say Docker Hub or Quake he and then pull it down in your tests everything is happening together in a pre-merge environment which just builds your confidence that you're not going to break anything right that if you had to update the container image first you'd worry that maybe you'll accidentally have some automated system pulled down that container image and deploy it early we don't have those concerns because everything is happening in that ephemeral environment before we take the plunge by saying we approve this please merge it and deploy it and and by taking advantage of Zool's cross-project dependencies support we can even do you know sequence changes to different layers of a multi-layer container image or to the software that we're going to be putting in the container as well as the configuration of the container image itself and test all of that together across multiple different Git repositories. Okay so it's nice getting kind of into the nuts and bolts of like how does this actually work so if you know people are out there and they're like this sounds really great but like what is what is the actual tactical way to start using you know the open dev services like what is the best way to start exploring what that actually looks like in a real life scenario so there's probably a couple approaches you could take the the easiest is most likely going to be to contribute to one of the projects we already host you just you know create an account push some code then you can you can see that the code review system in action as people review your code you can see the CI system in action after it runs tests against whatever you've proposed so that includes changes to open dev you know I think it'd be great if people were interested in open dev and then maybe they just contributed like oh hey here's a typo somewhere whatever you know that will expose you to a huge amount of what we do through you know through the process of getting the code contributed tested merged deployed another approach would be to just take the plunge you could say I've got this project and I'm gonna host it with you I think this is probably you know more applicable to people who are familiar with with Garrett and maybe Zool because they've used it in other other places I think it is worth noting that you know Garrett has a different workflow than say GitHub or GitLab and so you want to at least be somewhat familiar with how things differ we think or I think at least that the Garrett workflow works really well but it is you know worth being aware that it is different it's it's it's much more focused on say code review than than just generalized software I don't know I don't want to misdescribe say the GitHub workflow but GitHub is is kind of more generalized in and say hosting a project you may or may not use code review you may or may not even use an issue tracker or the wiki or whatever it's it's a lot more flexible in terms of how you want to use the tool whereas Garrett has an opinion that when you contribute code every logical change is a unit of code review and that unit of code review is a Git commit you can you can have those Git commits changes in a series and each one will be reviewed kind of in sequence versus say a pull request in GitHub where you can have a bunch of commits in a in a single pull request which is reviewed together and so there's just the slight differences in kind of how you approach using the tools but once you're familiar with them I think you know I like to say that using Garrett isn't any more difficult than using GitHub it's just a matter of familiarity you're using the same tools you're using Git you're using Git push it's just you know where you get pushing what is the website you're looking at and then being familiar with the workflow is I think where people get tripped up and so just if you want to get involved have a little bit of familiar familiarity with it and you can dive in another way to do that is we have a sandbox repo which just kind of lets you poke at the tools you can fetch that down push code review change up it will get tested though whether or not the test pass I don't know it's it's kind of the sandbox is a little bit of a free-for-all where you can you know we don't place the same stringency upon that say as our production code but the idea is to get you familiar with with the general workflow and see how things work also we do provide some extensive documentation in our info manual about these workflow patterns also general concepts things about adding new projects process documentation and so on a lot of the design in more recent versions of Zool has been directed to sort of make the CI system as self-service as possible and that's that that comes directly from the fact that that we as open dev administrators actually don't want to have anything to do with what jobs are running on the projects that are hosted in open dev so you know there's there's no there's no CI team that's going to like make sure that that that you're there's no CI team that's going to write the test for you or even write the jobs that run the test for you that's all stuff that that that projects that are hosted on open dev their developers can can create their own new jobs and and and the the nice thing about with Zool is that as soon as you write a change to create a job Zool will start running that job you don't even have to merge it because everything in Zool is speculative even the job definitions so you you write a change that adds a job to Zool Zool says oh hey a new job I'm going to run it and if you merge that change then that job and if you merge that change then that job gets run for all of the subsequent changes right so that's all very self-service and and it comes from from the sort of philosophy in open dev of of letting the the projects manage that on their own there's there's things that you can do on top of that for instance open stack has its own uh it has a technical committee that that sort of approves and sets standards for all the projects that participate in open stack and and so there are some things that they enforce and Zool also lets you lets you do that it lets you set some standards for for sets of projects that can all be accomplished in in open dev as well but but just to get started to add a new job for your own thing that that's entirely self-service we do have a standard library of jobs so for instance if you have a python or java program or something like that we've got some standard roles and zool jobs that you can just use out of the box and that's also a great way to sort of contribute and share things with the community because the more folks for instance the more folks on on uh either on open dev or in sort of the wider zool world that are doing things with java then then um you know submit like if you if you create a job to do something interesting with java submit that back to our our standard library of zool jobs and share that with everybody else um so we see all kinds of interesting things come through for that uh java more and more things with rust uh stuff like that so it's a really sort of interesting place to to interact with with other developers that that share a similar technology so we do have one more question from the audience i know we're starting to run out of time but i think it's an interesting one so um this viewer is familiar with zool and garret code review and they think it rocks which is great um so if one does build a zool and garret for projects that's not open stack what are the some of the challenges that they may run into and maybe even some of the solutions that y'all found that can help mitigate those issues um yeah so i think one of the things that motivates us running open dev is that you know there there are challenges it's not to to build up an infrastructure to support large projects is is take some effort right and so the idea behind open dev is that if we can collaborate together then we can just spread that load out right you can help us out we'll help you and everyone kind of can be happy so that's part of the motivation behind open dev um if you were to run this say for an on open source project or if you've got different requirements i think just generally you know running a zool and for example i don't want to say it requires expertise you know you'll have to learn how zool works i think to to do it uh reasonably well just because as part of your your spin up time and and getting users familiar with it you're going to have questions about like why is this job doing this thing and you know what's going on in the system right so you need a little bit of familiarity with how zool works it's it's not just something you're going to deploy and then forget about it um and over time i think that gets easier because your users will become more familiar with the tool and and they'll be able to help themselves and each other on the the code review hosting side um i don't know garret in the in the before days um you know garret was a fairly difficult service to run um upgrades were very difficult i would say modern garret has improved tremendously and it's it's a lot more direct um again not really set it and forget it but you know you as long as you're staying on top of an upgrade cadence so that you're on maintained versions of the software you know running garret is is reasonably straightforward um which is just great because they've they've had to put that effort into to get there um and then it oh go ahead i was gonna i was gonna say it depends somewhat on your your tolerance for uh behavior changes and regressions to um i mean the the life-cycle management part of it is is really the most challenging part for for any systems administration but um but you know for for garret specifically like we we actually run automated upgrade tests of the garret software so that we can have confidence that when it comes time for us to upgrade our production garret that that's going to to work smoothly and you know we've hit uh issues regressions with with upgrades and and so on in the past that have have prompted us to and clark more recently to uh to go propose fixes to garret itself so that upgrades will actually work um we we currently have more extensive upgrade testing than of garret than the garret project does um there are some some resources uh that that that we provide um so if you wanted to get started with zool if you go to zool-ci.org slash start um that'll link you to sort of a tutorial for for how to get zool up and running um there's that that should walk you through the basics and and give you a handle on what's involved in a zool system that tutorial actually includes a garret why does it include a garret because garret is open source and we can include it you know we can make a docker compose file that you you can just docker compose up this whole system to start playing with it and that's only possible with garret so um so so that's a good way to kind of get started with all of this and see all of the moving parts that's um that's still pretty far from a production system it is like you could you could turn that into a production system but it's going to take a little bit of work um it's not intended to be a an entire system in a box that's kind of something that that that's going to be up to you um there's there's some amount of help you can get from the community in terms of mailing lists and irc or matrix rooms things like that um uh if if you're if you're sort of very motivated and you just need a little bit of technical help that can be a great uh a great thing um i'm not sure that i would expect a whole lot of hand holding for like how to get your enterprise system up and running from scratch um from from the community that might be a bit much um so uh you can of course because all of open devs um actual infrastructure for running itself is open source uh if you want to see how we run zool or how we run garret for the open dev uh collaboratory uh you can look at all of the um all of the ansible roles and playbooks and and container images and everything else that we do that's that's all right there so that's a great way to answer questions um and then there are also some uh commercial resources available um so putting on my uh my my business hat uh i have a company called acne gating which is um how i'm able to work and contribute uh to zool um and what we do is is help uh enterprises that want to run their own zools so uh i can help you get a zool up and running uh and maintain it and and scale it out and and that sort of thing uh likewise um uh garret forge is a good company out there that provides uh commercial support for garret as well you know what it means if i'm back so i know we're almost out of time but we do have one more audience question um so this is around the issue management system so song su uses launchpad and storyboard at the same time and wondering wondering which system y'all plan to use mainly in the future um so i think where we we are at today is that we are happy with flexibility here like we we aren't mandating or or trying to mandate or or being strict about what issue tracker project management type system you use um so for example i'm pretty sure some projects that we host are using github issues they they mirror their code to github and then they're using github issues um we still offer storyboard we're trying to to keep it alive and and some projects are still using launchpad i think you know all of those are are reasonable we would encourage people to continue using open source tools both launchpad and storyboard are open source um it is an area where i think we're open to feedback and and contribution you know if you can help out and improving say the storyboard situation or integration with launchpad or whatever we'd be interested to hear from you um i'm curious to know if jeremy or jim have any other thoughts on that we do currently manage some integration for both uh launchpad and storyboard um in so far as we've got some uh automatic hyperlinking and in garret uh web UI to to both of those depending on on uh how links are formatted um automatic updating of tasks in storyboard or of bug tasks in launchpad um depending on uh what sorts of commit footers people put in their commit messages uh for changes in garret um but uh i mean i'm i'm quite you i'm aware of projects that are using other defect trackers and task management systems external to those as well jira is another popular one that that some folks are relying on again not uh open source but um but popular nonetheless so um i think this is a good example of sort of what it means to be a community maintained open source infrastructure um to to put it very simply if uh if more people showed up and contributed code to maintaining and uh both the software and the deployment of storyboard then it would get better um but right now that's not a priority for anybody who happens to be contributing and so it's sort of stagnating and and in you know uh a sort of um low level maintenance mode uh you know we're the the folks who are are are kind of the the core open dev team you know if the service crashes or something we're going to try to to bring it back up we'll we'll try to keep the thing running it mostly does that on its own because that's the way software is written these days uh but um you know we're we're we're not tasked with adding new features or uh or doing migrations to improve performance or things like that those are all things that that anybody could could come in and contribute to so it really the the priorities of open dev are the priorities of the people that show up and and and contribute resources and time and i think that's honestly a great thing to end on because open source software isn't doesn't exist without people um so i also just want to thank y'all because y'all are the ones that have gotten it to where it is among other folks in the community who've contributed as well but um definitely encourage folks who are watching whether it's live or they're recording later to reach out figure out how you can help and to keep this software running because it is instrumental to existing open source projects and i assume more projects in the future um so we are at time so um i've wanted to thank y'all for joining us today i have a few things to wrap up um but any last words from our panelists today uh if you need contact info it's it's on open dev dot org uh so yeah we'd be happy to hear from you both if you want to use the the open dev systems or if you'd like to contribute or both have a great time with open dev all right thanks y'all and like jim mentioned earlier events over the past decade have been where a lot of these conversations have taken place so i wanted to kind of talk about a few of the upcoming events where you can meet these folks as well as other members of the global open and for community uh in september we will have our first regional open and for summit asia um this will be hosted in south korea uh everything is live on open info dev slash summit and we will be opening registration and sponsorship in the next month we'll also be having a open and for days europe road show where there will be multiple open and for days in europe in may and june where you can participate and learn more about some of the regional collaborations around open infrastructure and last but not least we do have a ptg coming up that will be virtual i believe members of the open dev team will be participating in some pre ptg but also some ptg um sessions so if you do have questions you do want to get involved this is a great place to find jim jeremy clark and other members of the open dev and open info community to start collaborating thank you again for joining us today and like jeremy said have a great time with open dev thanks everyone