 You know, like, when we walked into the hotel place, I should have been like, I'll just sit and chat. Not keep drinking? It's like very touristic. Yeah, I need, like, my body. It's just, it's like my... It's just... I haven't been eating, so I'm like... You gotta eat. Raw. That was what I mean. So, I'm gonna go sit behind it and eat some stuff, and stop feeling like I'm gonna die. I'm gonna see you. Beautiful. And I'm gonna go. Beautiful. Okay, good afternoon, everyone. Thank you for being here today. Let me welcome Clang Paran for today's talk. He is currently an IBM Cloud architect and sole developer. In today's talk, we'll talk about why we chose Zoom over other technologies, what we did with it exactly, how we can, how you can do the same and where we want to take you in the future. Over to you, Clang. All right, thank you very much. And thank you to Brno. It's a beautiful city. I've been enjoying experiencing actual snow. I'm actually from Los Angeles, so this is about as opposite as it can possibly get. So, I'm really excited to talk to you guys today about Zoom and about CI and about testing and clouds and a bunch of crazy stuff. I also want this to be a two-way conversation. So, feel free to interrupt me. I'll just raise your hand or shout at me, whatever it takes to get me to stop and notice you. But I'm happy to take questions while we're going. There will be some time at the end, so if you have a bigger question, you want to hold it. I promise that I will answer these questions because not everybody really will know, coming into this room, what I think these things are. And I actually want to know what you think some of these are too. So, I will tell you what I think CI is. I'll tell you what Zool is, because that can be really confusing, especially if you Googled it and you found the Netflix project. This is not about that. And I will also explain myself and why I love pirates so much. So, first, what's CI? We're not going to spend a lot of time on this because I think the industry has figured this one out, but I want to make sure we're all on the same page. It's basically your crash test dumps. So, it's testing everything together, making sure that changes that are submitted work together with all your dependencies at any given point in time. Sometimes it's pre-merge, sometimes it's post-merge, sometimes you don't merge things until they pass CI, sometimes you find out later that you broke the build. Let's show hands. Who's developed software here in the room? All right, great. And how many of you have what we would call working CI today? Good. That's great. And how many of you would like it to be better? Yes. Okay, there are some satisfied CI customers here. That's great. So, we wanted ours to be better in OpenStack, and so we created Zool when I say we as a community. I wasn't part of what created it, but I certainly depend on it every day. So, Zool is a pipeline manager. It was developed by the OpenStack infrastructure team. If you're not, who's familiar with OpenStack? Great. Good, because you're in the OpenStack track. Good job. You found us. OpenStack has 900 to 1,000 active developers at any given time. I think it's maybe over 100 active repositories. It's one of the largest single points of contribution of OpenSource software that's ever happened. It's really been fascinating to watch the journey that it's been on. In the beginning, there was just Jenkins, right? There was one little server, and then there were five, and there were 10, and they would just sort of run tests and hope for the best. And over time, it became clear that there was a lot of manual just messing with things to make that work. A lot of basically pulling levers, so if you were a core developer, you would be doing your code review and you'd say, this needs to go, and you would have to go wait for the test to run, and then you would say, okay, now hit the merge button. And we thought, wait, we write software. Why don't we have robots do that for us? So a pipeline manager does that, and we'll get a little bit deeper into why you might want one of those. But it was made to massively scale these projects that have a really high volume and a really wide geographically diverse team. So you don't want somebody in Shanghai to have to wait for the people in Paris to wake up and approve the patch and see that the test passed. If everything looks good and all of the eyes are dotted and the T's are crossed, you want that patch to go and be merged so that everybody has it, so that everyone can rebase on it so that we don't have collisions. So let's get the code well-tested in as fast as possible. So why do you need a pipeline manager? Why does everybody need one? Essentially, what we want to do is replace manual landing procedures with code. Zool is that. It's several thousand lines of Python code that has encoded the way that OpenStack works. So there's several different types of pipelines. The coolest one, which you see a diagram at the bottom, is the dependent pipeline manager, which basically says if in the process of normal code review, a core reviewer has approved, let's say, five patches, Zool will go ahead and queue them up, A, B, C, D, E, and it will test them all in parallel, which is really cool. And if A and B together pass, A and B merge together, but then C doesn't pass, C, D, and E are thrown out. Now, it's actually smart enough to go ahead and try to kick out C and try to test D and E on top of A and B again, and so that eventually it'll find the bad patch and it'll merge all the good ones, which is really important when you have 250 patches approved every day, which is what OpenStack has, all in one single integrated testing queue. Go ahead. That is on a single patch. Each one of these boxes is a patch. A patch that would just basically be a dependent set of boxes like this. These ones are just arbitrarily ordered this way. If they depend on each other, they're all kicked out together. So the patch set could be a box as well. But the important thing is that things that are independent of one another, they don't have a good actual relationship with each other, or they haven't, they can make express dependencies across repos in Zool, which is really cool. If they don't have any of those dependencies, then they can be reordered and retested in parallel. And that's one of my things, like Zool's coolest feature is that it actually tries to look into the future in parallel. So when these are all submitted, Zool says, okay, make a merge of just master plus A and test that, and make a merge of master plus A plus B and so on and so on, yes? Well, if it's in a single repo, then the patches will either be based on master or they'll be based on each other. So cross repo, you can make a little annotation in the commit message and say, the garret change I.B. that I need before I can actually work is here. And it'll actually combine those two in a single test run, which is really useful also for pre-merge, for even pre-review testing, because you can say, okay, there's a bug fix over here that will fix me, my patch. So I can say I depend on that one on this repo, and then your tests will go ahead and pass because it's like this repo. It is useful for that, but we're getting too... I don't have good... This needs pictures or whiteboards, right? As soon as I point three times, like I know I've gone off message. So we should talk more about that, and we will in Q&A. But what's really cool about this is if you have this as your queue, if you have a cloud that's big enough to run five tests all at once, which used to be a given, but then we started testing bare metal at once, and it's like, oh, do I have 15 bare metal servers available? Maybe not. It'll actually test them all in parallel, and if we got all greens, if A, B, C, D, and E all passed, in one testing cycle you would merge five patches that were all tested and are known to be independently working, which is just really crazy good for continuous delivery, which is why we at IBM really got excited about it, and I'll talk more about that when we go to piracy. But the important thing here is that the pipeline keeps moving, and Zool is the only thing I've worked with that's able to do this, and I think it's actually really magical that it ended up, it's like an actual open source innovation. It's not like we copied something, we didn't copy UNIX, right? This is something that came out of an open source project, and I'm sure there's some hidden commercial systems out there that do it, but everybody can have this right now, in theory. But you don't yet, and I'll tell you why later. So now you know what Zool is, and you know what CI is, so why pirates, other than the fact that they're just amazingly cool and fun to talk about. But where this actually comes from is a group of us joined IBM to build a big open stack cloud, and we got there, and what we found was we expected to find IBM's sort of staunch reputation as we know how to deploy things. They were going to tell us how they wanted to do it, and then we were going to tell them, okay, we can adjust that, and instead it was like nobody really knows how open stack works, we all know how to develop it, but how do you deploy it at scale? And so there were a lot of competing ideas, and ours was the most radical, and we were kind of just going off script completely, and every time we had a conversation it was like, not this way, that way. And we felt like we were the pirate radio station. Like we were the guys out from here with this message that we're trying to give to everyone, but there's no official word behind it. So we got some official approval to go ahead and go down the rabbit hole a little ways, and we just called it pirate cloud. So from there everything became R, and PyRats, and Python R. But it also made it give us a culture. We get this little shared culture, all of our little tools have pirate names like Hoist and MainSale, but what the idea for it was was continuous delivery, and with all of the modern thinking incorporated into that. So anybody know who Jez Humble is? Jez Humble is our sort of, not disciple, he's our prophet. We handed out copies of the books, and basically said we want to deploy continuously using all these principles that are going throughout the industry, but with OpenStack, and in a crazy way, doing it by not only our changes, but merging open source changes as fast as possible. So continuous delivery is, I'm a dev shop and I want to change my product fast, so I change it and I push it out, and I change it and I push it out. With OpenStack, it has 250 patches a day emerging. I want to push those out, which is kind of different in that we didn't make those changes, so all we can do is fall back on our testing. If our test coverage is good, then we can trust that whatever's coming in from the community isn't breaking us because we're testing what we think works. So that was our radical idea, and off we sailed into piracy. And we immediately found that we needed Zool on our side of the pipeline as well. So OpenStack has its Zool, which is very mature, and I'll actually show you guys a little bit later the real time running one, but also we actually had a downstream Zool. So the important thing, if you want to run a Zool, you have to also run a Garrett. So the reason we chose Zool is that we had a globally distributed team of over 70 developers, between different OpenStack projects. So there was a storage team, there's a networking team and a kernel team, but they all had to work on one integrated queue. This sounds exactly like OpenStack. And then we also want to be able to fully test large batches of commits, but not have to wait a long time. So our testing, our full integration test, end to end, actually deploying on real servers, you know, on eight real servers every time, it takes two hours. We can't do that in a pipeline if people hit five patches in a row. We can't wait 14 hours for the one that failed and then try to reorder them. We need them all to be tested in parallel. Zool does that. And also just so happened that our staff was pretty heavy in OpenStack infrastructure development team members. So we actually kind of already picked Zool. We kind of backwards justified it, to be honest. We didn't evaluate anything else. But there are trade-offs. The biggest one, Zool, only works with Garrett. Has anybody used Garrett? Awesome. Do you guys like it? Okay, yeah. Anybody operate a Garrett for your org? All right, buy this man a beer. It is not easy, right? It is a big piece of software and it's not universally popular. It is a great piece of software, but it's a big Java mess and you have to run one. At IBM, all of these 70 developers we had were standardized on GitHub Interpress. And at first, we tried to plug our Zool into that. It didn't work, so we just said, you have to learn a new tool, which is Garrett. The other thing is Zool is really hard to deploy. Even we had people on our team who had deployed Zools, not just for OpenStack in other organizations. It's just hard. It was designed organically around the Jenkins and the logging infrastructure and the clouds that OpenStack's development infrastructure had. So setting up a Zool today is really hard. And that is a trade-off. If you need this pipeline, it's worth it. But if you don't have a clear, like, yes, I absolutely need that tomorrow, once you get into how hard it is to deploy, you're gonna find that that's probably the biggest challenge. You can accept Garrett, but deploying it is hard and that is something that the Zool team is working on heavily right now in the V3 effort. And I would argue that there is this possibility with GitHub we have actually developed a module for GitHub for the Zool. I think I'm using your patches. You're gonna give away my surprise. It is totally possible now. When we had Pirate Cloud billing, that was six months ago and, you know, things moved fast. We started this contribution to upstream like a year ago. Yep. It works beautifully and we are using it. I'm gonna loop back to that. So I'm gonna loop back. So this is in the past. In the past, this is a trade-off. It only works with Garrett. So that's okay. So once we got down this road, we go to our highly trained, highly skilled IBM developers who are just, you know, they're not ready for this open stack world and we found some that we had just way too many knowledge gaps. We were trying to move fast. This is, you know, maybe June of last year and everybody we talked to had the same gaps. Nobody knew what Garrett was because they were all just finishing this DevOps transformation into GitHub. They actually didn't really know what DevOps was because who, did anybody know what DevOps? Don't answer. I'm just kidding. It is tough, right? I'm not always sure. But what DevOps meant to us was very different than what those developers had in their minds. And they actually weren't in a culture of code review. So anybody not using code review? Good. They were using it, but not in the way that open stack uses it, very, very intense. Like, there's this kind of joke, like if you get your patch through open stack without any negative ones, like your patch probably didn't matter that much. Like realistically everybody's got opinions, but there's also just rules and it's important to get things right. So there was this culture of sort of like wave the patches through and we had to reset that. And, you know, we're a distributed team. My team is all over the world. We're in Sydney and we have people all over the US. And so we were like just running into this on the phone. We can't explain ourselves. So Pirate School was born. A few members of our team basically put together curriculum and wanted to show like these are the important things. You need to learn the tools like Garrett and the code review. You need to learn what gating tests are. If you haven't used gated tests, it may not make much sense. Once you have them, I actually can't live without them now. I have a few projects still that don't gate and every time it merges, I cringe like, oh, did we break the build? It's wonderful to have a gated test where you know that master at least passed the test once. They also needed to learn just how to communicate with OpenStack. Because we were running continuously delivered OpenStack where we're trying to shove the community's patches through, if we needed to do a patch, it would make no sense to have our local patch sit and have to be rebased and merged and maintained. The idea was get it upstream and then the community bases their patches on top of it. There's no more of this skew, right? There's little local skew as possible. There's sometimes where you just have to. The cycle of OpenStack patches at minimum two weeks, realistically, because you've got to get two core viewers and you need to attach it maybe to an artifact, like a bug or a spec. We had to teach them how to do that, create the artifact, go create a patch, engage with the community, so get on the IRC channel and talk about your patch, go to the meeting and raise your patch's priority, all these things. And always try to fix upstream first. Even if you have to do it locally for urgency, make sure your patches is going back to upstream. And the biggest thing that we taught in Pirate School was that the people and the process matter way more than the tools. Garrett is a symptom of the process that we want, which is we want to be able to have clear communication signals. And the process that actually people have looked at your patch and that tests have run, that's more important than Garrett or Zool or anything else. So we did a road trip. I actually attended some of them, each member of our team. We rotated in technical experts and then we had sort of our standard trainers. So a couple of people went on all these trips. That would be Chris and Spencer. But we sent people to all the places we had teams and they did a little class. So what did all those teams love? They loved learning practical DevOps. We used Ansible. They had never used Ansible before. They loved it. They fell in love with the idea of being able to just sort of write deployment in a sequential manner. They had been used to Chef and Puppet and they didn't like that modeling language. They had to be able to just say do this, do that, do this, do that. They loved Garrett. They liked having reproducible environments. This was something that they, even though they had been trained on a version of DevOps, their version of DevOps didn't have the idea that your developers should work environments that are almost exactly like production, which is really critical. And once they got that, they realized that the CI system was actually even better. They just throw patches at the CI system. Once you learn that, you realize you have a cloud behind that CI system and your laptop becomes meaningless because you write a patch, submit it, write a patch, submit it, write a patch, submit it. And the CI system is running the tests all in parallel. And if you want to be able to debug it, you just throw something in your patch that says, don't merge me, I'm debugging. But you don't end up needing to like vagrant up unless you're at the early stages. So people really, really fell in love with that. And then because of that, that sort of reinforced writing tests. So people wrote more tests that stayed and lived longer. And that was a really positive thing. We ended up getting fewer bugs. We ended up not skewing off of upstream really at all. We had less than 500 lines of code skew at any given time, even though we were developing features in Nova, we were developing features in Cinder. And here's just a few shots of Pirate School. I think this is from Shanghai. That's Spencer and Jamie Linux there. He was our technical expert for that one. Here's the Shanghai team. Here's Beijing. We gave them pirate flags to hang in their office. So what came of this? We definitely had a massively improved developer velocity. People were able to get patches into OpenStack and get them out onto this cloud that we're deploying in sometimes in a matter of just a couple of days, even though the normal cycle for an OpenStack patch could be two weeks. Because a lot of our things that we were doing were bug fixes, they get pushed through faster and they're more obvious. So we had really high developer velocity. And that also goes for our own deployment tooling as well. When we weren't going upstream, when we were doing our own deployment tooling, which wasn't upstream, we were getting patches in in just several hours. Basically, people were getting really good at changing the system fast and then we were able to deploy. Our ops team was deploying three times a week, which was not as fast as we wanted to go, but that was as fast as the ops team was willing to go. We actually wanted to just deploy every day and give everybody the button. We got a bunch of feedback that people felt that they were able to communicate with teams they didn't even know before just through patches. So we had Slack, right? We had all these communication methods and those were great. But the team in Beijing had never met the team in Raleigh, right? They never really interacted, even though they're working on the kernel team and once the networking team, they should interact. They just hadn't really found a way to talk. You know, you have native Chinese speakers, native English speakers. Sometimes it can be difficult even through text. But through code, having the code review process and the inline commenting, they had never experienced that before and they felt that that was a huge win. They're now actually working on other projects using GitHub Enterprise, which has the same aspect, but they learned it through this and they didn't know that that was even possible. So that was a huge one. And a big thing we also got was that people learned more about what open source was. They thought it was a different license than the one you sell people. There were a lot of teams in IBM that have, you know, they interned at IBM and got a job at IBM and have only worked on software at IBM. And they've been using Linux, they've been using Ansible, Python, everything, but hadn't actually participated in open source communities before. And just that one step of getting a patch through into OpenStack and seeing that there were people out there who wanted to talk to them that weren't IBMers and that wanted to continue that relationship has changed things. And there are more open source projects coming out of those teams now and they're wanting to open source things more often because they're finding, like many of us have found in open source, that it's not just the way that you work, it's kind of a way to live. It's this cultural sort of happiness that comes from it, and it does matter to people. So before I move on, I will say the project, was actually put on pause and we're not... We don't have that cloud up right now. Basically, the capital investment was moved to another part of IBM as large corporations will do. So the end result was that we taught those teams how to work better and we learned how to play OpenStack, which is still happening, but Pirate Cloud is actually shut down. So I like to say, our Pirates all went and worked for the state. They're not Pirates anymore. But they still remember how to be Pirates. So at this point, you might be wondering how you could do that. Well, if you're a puppet shop and you want to have Zool, the OpenStack info team made a thing called Puppet OpenStack CI that will get a Jenkins, a Zool 2.0, I should say. Paul, who's sitting right here, is going to talk about Zool in more depth and the Zool 2.5 Ansible revolution that happened recently. But Puppet OpenStack CI will set you up an older one that's more simple. It's made for people to do third-party CI. So if you want to do something behind your firewall and submit results to OpenStack patches, that's what it's made for. But we actually successfully used it to just set up a Zool and then pointed at our own Garrett. So you can do that. The thing is it doesn't install Garrett, so you have to deal with that. And you also need an OpenStack cloud, which might be the biggest hurdle. So if you don't have access to an OpenStack cloud today and you want to have CI like I described, there are several options. You can allocate a static number of nodes to a Zool that is complicated and it's static. So you're going to have to deal with cleaning the nodes up and all the things that clouds do well. If you want to install OpenStack, there's a whole bunch of ways. There are also actually many, many OpenStack public clouds that are reasonably priced and are waiting for your elastic CI load. And I've included a link here. I know it's hard to see. If you want my slides, my email address will be at the end and you can ask me for them. I'll gladly give them to you. And there are also managed OpenStacks. If you have a really large CI need, there are some companies that will, including my own, that will build you and host you an OpenStack cloud. So there are options for getting an OpenStack cloud. Yes? Do you know if there are any trends that will be coveted from OpenStack? There aren't, but I will say it wouldn't be that hard. So the thing that couples to OpenStack is called node pool. And it basically looks at what Zool's doing and makes the cloud make that happen. So there are some hurdles to that. You need to be able to build images for whatever cloud you would be going to and you would need to write a node pool for Amazon or for Google Cloud. But it's certainly possible. It's just that everybody who works on Zool also works on OpenStack clouds. So we all do have OpenStack clouds. But if you don't and you wanted to, there's nothing technically in your way that's really daunting. It's just basically writing a driver. Yes? What's up? We're running 2.0 for Pirate Cloud. So we still had Jenkins in the mix. I actually have another option if you want to do that. This is not an announcement because this isn't actually available. But my team has now moved on to making something called Bonnie CI. That's named in honor of Ann Bonny, the famous pirate. Of course, everything is pirates. But it's basically a continuous integration service we're offering to developers of open source projects on GitHub. Right now, when I say offering, we've invited two. We have very low capacity and we're brand new. And what we have is basically as little 2.5, which we're not supposed to do. So we're not doing any development or adding features to it. Other than we have taken your GitHub patches and we've actually stacked on a few, I think Jonathan Harker has been working with you guys to make Zool 2.5 talk to GitHub. And we can do pre-approval checks. So if you submit a pull request on your project that is hooked up to this, it will run tests that you have defined in Zool. It can also gate. So it can actually, when you say approve this and you're in the right group, it will go ahead and run your tests and then merge your code. So that right now is basically just beyond proof of concept. On the roadmap, we will have an automated sign up. We'll have massive parallelism. Right now we have a quota of eight nodes on a borrowed cloud so we can't really run any tests. And we'll have all these other things which are actually part of Zool V3, which I think Paul will talk a little more about, right, in your talk tomorrow. So go to Paul's talk talk more about Zool V3. But the biggest one there is the in repo job definition. Right now Zool, you have a config file for Zool. What we want to have is in your repo, you say run these tests on these triggers so that you can maintain it yourself rather than having a centralized control. And this is just a few things that are going on in Zool V3. It's a massive refactor. It's been going on for a while. We're hoping it's done by March, but it is moving forward. So I promised to answer these. What is CI? It's necessary. What's Zool? It's awesome. And pirates are everywhere. So thank you so much. Thank you. Questions now? I got a question in the back. I'm going to restate the question because I just remembered I was supposed to restate these questions. If you have the Jenkins with 150 or so Jenkins plugins and you're using it today and you want to have Zool manage the pipeline, I think that's the question. How could you do that? Could you replace Jenkins with Zool entirely? Yes. In fact, OpenStack's Infra Team has done that. And that is what is called Zool 2.5. They basically just rip Jenkins out and put Ansible in. And the only thing that Ansible couldn't do that Jenkins did was give them a web UI for watching jobs run. And they basically just made a little web socket thing that is running today. And I can show you in a second here I can all tag to the status page. So the OpenStack Infra Team's Zool doesn't run Jenkins. And Ansible gives you basically everything most of the Jenkins plugins that Jenkins JobBuilder supports work in Ansible. So yes, you can. Would you want to? I'm not sure. But you can. Can you keep your own existing Jenkins with 150 plugins and integrate stuff out of Zool? Yes, you can. But that's probably not going to be true. Just restating the question, can you keep your Jenkins and Renzool? The Zool v3 is a zero Jenkins refactor. And realistically, once you are free from Jenkins, I love Jenkins, by the way, respect to the developers and the community. It's an important project. You've outgrown it. Once you're free from Jenkins, you probably won't ever want to see it again. So it can scale, but doing so is not something that I would ever want to have to do again after you've inputted it. Can you run Java with this like OpenStack? Because Zool without Jenkins is the only one on OpenStack that you can openStack. That's true for Zool completely. Zool doesn't really work without OpenStack. Do you see the same version by for instance OpenZool and let Zool and Docker run the same way? So are there hard version dependencies between the components of a Zool system, like NodePool, Jenkins, Zool? So, yes. Right now, NodePool and Zool are still kind of OpenStack-infra's world. So you deal with what is in those repositories and they're not making many guarantees to users outside. There's plenty of companies that have made that work, but NodePool has advanced now to actually be backed by ZooKeeper, whereas before it was backed by Gehrman, ZeroMQ, and MySQL. And there's an ongoing effort to sort of move it forward for Zool v3 that leaves the 2.0 and 2.5 users behind. But that's kind of intentional because the idea was like, hey, we never made you any guarantees anyway, so here's the tag. Good luck. So I would say it's the kind of thing if you want to adopt today, you're going to need to be involved with Zool development directly. If you come after v3 is ready and it's being pushed out, there's going to be a willingness to make a lot more guarantees. Basically, v3's being written so that Zool becomes more useful to not OpenStack. So it will get easier. Right now, it's a wild and wooly place. Yes? So the question is, just to restate, is the code review approval process and merging handled by Garrett, which is yes, and Zool does basically just vote. However, Zool v3 should actually do the merging itself. So there's a problem, which is that right now the way it works is Zool does emerge in the way that it thinks Garrett will do it, because Git has several different ways to do merges, and then when everything works, it pushes the submit button via API on Garrett. And sometimes Garrett wouldn't work that well except Zool's gotten good at not making that mistake, I think. So it only uses the merge types that Garrett can do. But that actually breaks things. Some people will complain, like, hey, no, my patch should merge just fine. Why does Zool think it won't? You just need to use this strategy that Git would do normally. So eventually, Zool will push the patches into the repo. Right now, it hits submit. Yes? All tests are stable and rock solid. All tests are stable and rock solid. Are you measuring the force also? The easy answer to the question is, are there problems with tests and does that work in reality? That's development's problem, man. The reality is that OpenStack has a bunch of tools to try and find misbehaving like racy tests that only fail 10% of the time, things like that. It's an ongoing battle. OpenStack's a massive distributed system and testing it in isolation on clouds that are massively distributed and change and the red car beats the blue car half the time. It's just a challenge. And Zool doesn't fix that for you. So all integration testing runs into this at some point. Does anybody want to see the running Zool, by the way? I can really quick pop it up. Oh, yeah. We got it right there. So I just like to show this. Now, this is actually the most busy time ever. See, I see policies, and he's like, what's going on? This is release time. OpenStack, Okata release is about to release. There are some patches that need to land before that happens. But what you have here are three of the pipelines. Check is submitted patches that aren't approved. So those are pipelines of one, most of them because none of them have any dependencies. So they all are just running alone. But then this nice long pipeline right here is the gate. And it usually isn't that long, but right now we have a lot of approved patches. So no one has been in the gate pipeline for 10 hours and 43 minutes. If you look, these are all the tests. These two have already passed. These ones are running if I click through here. Oh, that's right. It doesn't have a web interface yet. You can tell that to a port and see that tests output. Or once it's finished, if I click here, I get a log. This one is still using the Jenkins? No, this is not Jenkins. So this is actually running test via Ansible, basically. And so that's why there's no Jenkins front end anymore. And that log file hasn't been copied yet. So eventually you'll see there's a... You can't see the URL at the bottom, but that log has been created, but it hasn't been pushed to the publicly visible log serving server yet. But you can see what's happening here is this is basically the front of the patch. This one is testing this one and that one. Right, and so on and so on. And then you can see way down here this one failed. So it got kicked out. It failed a test. But tests continue. So it basically... If all these pass, it'll merge to here. And then there's a new gate head right here, basically. That's testing all these behind there. And then gray ones are cute. So we're actually out of notes. Because if you look down here at the bottom... Gosh, there's a lot today. Yeah, whoa. How far does this go? This is the number of test nodes that are running right now. Looks like... Could go higher. Maybe there's a problem with the cloud and why it's queuing things. But it sticks around 300, 400, 500 at all times running nodes. And this is across several clouds. The OSIC, which is like an open stack until OVH, right? I don't even know how many there are now. One way for your company to help open stack, by the way, is to donate space on your public cloud. So if you run a public cloud or know somebody who does based on open stack, it helps to donate nodes. And then you can see the number of jobs launched per hour. I mean, this is a massively scalable thing. Most places aren't this massive, but we found that it worked well. Scaled down to 70 from 900. So I think it probably could even be useful for teams of 5 or 6. Just the idea of not contending on each other and not waiting to pull the lever is extremely powerful. You have another question. Is there some logic in the scheduling? It's pretty basic. And it's not my area of expertise. So a lot of the magic, too, is actually a node pool, not in Zool. So Zool just kind of expresses demand. I want to run this job. And node pool looks at that demand and spends up nodes with the right amount of money. So I think it's a good point and spends up nodes with the right image to test that. So node pool has some magic at keeping like min and max ready nodes. No, I'm talking to you about which patch does the person need to order. No, it's just approval order. They come in an event. No, there's nothing like that. So the way that the order is determined is there's an event stream from Garrett and they get queued as the events dictate. So some events will lead to many, many things getting enqueued because the end of a patch, everything else is approved and the last one gets approved. They all get enqueued in the right order. But yeah, there's no magic like that. That would be cool. Yes? Like provisioning and deleting of instances, that's actually node pool. You can run Zool without node pool if you just have a static set of machines. But you have to handle the cleanup and setup of the machines yourself. But there are some third party places that do that. Absolutely, they've got two of their new model of switch and a few servers hooked up to those switches physically. They don't really have a cloud there. They just want to test their neutron driver with that switch and so they statically have these nodes that are just available to Zool. So basically it's just running the launcher on those nodes and it just pops out jobs and then when they're done it keeps reusing the same node. I have not. What is linchpin? I will take a look at it. That's interesting. Neat. Learned something new every day. We have more time but we can also go early. Yes? Have we seen performance issues? On the one that we ran for IBM, no. But we never pushed it anywhere near where OpenStack has pushed it. Zool's a single process. I think it uses a couple of threads but it's Zool itself is it's a single process for the scheduler and then there are multiple processes doing merging. So that scales out naturally. You basically can spin up more mergers and that's the things that just pull the patches together which is actually somewhat computationally expensive. But no, I don't think Zool has been the bottleneck. The clouds themselves are definitely the problem. Sometimes clouds go slow, sometimes clouds go away. And that's if there's one thing that leads to bad days and infrareds, it's a cloud storm. But Zool itself is not doing anything really hard to scale. Yeah? You can imagine how many projects it is. Yes? You just reminded me so you were saying you have seen problems with cross-repro dependencies making Zool slow. I totally forgot about we absolutely saw the same problem. So we were taking batches of patches from OpenStack and basically doing the one big merge commit. And in order to do that and make sure that they only all merge when they all tested we made one big monster depends on all the other changes commit. And yeah we would see five minutes for Zool to resolve that in Garrett because essentially Zool would see these 25 depends-ons and it would have to go query Garrett and ask for all the commit messages for that change. And then that would have dependencies and it would have to dive down a massive dependency tree and there was no good caching between Zool and Garrett. So it would just sit there and basically just talk to Garrett for five minutes before saying so yeah we did see that when I totally forgot it that the answer was just wait five minutes wasn't all that big of a deal. So one more question anybody alright great well thank you so much for your time this has been super interesting for me I hope it was interesting for you so what is 0 or 0 HDMI it's possible yeah I mean I'll try VGA but HDMI it might be better I think I think the reason I really prefer V3 just trying to give you a full replacement it's not just like 2.5 was kind of panicked trying to get rid of public victims so it doesn't necessarily do all the things that I think it does but 3 is like hey there's users that really need that web interface I'm just coloring that's your use case well for us we want to be able to basically give you a similar experience but with gating hey worse is it the same is it the same okay sweet yeah motivation and all of us just trained to build to each other we don't have to do that now we should do that we have quite a lot of protection preferences but you have a solution if it's going to be able to do that we're doing the body as well and we have a little get-of-integration on the CI that you can install and it will essentially stream with you if you put it in hey anybody can go around our goal is to we'll have 10 minutes for talk, 15 minutes for Q&A we will show you the time with 10 minutes for man 20 please repeat each question is the audio being recorded this is the mic this is the mic this is the mic there was no activity stream, so my view is like okay, as you said, here's the view, this is like the last question. It's pretty much the view. That's one of the reasons we're here. And I guess the view is complete, so you're just completely represent by if you think you can actually use these proposals on the original, does he have more views? Well, so, Heat 3 doesn't even need to get hip-stop, but we are working on it. Like, Heat 3 is still in such a flange, so we're not even trying to rebase to get up matches on it, so we are influencing Heat 3, so it'll be easier to write that matches. The GitHub integration, it's like this type of hashtag, it has to dive in, it's a little change thing. Heat 3 will have, it basically has a driver in your face, so you can just write an out-of-treat or a condo, that will just import, and you have to have a key, and both of you will be able to reuse the old one, as much as possible. But one thing we found, we're going to have just added a whole bunch of EPI calls for the review system, so that for specifically, now you can get a webhook on approval, and you can go query it based on that webhook and get cool proof, and basically the same as trigger models have here, so we actually are designing one that works pretty close to the way here.