 Good morning. Good afternoon. Good evening wherever you're handling from welcome to another edition of cloud tech Thursdays here on OpenShift TV I am Chris short executive producer of OpenShift TV, and I'm joined by my fellow of Red Hatter's Josh Mike and Amy Mike. I'll hand it over to you for the starting of introductions Yep. Hi everyone. I'm Mike. I'm the stuff community architect for Red Hat. I'm Josh. I work on cloud-native projects here at Red Hat, mainly community support and facilitation Amy I'm the open-stack community person for Red Hat and you are an open-stack person. You know me as spots So I'm joined here with or rather we're all joined here with Jim Blair Who is the project lead for the Zule project gating system? And it's a founding member of the Open Dev Collaboratory team as a sysadmin and hacker He gets out of it gets that gets to write elegant code and then try to make that work in the real world You start ACME gating to provide Zule support and services. Zule is a project that is a gating system That helps you stop Stop merging broken code make sure that every change is fully tested and for merging and test those changes amongst Multiple services or microservices rather. So I'm gonna let the expert here Jim. Go ahead and take it away Hi it's good to be here with y'all and Here if why don't I share my screen real quick if that's okay So yeah, as as Mike said I'm one of the original authors of Zule and the current project lead and and and Zule is a project gating system and I Guess the first and obvious question is what is a project gating system? Is it a CI system? Is it a CD system? And to that I would answer. Yes, it is But I see sort of the the the the world of CI and CD as a continuum and The first CI systems if you look at them If you remember what CI stands for continuous integration, right? the first CI systems were you would merge a change to your Java project and you would run the build after doing that and see if the build broke and Basically, this was this was post merge testing. This is testing that was focused on the past Things have moved a bit since then and we have pre-merge testing now So you you write a change and SCI system tests that the change that you wrote works Before you merge it, which is great because you can find out that you that You can find out that it's broken before you've broken it for everybody else This kind of testing is focused on the present So I think gating is the continuation of this and it's a future-looking system. It's about saying not only You know does the change that you just wrote work? It's does the change that you just wrote work when it's When it's going to be merged with the tree which might have moved on since you wrote it other people might have merged changes since then Does it work with all of the other changes that are in flight? you know your colleagues are writing changes at the same time you are and You may or may not be talking to each other when those all get sequenced up to merge is is the thing still going to work and Does your project have dependencies that you know, is it going to work when this dependency changes? so That's that's how I see The purpose of a project gating system is its CI and CD that is focused on the future so a Quick little bit about Zoolz development history. How did we get here? How did we end up thinking that we needed to? make Yeah, but you go back on the the video is actually shown in the in the stream. Oh Yeah, let's see. How do I Can I share? I'm not actually sure if I can share the video and the screen at the same time. It should be able to If I share maybe do two connections Shouldn't have to I have to Yeah, it's normal for zoom if you don't specifically turn the video off. So you turn the video off So don't don't just don't turn the video. Oh, okay. So all right. Yeah when I shared it turned the video Off, but then I can turn it back on. It looks like cool. Okay. Okay. So we've got we've got a slide and we've got me now Yep Okay, and the rest of us actually great Cool So yeah, any questions so far while we're There is a question, but I want you to get past the Past the intro first Okay, people have sort of context Okay, so how did we end up thinking that we needed to a system like this, right? Why why wasn't Why couldn't we just Check to see if our build was broken Every morning and so Zool came out of the open stack project it We started working on Zool fairly early on in open stack development and the reason is is You know, if you look at this slide here, I've got a lot of logos on here None of those are actually the open stack logo and that's because when you know, we present open stack Sort of to the world we talk about it as as a single product, but really underneath it's a bunch of Independent projects semi independent projects that are trying to work collaboratively to build a cloud system And so each of these little logos here is representing one of those projects and When one logo in particular is is I think near and dear to one of our panelists today So these I Guess the problem that we were presented with is how do we How do we develop all of these projects together and make sure that? That we can you know every six months release something called open stack where they all work together and And so that's kind of why we ended up with developing this system that was focused on Testing changes before they merge and testing them with other projects dependencies and other projects that sort of thing so that's that's sort of where Where Zool came from and why we we started to go down this road in the first place a Key aspect of how Zool does its work is something called speculative execution We we didn't invent that speculative execution is a pretty important concept in computer science and computer engineering and the idea is that instead of just You know It's basically about Saying what if we What's going to happen in the future if these things happen right so in in and processors You've got branch prediction where processors try to anticipate what's going what are the next few instructions going to be? and And if they're right then they save time if they're wrong then they discard their results And they start again and so that's actually something that that Zool does is it it anticipates that changes are going to merge to other projects and Incorporates them into its testing So we've got two little graphics here on the on the left It's kind of illustrating how you can use this with dependencies So you could have a front-end UI change that depends on a back-end server change that depends on Changes underlying libraries And and you can build up the sort of speculative chain of what is going to happen when all those changes merge and then on the right is is a illustration of How Zool sequences those changes in its testing right? It's it takes the latest merge change and then it it builds on top of that it says I need to test this change that somebody wants to merge and I need to test these three other changes that depend on it Right and if one of those fails it pulls it out of the sequence and keeps going So that's that's a pretty Pretty central concept to how Zool does its work And this my how do you track? How do you track which changes depend on which other changes? Yeah, so that's a that's a good question There's a couple of ways that that that we do that one is There's the there's of course get dependencies, right? So you actually If you're in GitHub, you know, you'll you'll see this as a series of commits in a in a pull request, right? If you're in Garrett Then each commit gets its own change, but it still recognizes the the git commit hierarchy and The git dependencies are Immutable right like there you can't you can't pull one of those out you you you have to merge all of those changes But that is part of the dependency sequence another is the implicit Sequence of changes that happen when when people approve them for merging, right? So I Guess an important part of what I think belongs in a gating system that I may have glossed over is that the gating system Is doing the merging, right? So there's no there's no person pressing the the the you know merge this PR button on github the gating system is responsible for doing that because it's also responsible for for Determining, you know at the moment of merge if the if the change passes tests or not so when when people Yeah, if they're in Github or Garrett or whatever and they're they're reviewing proposed changes and they say yes I want this thing to merge they do something to indicate to to Zool that they want it to merge So in in Garrett you leave actually in both you just you leave some kind of approval on the on the change or the PR and And Zool kind of takes it from there And so you can you can imagine that as you're going through your day and you're you're saying oh, I want this to merge This one needs more work go and make a change to it and then Maybe you find a couple more changes that are ready to merge and you approve them Zool is is is going to start start sequencing those as you approve them So there's this implicit ordering based on on that and That's actually kind of an illustration of what might be going on here on this slide, right? So perhaps somebody approved a nova change to Nova changes in a row that Nova is an open-stack component, right? There's another open-stack component called sender Somebody might somebody else might have approved that But these projects are both open-stack projects they need to be tested together We need to make sure that that Nova change isn't going to break that sender change and and that sort of thing So so this implied Dependency sequence is is kind of the next thing and that's that's the that's how you end up with these sequences of changes Looking at Zool here the third thing is I think the most powerful way of sequencing dependencies and And it's interesting because it isn't it's actually not something we started with in Zool It was it was really an emergent behavior but but now we see it as is absolutely critical to it and that is explicit dependencies between changes and So that's the idea where I guess going back to this thing where you you have a front-end change and you actually you say As part of the commit message This change depends on the back-end change and so not only do your reviewers see that when they're looking at the commit message but Zool sees that and it incorporates it into its Sequencing system and if If you want to indulge me for a minute or so I actually have a sort of Walkthrough for what that looks like if if folks would like a little visual Illustration of that Yeah, I think so. I don't know if it's would it be time for the first question. I think it might might be yeah So somebody actually asked from YouTube stream They're curious If you'd be willing to compare Zool and GitLab CI Depending on how much you know about GitLab CI Yeah, so there it's GitLab CI is is pretty powerful You know if you look at the merge trains Feature in GitLab CI that that certainly has some You know it kind of echoes somewhat of what I'm talking about here about their relationship between changes and particularly the sequencing aspect I think one of the There there are two things about Zool that I think make it really kind of a a standout System and and one is it works with a lot of different code review systems And it works in a lot of different cloud environments. So It's It's actually great for bridging Across projects that might be in Garrett or GitHub Or pack your or Or GitLab all of these we've got support for all of these in in Zool, we're still working on the GitLab driver but that's you know work in progress and and as well It supports a lot of different cloud environments. So Zools execution is it can be you know, not only in You can run it on a container system So you can you can run Zool jobs in Kubernetes or in OpenShift You can also run them on VMs in you know any number of public clouds Or private OpenStack clouds and that sort of thing. So there's really a breadth of what you can use Zool for as compared to some of these other systems and And I think this sort of explicit dependency thing that we're sort of dancing around right now Is is another really key unique feature? awesome Does that answer your question about a seven second delay? Yeah, well, we'll pick him up later on or we'll pick them up later on and If they have follow-up questions Yeah So, yeah, why don't I talk about that a little bit more? so I've I've contrived this example and I'm I've intentionally made it not OpenStack specific. Hopefully this is something that that everybody here can can relate to So I've got these three logos here because I apparently like slides with logos But you know with these logos, how can you how can you not love them? so over on the left we have The the logo for sender the OpenStack sender project. That's OpenStack's block storage system and in the middle we've got go for cloud and the right of course Kubernetes and and the example that I'd like to to walk you through here is this idea that that we want to add a new feature and Expose it through this entire stack. So we're Let's let's add Support for secure delete in in sender and you know Don't focus too much on the details here because I I This is this is a synthetic example and and I'm just trying to create something that sounds plausible Whether or not we could actually do this or need to do this, I don't know but to me this seems plausible, right? So we want to expose the idea of of you know Doing secure delete when block devices are removed From from sender and we want to make that happen in Kubernetes when say you've got a persistent volume That gets deleted you want you want that PVC to To cause the the underlying sender API to do this secure delete, right? So this is a new feature that we have to move all the way through that that entire stack so we we'd start by uploading a change or writing a change to sender and Uploading it to open devs Garrett Open stack uses a Garrett code review system that's hosted by open dev for its changes. So this is what that looks like This is basically a change that's in review We've got the commit message here on the screen where we we were adding the API option to to perform secure deletes, right? So the next thing what we need to do is add support for using that API to go for cloud, right? So we open a PR against go for cloud and And that lets the the the world of go access this sender API and then finally in Kubernetes we update to the latest go for cloud and and have it Use that secure delete option when sender when volumes are deleted So at this point, we've got three changes out there. None of them have merged yet And you know, we can test all of them independently But there's there's there's no relationship to each other. We don't The the people reviewing the Kubernetes change. They're kind of taking it on faith at this point that everything else works You know, they see this PR show up and say hey We totally added this API to go for cloud We're pretty sure it's gonna work You know, please merge our change. So Maybe maybe we want to use Zool to to show that this entire sequence can Can actually be tested together. So so we go back to our Kubernetes change and we add this depends on footer to to the change and and you can see that It says depends on and then it's got the URL for the go for cloud change. And so this is how you do that explicit dependency Specification in Zool. We just made it URLs so that So that you can actually support cross We call them cross source dependencies where where you have Changes in GitHub depending on changes in Garrett, right? So so anyway, this is the Kubernetes change saying I depend on the go for cloud change the go for cloud change depends on this change to sender and if we go back to So so at that point, oh, sorry So I just want to highlight again the depends on here That's it's it depends on to a different system. So we're not even saying we depend on another GitHub change We're actually depending on changing Garrett. That's okay Zool knows about GitHub It knows about Garrett it can stack those three changes up and this starts to look like that that That diagram earlier. We've got these three changes and the little lines Show the connection in Zool This is by the way, this is what the Zool status interface looks like it actually shows you these this relationship So so that's how you stack those up. We can extend this one step further though We This is a brand new feature and perhaps our Zool doesn't actually have a job that exercises it yet and so we can make we can extend this idea of speculative execution where we're saying I Zool I want you to run my tests with these three other changes. We can also say to Zool. Hey, I want you to run a job that you've never seen before So Zool's configuration is in YAML files, of course And so this is what a Zool job definition looks like So if we wanted to add a job that says, you know, exercise the sender secure delete option with Kubernetes We can call that job sender Kubernetes secure delete we can parent it on a job called say Dev Stack Kubernetes Dev Stack is open stacks environment for Performing development, right? So it's it's kind of like mini cube for open stack So you can imagine that there's a job somewhere in the system that sets up a dev stack for for testing open stack and sets up a mini cube for testing Kubernetes and maybe does both of those things for testing the two systems together Zool has a really powerful Inheritance system for its jobs. So we can actually take you know You can spend a lot of effort setting up a job that sets up your infrastructure and then make a bunch of little jobs that make small tweaks to that and so that's sort of what we're doing here we're saying let's take that Dev Stack Kubernetes job and Just Do just add on to that a little bit to test our secure delete system So underneath his job definition We've got the the project definition where we say I want you to run all the unit tests I want you to run the functional test and I also want you to run this sender Kubernetes secure delete So if we wrap all of that up into a change and upload it to To Garrett then in Zool Now we've got a sequence of four changes and the last one runs that extra job that we just defined and When that's done it'll report back to Garrett and you can actually see on the bottom of the screen here the little Sender Kubernetes secure delete job succeeded so so that's You know it's kind of a Whirlwind walkthrough and it's it's high-level, but that's how you can set up these complicated series of of changes across different projects and and See results see functionality changing in your dependencies before any of these changes merge So, you know in this example nobody had to actually merge a change to see this job run to completion and exercise our new feature and that's that's a really powerful tool and we use it all the time to test changes between open stack projects and between dependencies for You know external dependencies as well Now Jim at this point if we work flowed this it would then test this new Merge and patches against all of open stack if it was an internal open stack type deal Yeah, and and you know it's one of those things of Like what what's the test area that you want to cover right? We can They're sort of in in open stack. There's this core set of projects where we we kind of think that Keystone and Nova and sender and some other things that kind of they they're all really tightly Interrelated and we need to make sure that they never break each other and so we're always testing those projects against each other As open stack has grown over the years and as its interactions with other communities have have increased The surface area has has gotten huge right and so You know early on we We we were testing as much as we possibly could all the time that kind of grows exponentially And it's it's ultimately not sustainable to test everything and so we've gotten better about saying You know these things we always need to test This thing we only need to test some of the time You know when when this part of the system changes that sort of thing so So yeah, if we if we you know added the workflow vote on this test to approve it We would certainly be Merging This change only if it pasted past the new test that we added We might not want to always run that change though Sorry, we might not always want to run that test and so we then might make that contingent on well, you know Let's only run this job if we're touching the part of sender That's relevant to secure deletes and that sort of thing so and the other thing to know about The workflow here is that because of this dependency Chain that we've made This change isn't going to merge until the ones ahead of it have merged So since we depend on that Kubernetes pull request We're not going to merge this until that one passes and that one's not going to merge until the gopher cloud passes And the gopher cloud one isn't going to merge until the sender one passes. So Sorry not passes but merges right so you we can maintain that strict sequencing to make sure that not only can we prove out that this Whole new feature works in advance But we can make sure that we don't merge any changes that depend on it until You know the the dependency changes have merged as well So can you accidentally get a circular dependency? Has that happened in the past? Circular dependencies are a really interesting Thing we actually so we with Zool we started off with a pretty strong stance against circular dependencies because we wanted to make sure that the the Continuous deployment of an open stack cluster always worked right and so We wanted you to be able to upgrade your your nova nodes and then Upgrade or you know, maybe upgrade Keystone first and then upgrade Nova to use a new Keystone feature and so forth, right? So we kind of We said that we couldn't require cloud operators to Shut down their entire cloud and then bring everything up again. We had that we had to make Stepwise upgrades and so we mirrored that in Zool We made Zool enforce that by saying you can never create a circular dependency it always has to be a What we call a DAG, right? It's it's just like get it's a directed acyclic graph There it always has to be a strict strict tree structure And and you can't have the you can't have a cycle where one thing depends on another That worked out really great for open stack, but this is also kind of one of those it's an illustration of how we as a community have evolved over the years as More people who aren't open stack started using Zool. They said we don't have that restriction in fact Our software works a lot better If we don't do that, you know, we're doing things like So some of the other users now are are in like the In the manufacturing space. We have automobile manufacturers Embedded systems manufacturers stuff like that and they're like we don't have that upgrade cycle You know, if we're upgrading the firmware on a car, we kind of do the whole thing all at once We don't we don't we don't just upgrade the the entertainment system and then do a stepwise upgrade of the You know of the transmission control system, right? Like that doesn't make any sense. So so They said circular debensis are really important for us because that's that is how we do our development And so we've actually added the the ability to do circular debensis and they work kind of like how you'd expect or how you would As you suggested accidentally do right where you you know You stick this depends on header and change a that points to be and then do the same thing and be that points to a If you do that Zool will recognize that that is a circular dependency And then it actually changes its algorithm a little bit. It says I'm I'm not going to do a strict ordering between these I'm going to recognize that all of the Participants in the cycle need to be tested together And then merged or not merged as a unit So it's still kind of that the graph that will make still kind of looks like it normally does except that there's You know potentially a Grouping of changes that behave That move together and and all of the changes in that group have all of the Git changes checked out for them at the same time as well. So I think that for me to kind of sum up this line of Thought Zool Zool is a tool that lets you ask what if right? It's about saying What if I merge this change what if what if These two changes that depend on each other merge What if this other dependency merges like what what is what's going to happen in that case? And and I think that's where the the power comes from Just You know, I mentioned some of these other we've been talking about open stack and Kubernetes a lot, right? So these are these are some of the other Zool users That are out there you can actually on our website a lot of these users have Stories Posted there about how they're using Zool. So if you want to Read about them open lab is is an interesting one here because they they perform Sort of cross-community testing they're really at the heart of what I've been talking about They do cross-community testing between open stack and and Kubernetes so they're you know If you were to to do this whole sender secure delete thing that I was doing that's probably the most likely place for that job to run Because they're doing that kind of testing today between open stack and and Kubernetes But yeah, we've got a lot of different folks here using it for a lot of different kinds of development which I think is really interesting and exciting and These are some places you can find out some more information about Zool We've got a website at ZoolCI.org Where you know, we've got docs Link to our mailing list our IRC channel on on free node where you can chat with us and that sort of thing the docs have Instructions for getting started so you can get up get a Zool up and running pretty quickly You can also look at open dev.org which is a a very large public Zool system So that's where open stack. That's what opensack is using for its development So you can you can kind of see it in action there You can see the example jobs that we've got the standard library of jobs So that you know, if you if you do use Zool, there's a whole community of Folks working to make reusable jobs so that you don't have to start from scratch and then we've got And you know what I This is one of those things where it for me it almost goes without saying but I should say it And I can't believe we've gotten this far without me actually saying the words open source Of course Zool is a community developed open source Piece of software We're you know, we're not driven by One company this is actually You know a bunch of a bunch of people Working for a bunch of different companies as well as some people working on their own time Contributing to make this thing better so So anyway, yeah, it's open source. You can download and run it and there was also commercial support and hosting offerings that you can find out on the on the website and Awesome, you want to if that's it for the slides you can stop sure. Oh, actually you've got yeah So that's that yeah, that's it for the slides. I'm happy to This is probably a great time to stop for questions if there are any but if not I can Just sort of show you some of like I said, just show you what's going on on open dev.org live right now Here's it. What's the what's the technology stack with in Zool? Yeah, so it is written in in Python and It's actually kind of a it's a sort of microservice architecture and And so it's a it's a bunch of different Python services that that work cooperatively, so there's there's a web service. There's a scheduler that does all of the deciding, you know what to do with What jobs to run that sort of thing and then the executors are sort of there are most important scale out Feature so one of the things one of the Pain points early on actually was as as open stack was was growing exponentially We had to scale our CI system to to match it. So So we actually the the execution component of Zool is Is the scale all component as well And why oh slow computer there we go So if we if we look at This is the the Grafana instance for open dev and If we look at it we can See down here Yeah, okay, so these are the Looks like we've got 12 Executors online right now for open dev. So that's that's 12 servers That are just responsible for running the jobs That's not 12 test nodes. That's 12 Machines that are running the test jobs on the test nodes Right now. We're using Let's see almost Was it round Maybe eight nine hundred Test nodes right now. So these are about eight or nine hundred virtual machines that are that are in use right now Running running tests. So so yeah the the the stack internally is Python and Like I said microservices for scaling out We At this point We so we're kind of agnostic about whether you run Zool itself in containers or not you can you can you know Install it on a bear server or a VM or whatever in in the way that you would normally install install a Python program But I think at this point most of the larger Zool installations are running in containers Some of them are running Zool inside of Kubernetes. Some of them are just running it inside of Docker You know using Docker Docker compose that sort of thing but we that that seems to be the way What people are moving to And and it's really nice because you know we we build we build container images not only on every release but every commit So it's pretty easy to just run the exact container images that we're running in open dev for Zool's own Zool Let's see another Important thing to know about the tech stack is how the jobs are written. They are written in ansible and that's because Zool is a Multinode aware system. So, you know with some CI systems they they've They've got the idea of one test node down for running a job, right? But if you wanted to say, you know, you're doing work with Kubernetes and you wanted to actually stand up a full three node Kubernetes cluster With with Zool you can do that pretty easily. You can you can ask Zool for one node You can ask Zool for three nodes for your job and And Zool will happily provide you, you know, three virtual machines or three containers or whatever it is that you ask for and but if you if you think about writing a Job that uses multiple nodes, you kind of need an orchestration system You need a way of saying I want you to run this on node number one And then I need you to run this on node number two and this other thing on node number three and so we needed a system for saying I want to run scripts or programs or something on on arbitrary remote nodes and There was this great system called Ansible That was developed just for doing that and so we said well rather than you know reinvent the reel and and create our own Orchestration system for doing CI on on nodes. We said let's just use Ansible so what you get as a Zool job author is You write a playbook and Zool runs Zool sets up the inventory for you with the the nodes that it gets And then it runs your playbook against that inventory So if you if you know Ansible and you like Ansible then that'll be really familiar to you if you If you're like, oh, well, actually, you know, I don't I don't want my CI system to be opinionated about what You know how I do my deployment. That's fine, too Ansible as it turns out is really good at running other stuff So if you just have a shell script to do whatever it is that you need to do that's fine You can write two lines of Ansible that say run the shell script on the on the test node and you're done You've escaped Ansible and you know, you can you can do whatever you need to so Yeah, I think that's the the The highlights of the the tech stack there. There's you know an SQL database can be MySQL MariaDB Postgres Postgres girl, they're all supported and then Sort of behind the scenes we have a zookeeper system for helping those components Communicate with each other, but those are all Kind of behind the scenes details You know as a zoo Administrator you might need to know that they're there as a zoo user. You don't need to know that they exist at all so So yeah, this is a this is the current utilization of zoo on open dev and This is the the actual web interface for zoo So if we go look at zoo is a multi-tenant system. I should say so Starting here, we're actually looking at all the different tenants that we have in zoo We've got open stack, which is our largest tenant. We have zoo itself we have some Python projects the the cryptography and the packaging folks have some Have some projects in the system because we're actually All of the communities involved here are heavily dependent on these Python packages and And we want to make sure that this is going to that that cross-community testing that I was getting at earlier We want to help the Python community make sure that That they're able to test on all the platforms that they that are important to them And are important to us too. So we actually have some arm testing going on for these projects down here, but I'm going to Hop over the open stack tenant real quick and just show you what a fairly busy zoo system looks like so no Again, this is live so I don't really know what to expect here. This is This is what happens the the check pipeline is what happens when people upload changes So this is kind of before they've When you write a patch and you upload it for the first time this is when we run the the initial test I've even not gone through code review yet or or or that sort of thing. So If we actually go ahead and expand by default So if we look at this there's a bunch of in progress some have succeeded these jobs are paused All of these jobs are running for this change, right? So so each of these each of these little little boxes here is a change to a different project in the system and then if you expand that you can see jobs that have All of the different jobs that we run for that change. So a bunch here have succeeded some have failed Some are still running This is a long series of changes, right? So here's a here's a change that depends on several others so if This should start looking exactly like that slide that I showed you earlier And and this is you know exactly how we represent it in in Zool's status interface So these lines between changes are dependencies and then the gate pipeline is where we Changes that are about to merge, right? So Here's a queue of two related changes to Airship projects, right? So this one's going to merge first. This one is being tested with it In sequence and and I actually don't know if this is a get commit dependency or or just a dependency that's formed by the, you know, the approval sequence Or it depends on or what we'd actually have to go take a look to figure that out But regardless right now right now Zool considers that they are dependent on each other And so this one's being tested with the one ahead of it and And that's happening in parallel so that they merge more quickly This one these two are sharing a queue, but you can see that this one The one at the top here has failed which means that Zool has decided that This one no longer depends on it So we're still running the jobs on this because we're gonna we want to you know We've sunk a lot of time into running these tests already and we want to report back to the user as much information as possible So we're gonna keep running The jobs until Everything's done, but in the meantime As far as Zool is concerned this change is now The next one that's going to merge assuming that everything passes so no line connecting these two Is how we determine that Zool is has pulled that out Yeah You want to actually click on one of those I'm curious what this looks like in terms of like say if I'm a developer Like actually particularly into a failed test big deal developer trying to debug the failure what this looks like Yeah, so let me let me click on two things one Just while we're here. This is If you click on that open stack ansible one, I'll look at it from a core perspective Oh, okay Yeah, I just wanted to show you the streaming console output that we have so I mean kind of a standard feature bit You know if you if you like to watch console logs We've got that here What so what's handy or is that regex search? Yes, yeah, like, you know fail Right fail. Yeah, so Yeah, that's it's that's pretty cool So that's one tool that you you know if you see that something is is failing you can And and things are still in progress, you know, you can you can check on it while it's running. Okay, so where was the ansible? This one farther down, yeah the one thing so Do you see one? Yeah, and we just added some support for the latest CentOS and Ubuntu so else I think Debbie and Should I should I try this one then the ansible deploy AIO metals into okay? All right, so we'll and to be honest sometimes we just recheck and we go through So Yeah So we're we're actually kind of at a at a awkward place here in that Because Zool is still running this job we can access Yeah, we can we can access the logs, right? So we've we've got the logs here in In our log storage system, so you know, you can see the job output there's a Whole host like we collect a whole bunch of logs When when we're doing this I kind of want to show you the What it looks like when it's finished reporting Because Frankly because it's Easier to use and More There's some some neat Feature there, so I'm just gonna look for that same job and see if it failed recently And we'll look at that instead, okay so So so basically As long as the change is still in progress You know, which is what we've got here, right? We're we're still running which one we're still running this job Which means Zool hasn't like it's not final yet Zool It's clearly not going to merge but but it hasn't Zool hasn't made a definitive determination because all the the jobs aren't finished yet So it it hasn't yet reported back to the code review system the the final results things could still change So so that's why in progress we have access to the the raw dogs But but we don't actually have a final result yet to look at in Zool When it reports back to the code review system This is the the view that you get right and so there's actually a nice a nice Screen that shows you the summary of Over the build, you know when it started didn't finished and the branch the project the pipeline some other stuff We do this thing where we because the jobs are written in Ansible and You know Ansible is a series of tasks. We can actually look at the first task that failed And show it to you With the assumption that maybe there's something useful there. So So this is This is the this is the Ansible tasks that failed And this is the last few lines of it So often as a developer you can hop onto this You know this build summary page and and actually just see the error that tanked your job Right there This doesn't mean anything to me. I don't know if it means anything to you Amy Kind of looks like the container didn't build properly Which is why sometimes recheck works for us because we're a deployment Project so if the containers or the VMs didn't deploy properly will sometimes fail just from that Yeah, so this is this is the I'm feeling lucky Thing in Insul right you can look at this and and and and maybe this will just point you right at the error And you can go and fix it We've got all of the logs That So this is this is the slightly more Attractive version of that raw log storage that I showed you before right so, you know We've got all of these ETH tool config files things about things on the host and and And and this is you know something that we've we've learned after Years in the trenches in open stack development right this this job took an hour to run and My goodness after the end of that if it's failed you sure want to figure out why it failed, right? So that's why we collect so much log information so that when when something goes wrong Hopefully we've got everything that a developer needs to figure out what that was And they can go back and see everything that happened If the zoo has this artifact system, so you might We don't this particular job doesn't have a lot We've just got this script to download all of the logs and then the zoo manifest itself but examples of other artifacts that we have our documentation builds preview sites for documentation so if your job builds documentation you can just add a link here and then Developers and reviewers can click on it and and see the See the see a preview of the doc build Container images tar balls things like that. You can you can stick here as well. And then finally we have a View of The That's a big one A view of the the tasks in the Ansible playbook that make up the job so This is the really big thing that I'm going to hide But if you if you look at this The the job is made up a bunch of playbooks and each of those have tasks and you can you can kind of drill down into each of These tasks and and see what's going on So this particular job has one really long task in the middle that produces a lot of output, which is Yes, you know you can scroll through it all if you want but So anyway, that's those are some of the the tools available to Developers when they they look at failures like this So the folks have different kinds of learning methods. I'm curious Are there any ebooks or anything available for folks out there to learn Zool or what's the like? canonical source of info for folks to go grab So I would Go to Zool See I org and I don't think we have an ebook at this point But we Let's see we've got If you've watched this you you you've probably already seen everything that's in this video, so that's Kind of a short version of what I was doing earlier if you go to the get started page, we've Got a tutorial so This will get you up and so if you're if you're a hands-on sort of person This will get you up and running Just on your local laptop or workstation or whatever Using Docker compose so this will bootstrap an entire Zool system Using Docker compose and you can you can play with it there. We're actually in the process of expanding this tutorial system now so that You'd start with this and then build on that for for other tasks There are there's some other Parts of the Zool documentation that kind of Tackle this from different perspectives. So, you know, if you just want to read more about More about Zool concepts and things like that we've got discussing session sections there and I think there, you know, some videos on YouTube of Conference presentations and things like that And and finally there is a Zool from scratch section in the documentation as well so It's kind of the opposite of the quick start whereas the quick start is like just get me a Zool up and running So I can play with it if instead you are more of the No, you know, actually I want to I want to start with git clone And and go from there. You can do this all from scratch things. So Yeah, I'd love to have you know more of the ebook style stuff, but we do have You know a couple of different styles of Approaching this Web page and doing a ebook if you really want to So that's awesome. So thank you very much for this Overview and dive into the Zool. Oh, you're quite welcome. It's been Great to talk with you about it. Awesome Mike Josh Amy anything else? No word. I mean, I have other questions basically having worked on the Kubernetes project in prow We're once again, I'm looking at this and saying we didn't learn nearly enough from Zool. Yeah Although our biggest challenge is we have looked at the sort of multi project dependencies And our biggest blocker there is running out of cloud resources to actually run the tests Yeah the Because you know given microservice architecture there there are more projects in Kubernetes than there than there are an open stack The projects are smaller and they're more of them So but that could go on for quite a while and we're at times Yeah, no, so thank you very much James for coming on really appreciate it. Yeah, thank you for having very informative Thank you Thank you and without further ado We'll see everybody soon Coming up tomorrow on the channel. We will have an open shift commons briefing about Cloud native network functions. So if you're curious about telco space, which I know some of these people on this call are Tune in and we will discuss. Yeah, and then in two weeks for cloud tech Thursday We will have the folks from CERN Actually, Amy, you should plug this. Yeah, we'll have Delmero who is coming to talk to us about scaling at CERN on open stack So that'll be a really good talk because they're the largest user of open stack in the world So yeah, we're kind of to open stack you related things in a row I Personally takes all for granted because I use it all the time So it was really great to get some more of the history and some of the more in depth usage Which I'll go and use daily now. So I really appreciated this gym So yeah, we're looking forward to having everyone in two weeks And spread the love about Zool Yes, please share you out with folks who you see might need it. So thank you and stop merging broken code Best tag line ever All right, y'all take it easy be safe out there. Take care. I'll