 I want to introduce from the infrastructure core team, Elizabeth K. Joseph. Thank you. So thanks for joining us. And I know you're on the Infra core team, who is the project team inside of OpenStack that kind of oversees and manages this incredible infrastructure that keeps OpenStack moving forward. Can you tell us a little bit about how that works? What's set up there? Great. So we have this big CI system, continuous integration. The developers will interact with it through changes to Git, which they push up to Garrett, which is a code review system. The code review system then triggers a bunch of tests to be run. So things are sent off to a tool called Zool. And Zool sends things off to NodePool. And NodePool is where we interact with a bunch of different clouds to run all the tests. OK. So where are those clouds located? I mentioned that there were 20 regions or something like that. Where are they located? So if you look at this map here, most of them are in the United States. We've got one in Europe right now. OK. So a few minutes ago, I put up a slide that had 20 regions of clouds just in Europe. And this is a multi-cloud application. Can we expand this application to include some of those clouds in Europe? Yeah, absolutely. We've got some queued up here. OK. Cool. So let's go ahead and add some clouds to NodePool. And now, how are you going to do this? So it turns out we allow anyone to propose changes to our infrastructure. So if you want to do it. OK. Even I can do this? You can do this. We have you set up here. You're in the project config, which is our Git repository. OK. Make sure I'm up to date. All right. Great. So what am I editing? So we're going to edit the NodePool configuration, because what we want to do is add these clouds that we've got ready here to NodePool. OK. So we're going to actually pull down the capacity of one of the existing clouds that we have in here, because there's a lot of them. We want to push load to these new clouds. OK. So this is? This is the OSIC Cloud One. So OSIC is one of our donors. They donate a lot of cloud to us right now. So we're just going to drop them to zero for this demo. And this is in production. So if anyone is pushing changes right now, you're probably not because you're here, right? But if you were, you'd be interacting with this thing that we're doing the demo on. So that's pretty fun. All right. So we've dropped that to zero? Drop it to zero. Now we're going to add the three clouds that we have donated for this demo. So the first one is data centered. OK. So each one of these clouds gave us a quota. So then we're going to add 10 servers. Next one is city cloud. They give us three regions. Three regions. OK. Yeah. So where was the data centered cloud? Where was that one? That was in Manchester, I think? Yeah, right. OK. So this is Frankfurt? Yep, Frankfurt. So there we're going to give 50 in each one of these. So in city cloud. And then in London, another 50. And then another 50 here. This is in Sweden. OK. The final one is enter cloud suite. The quota here for there, which one is this? This is Amsterdam, 12. Yep. And then another Frankfurt, 11. And in Milan, 10 here. OK. So that was seven regions that we, and this is, you know, we had this config file that we had the entries, but this is basically just, you know, defining a region, some images that are there. And right now we are activating them. Right. All right. So I just want to commit this change and push it up to Garrett. That looks good. Have a commit message so we know what we did later. And this process that I'm going through, this is the process that all OpenStack changes go through. That's correct. That's right, yep. OK. So I use git review, which pushes it to that Garrett review system that you were talking about. Yep. And now since this runs some tests, I did actually do these steps and push a review just a couple of minutes ago that will have already completed the initial set of tests, I think. Can look at it here. Yep, looks good. But if we look at this, we should see those changes that I just made. OK. And it's now in the review process. So what happens at this point? So now it needs to be reviewed by a couple of core viewers. OK. Well, I see a couple right there. Park Boylan and Jeremy Stanley. Will you guys approve my review? I think I followed the process, please. OK. All right, so they, oh, and Dirk Mueller said that it looks good for a keynote. I mean, one of the things that's interesting here is a lot of people may not realize that this is, you know, this is what we're on production systems right now. So this is the same review system that every change from OpenStack goes through. And if you go and you look through review.openstack.org, you'll see this, but also lots of other work going on. So this change has been approved now. And sometimes people say, well, you know, open source isn't that a little scary that anybody can just like make changes to the code. But this is, you know, I think one of the really cool things about this system that we built up over the last few years. Yeah. So how does that work? You know, it's firing off some tests now, right? Right. So we ran an initial set of tests called check tests. Once those come back, that's when normal humans usually review the code. So they know that all the computers think it looks good. And then we can start taking a look at it. So it comes back in the check queue. People review it. And then it goes through a final queue, which is what this is going through now, the gate queue, which is making sure that nothing changed since the check, maybe adding a few different tests. And then it gets merged into the repository and then. And this is happening on a continuous basis, right? I think that we have the stat here. It says 2,000 jobs per hour. Right, right. And this is a graph that kind of shows how that works, right? Yeah. So this is from the end of September. People are doing a lot of changes at this time, trying to get ready for the release. So you'll see sort of throughout the day that the numbers go up and down. But this is where we broke 2,000 jobs per hour for the first time. Which is pretty exciting. And so how is a job defined when you talk about that? Right, so a job can be anything from a simple linting test to make sure that the code looks good, from sort of an aesthetic and policy point of view, to things like unit testing and functional testing and then integration testing, which will actually bring up an entire instance of OpenStack and make sure that a change to one project, say NOVA, didn't break something in Neutron. So we make sure that the functional testing is done and integration testing to make sure that. And there's upgrade testing and all kinds of jobs that get run through this process. And actually, I've pulled up the job status page here. Maybe we can see that quickly. So Zul, you said earlier, is kind of the job manager. And right here, I see my project config change that they approved. It's passed one test. OK, good. I have one more to go before it gets merged into the system. And there are a bunch of other things happening in here also. Yeah, so if you look at the first column, those are jobs that are in the check queue. So again, that's the first initial check. You're looking at the gate queue right now. It's pretty small, again, because we're all here. We're not pushing changes. And then there are several other queues for different sorts of jobs. After it's merged, say, if it's a documentation job, it may publish that job. If it's something that's related to a website, it may publish the website or upload some tar balls somewhere or something. So where is all of this running? So this is running in the cloud. In the cloud. Yeah, OpenStack-based instances that we have donated to the infrastructure project. So we interact with them completely through the OpenStack APIs. And once we get them running, configuration management to manage all the fleet of servers that we're running these on. Yeah. And how many nodes are running through these systems as we are running all of these tests constantly throughout the day? Right. So right now, we have over 2,000 instances available to us. At the beginning of the year, it was around 1,000. So in the past couple of cycles, we've added a lot of capacity. And this is all because of companies donating clouds to the infrastructure project. OK. So which of those companies? Right. So we have a few right now. They've sort of come and gone, different ones over the years. But today, we have Rackspace. They've been a long time contributor. They've been around pretty much the whole time. OSIC, they're the ones that we just pulled out to do this demo because they actually donated. Sorry about that. We'll put you back later. Their capacity is pretty huge. So all the jobs would go to them probably instead of our new cloud that we just added. We also have something called InfraCloud, which is run on hardware donated by Hewlett Packard Enterprise. And that is hardware that we're running OpenStack on in the OpenStack infrastructure project ourselves. So we're running that as our own Toyota Cloud. So you are a public cloud user. And in that case, it's kind of a hosted private cloud that you guys are operating yourself. That's right. And we're kind of, it's a bit of an experiment. We're trying to sort of dog food OpenStack, make sure we can actually run it instead of just being consumers of OpenStack. So that's the InfraCloud from HPE. Internet is also one of our providers. And then we also have the one in Europe, of course, OVH right now. OK. And so OVH was that OVH has multiple locations, right, in the US or Canada. I think that they have a couple locations over there. And then they were also the current dot in Europe, which was in France. So I've been checking the status here. And it looks like my test has run through. And if I look over here, my change has merged. So when a change gets to this state, I pushed it. It's reviewed, automatically tested. What happens then? So once it's merged, it depends on the project. So in the case of OpenStack infrastructure, this project, it'll get shipped off to our server, which will then pick up the configuration and start adding the clouds. OK. So we have a status page over here that I think it looks like it's not, hasn't, oh, actually. So yeah, so we have some cloud, some instances building over in city cloud. So this will take a couple minutes. It's going to build some instances. And you'll see from these columns here, there's building. And that's where it starts preparing the instances. Once they're ready, they'll show up in the ready column. That means they're available for testing, running tests on. In just a moment, we'll see some in the in use column. That'll be the ones that are actually actively running tests. So if you were to push a change right now to this, anywhere in OpenStack, really, your change will probably end up in the in use column right here. OK. So we've got them building in all three. So and we had three regions that we added for city cloud. One for data centered and three for enter cloud. So those are all in different locations. And we had prepped this config file beforehand. But in order to change the application that drives all of the CI, we only had to change two files. We had the node pool file that we looked at earlier. And there was one other one, which was, where was that one? That was in a system config. And then it was in modules, OpenStack, project templates, node pool. And then it was clouds.yaml.erb. So this is a Ruby template for Puppet. But if you scroll down to the bottom of the file there. So I see the three clouds here. And this is really the configuration that we had to add in order to expand capacity to new clouds. Right. So our infrastructure has support to just adding a few different pieces of information. So here you can see there's authentication information as well as we name it, we give the region. And in certain cases, like in one, we wanted to use the, let's see, it was enter cloud. We're using a different glance API version. In data centered, we're using a slightly different Keystone identity provider version of the API. So we'll make certain tweaks based on performance or what the cloud supports. And we're able to do that. So right now we've got this view here. We've got some. So what we have here is seven regions that we just turned on, three new clouds, two config file changes. And in a couple of minutes, they've gone from not being part of this fleet, as you mentioned earlier, to now running test jobs in production for actual work that's happening on OpenStack. That's right. That's pretty awesome. So I think we have a final slide over here that kind of shows what infra looks like now in Europe. And we've added, as I said, seven new regions in Germany, UK, Sweden, the Netherlands, and Italy. And all of that in just a few minutes. And the thing that I want to just mention before we wrap up is all of the tools to do this are also open source. Absolutely. Yep. If you want to get.openstack.org, it's all on there in the infrastructure repositories. Yeah. And there are some other companies who are using these to do their own CI systems across multiple clouds. And so it's great because this is an example of where our community has built up amazing tooling, amazing infrastructure. And we use it, obviously, but other people benefit from it as well. Well, thank you, Elizabeth, for joining us. And thank you to the infra team for letting us play with your cloud in production. Thank you. Thank you.