 Can I just go? All right, cool. All right, hi, everybody. My name's Corey Bryant. I work at Canonical. And I'm on the OpenStack engineering team. And I split my time between packaging OpenStack for Ubuntu. And the rest of my time, I work on developing the OpenStack charms. So today, I'm going to be presenting on OpenStack CI as a service. And I'm going to kick the demo off really quick. So let's see. I'll talk through what I'm doing here. I'm cloning a local snapshot of Horizon that I snapshoted from upstream on GitHub just a couple days ago. I'm doing a shallow clone, and that's a little bit quicker. But what I'm going to do is I'm going to make an update to Horizon dashboard, to one of the titles on the screens. And I need your help. So somebody's got to just give me some five words or some sentence that's silly and stupid or just not too offensive. Ham is a sweet beef? Meat? That's good. That's good. Last time I did this, it was a honey bed. You don't care. Anything else? It's your turn to be famous. Ham is? Oh, I'm not sure how long this is allowed to be. We'll go with that. So we got our change. Let me just double check that we don't have a syntax error. Look good? Can you guys see that, actually? All right, so we're right there. Page title. Ham is a sweet beef, and we love green eggs and ham. All right, so I'm going to push that to my branch. And I'm going to switch over to Jenkins. Oh, shoot. Good call. Thank you. OK, so Jenkins is right here. Let me just, we've got a job here called git watch right here. And that's watching the git commit. This is just basic Jenkins stuff. And then that's going to kick off git deploy with deployer. So if I just want to check here and see if we've got, OK. So that's good. So if I look in here, it's on SCM change. And so it saw my git change. It's following my git repository. Side note, my git repository is on Launchpad. We just got git support on Launchpad. So it could be GitHub. It could be anywhere. And so that's going to kick off a git deploy with deployer. And so what this is doing, if I look at the log, it's going to do a deployment of OpenStack from source. It's using the juju charms. And so that's one of the things I've been working on this last cycle is getting support for deploying from git into the juju charms. Typically, they obviously deploy from Debian packages from Ubuntu, right? So that's what that's doing right now. I'll come back to this. It's going to take probably 20 minutes. So I'll just come back and revisit. So why OpenStack CI and why canonical? I think a lot of the CI systems that are probably available right now are smaller development environments. And so this is using juju. And it is enabling multi-node deployments at scale from source. And so two of the things we're really focused on at canonical is quality and quality at scale, right? Because you're going to be finding different bugs once you're scaling out, right? So this gives us the opportunity to do that. We frequently work in our oil lab, which I'll talk a little bit more about, with vendors. We have 30 vendors that we're working with, hardware and network vendors, that all work with OpenStack. And we're testing all sorts of permutations and combinations of OpenStack deployments in the oil lab every day over and over. Deploy, test with Tempest, and then destroy, right? And then so it's really important stuff, right? And so one thought is that this could be really helpful for those vendors. They can do deploys from tip, right, with their code, and know as soon as possible whether things are working or broken, right, as early as possible. So that's the goal. But if you take two things out of this presentation, I mean, it's about testing quality at scale, and that's the big difference. So this is just your basic CI, just in case you don't file a CI, or understand CI. This is basic. It can be tweaked and whatnot, but basic thing, developer proposals have changed like I just did. It triggers a deployment and tests, and then code is reviewed, and then code's merged, hopefully, you know, raise your hands. But if not, there's an error, and you start back over from scratch, right? So what we did there, I made a commit. It's triggering a deployment of OpenStack using Juju, and it's gonna be running Tempest. If you're not familiar with Tempest, anybody not familiar with Tempest? Yeah, so Tempest is upstream OpenStack testing suite. There's thousands of tests in it. Tests all the various APIs. You run it against a deployed cloud, a running cloud. And so it tests all various APIs and all sorts of scenarios, you know, like boot an instance, attach a volume to it, take a snapshot, and just all these kinds of things, right? And so by running Tempest, it gives you an idea of the health of the cloud. But it's not limited to Tempest. I'm running Tempest. There's Rally, and I know a lot of companies, you know, have their homegrown tests too, right? So it's flexible in that respect. But you know, the whole idea here is obviously just catch bugs early and often and run with the latest and greatest code. If you want, you can deploy from stable releases or master, so, you know, that's an option. So Canonical CI is a service. The, I've probably kind of mentioned these, the goal is to be able to validate code with upstream Git, TIP, or stable releases. I've tested it with stable ice house, stable Juno, stable Kilo, and master branch, and it's all been good. There will be breakage in our charms as master evolves, because our charms kind of, our charms do the deployment, the config management, the life cycle management of Cinder or Keystone or this or that, and so we obviously have to update those every cycle. And what we're gonna be doing is applying patches as soon as we can to our next branches, which are stable, or our development branches. But you also get the option, the ability to validate new vendor code against upstream Git, TIP, or stable. So let's say you're Contrail or, you know, some other NFV or SDN or storage solution that plugs into OpenStack. Well, this could be very useful, right? Because if you are able to keep testing from TIP and test your solutions from TIP, you'll know before we release packaging that you're working, you know, before Liberty even comes out into a stable release, you'll be testing, you'll be tested and sure. Same with, you know, enterprises, large enterprises. They've, you know, same kind of scenario, you know, be able to test from TIP and verify. So like, let's say you're running a public cloud or something, right? And you wanna test all this latest and greatest stuff, right? Well, you can do that, you can have a test environment deployed this way, test all that stuff out. When Liberty finally releases, you got a pretty good feeling that you're in a good position, right? So a little bit more about the scale. Juju's very scalable, it's really easy to scale. It's multi-node deployments. You can deploy to metal, to a cloud. You can deploy to AWS, Google Compute Engine. Azure, you name it. You can deploy locally to KVM or, I got a meeting. Sorry, gotta go. Yeah, so a lot of substrates that you can deploy to. And as I mentioned, you know, verify the deployed cloud with Tempest, Rally, or other tests. And just at the bottom, those are kind of the different things that go into this. Obviously OpenStack goes into this. Juju and Maz are the tools that we like to use. Jenkins, CI, testing could be Rally, could be Tempest, could be whatever. And Oil is kind of a side piece of this that may or may not be involved. So how does the magic happen? Actually, let me look over at the demo and just hopefully get a good feeling here. All right, so if you can see this, let me see, deploying OpenStack. This is the start of using a tool that we call Juju Deployer. It's a bundle deployer. It's got all the config for an OpenStack Juju bundle, like all the charms, any changes to default configs, any relations, they're all defined in this bundle. It's probably 150 lines of code, or YAML. Anyway, you deploy that, and that's what this did. This next step is pulling branches down, which are our charm branches. Then it starts to deploy services. So deploying service salometer using this local trustee salometer branch. Then we're gonna get down here into status. It'll start spitting out status about oh, heat is installing. Let me just search on heat. Heat is installing, and then finally, towards the end, it'll say heat started. So I'll come back to that, but good news is that's coming together. So what goes into our, how do we deploy OpenStack at canonical? This is the stack. It's charms, it's Juju, and then you have the substrates on the bottom, which are metal, which is mass. It could be VMware, which is coming soon, or it could be a cloud, or it could be local containers, or LexD is coming soon too. So does anybody know what Juju is? Probably heard about it. I'll just give a quick overview. Juju is essentially, it gives you the ability to deploy all these different microservices, right? And think of those services as being Cinder, Glantz, Nova Compute, those are all different services, right? And so you can deploy those and relate them. And charms are the code that makes up a service. So charms is just, it's Python code or Bash code or whatever that's in the charm store, in this charm. And if you went out and read the installation manual for Cinder, a bunch of the commands that you'd read there would be in there. So it's gonna install Cinder for you, the Cinder charm. It's gonna manage the life cycle of Cinder for you. It's gonna manage the config of Cinder for you. It's gonna manage relations to other services for you. So for example, most of the OpenStack services need a database, right? So if you just relate Cinder with the Cinder charm or the MySQL charm, they know that I'm creating a Cinder database in MySQL. It's just as easy as that, really. Juju's open source, it's got a nice GUI and it's got a command line and it's got an API. So Juju can be, you can deploy services to these different substrates. Maz is metal as a service. It gives you the ability to cluster hardware, physical nodes in your data center and treat those as if they were just virtual instances, right? So once you have a cluster of hardware, it'll provision an operating system of your choice to it. That could be Windows, could be RHEL, could be SLEZ, OpenSUSA, CentOS. Could even be Ubuntu. So yeah, so it'll do that and then Juju works with Maz to be able to deploy these services to Maz just like we could deploy to OpenStack or deploy to Azure or deploy to AWS. So you can deploy those same charms to all those different places if you want. So Maz is used, it's used by several banks in telcos. The open compute project is using Maz and the London Stock Exchange is also using Maz. And so there's a new config option for the OpenStack charms, which is how we enable the Git repositories. We point the charms to the Git repositories that you're gonna deploy from. So I'll just spend a little bit more time on Juju. This is just a snapshot of the GUI for Juju. This is kind of a, if you look at these different services up here, you'll recognize some of the names, Cinder, Keystone, all these OpenStack charms. On the left we've got Calico plugged in over there as a SDN NFE plugin. The really cool thing about Juju is that it's really dynamic. So if you want to take Calico out and put in Plum Grid, you can just easily just plug them in, right? If you wanna take Ceph out and use Swift, you can plug it in. If you wanna put Nagios in there, you can just put that in there to do some monitoring or ganglia. If you want to, well, we have an HA cluster charm so we can build an HA and so just put that in there. So there's all sorts of flexibility and then at Mark's keynote on Monday, Mark Shuttleworth, I don't know if anyone was there, but you could plug in a Juju Chaos monkey charm and he just kind of wreaks havoc and he was yanking networks and stuff the other day. So we thought that was pretty cool. Just go back to the demo. So we're looking pretty good. So if I just search for started, Ceph has started. We've got a three-node Ceph in here. Celerometer started. The Juju GUI, we put that in there. So things are looking pretty good. Oil, real quick. It's the OpenStack Interoperability Lab. I kind of talked about it a little bit. It is an extension to our partner program where we are testing all sorts of permutations and combinations of OpenStack deployments with vendor software and hardware involved. So there's 30 oil partners at this point. 12 of those partners are in oil production and 12 partners are in the process of being onboarded into oil production. There's a process of in oil where you start doing testing and staging and then we have to work back and forth to make sure everything's working as expected before we can get you promoted to production. And a lot of partners are charming their software. So for example, Calico was on that previous slide. So it makes it really nice for the whole ecosystem. And that's what it's really all about is testing the OpenStack ecosystem, right, and making sure that everything works. Just a couple more numbers. There's 27,600 is the number of OpenStack cloud deployments since June of 2014 that have been done in the oil lab. There's 43 different hardware configurations, including AMD, PowerPC, Little Indian, and ARM, and x86, obviously. And 81 combinations of OpenStack services tested in just April 2015. So it's really cool. And so one other thing is that at the end of the month, the oil partners, they get this report that goes through all the tests that have been run, combinations, successes, fails. So it's pretty nice. These are our oil partners. You see just a lot of hardware vendors, storage vendors, and network vendors. Okay, so what is supported from source? I mentioned earlier that I've tested Stable Ice House. These are GIP branches, like Stable Ice House, Stable Juno, Stable Kilo, as well as Master. There's breakage in Master. I mean, I opened a bug a week and a half ago because I couldn't deploy an instance and there was a bug. So there's bugs, you know? I mean, things get past the gate upstream, right? So, but it's supported and we're gonna make any fixes in our development branches of our charms. The charms that are currently have deploy from Git support are Cinder Glance, Nova Compute, Nova Cloud Controller, the Neutron Charms, OpenStack Dashboard, the main core charms. This next cycle, I have to get support into Cilometer and Heat and Swift, I think. So those will be coming and we'll have a complete full solution, but we've got a pretty good core solution right now. The deployment I'm doing right here is actually a mix of deployed from source and then the ones that don't support deployed from source are deployed from WN packages, right? All right, before questions, I think we've got plenty of time. So now that the slides are done, still deploying, it should be wrapping up pretty soon, but I'm gonna log into, let's see, it's moving really slow, sorry. So just, this is the deployment. As I mentioned, charms have the relations, so the final step of the deployment is that they add these relations to each other, so they relate all the charms that have defined relations. And it's really close to coming up. Once that comes up, I'll log into the dashboard and we can look at the change that was made. So I just have a little script here called check commits. It's just SSHing into each of these units and doing a Git log. So just to show you which Git commits we're running from. All right, well, the connectivity is a little slow logging into the system. It's gonna deploy pretty soon and we can look at the dashboard. Does anybody have any questions for right now? Spend some time doing questions. This is running on our test environment. It's initially a Maz deployed, a bunch of physical modes in Maz. We deploy OpenStack to that and then we deploy our test environments, use what's called the Juju OpenStack provider. So when we deploy these services, they're being installed into instances that are OpenStack instances. Yeah, yeah, yeah. But like I said, I mean, you could do the same exact deployments to various clouds. Of course, yeah, this is just, this is really just a prototype. It's as simple as you can get with Jenkins, right? Watch a Git repository and kick off a job. But yeah, yeah, I mean, there's a lot of flexibility there with the tests that you're running and whatnot. Any other questions? Could you elaborate a little bit more about use cases? I mean, a customer comes to you and says, look, I have this problem that you may solve and you do your work and then you give it back a test report, as I understand. So this would be something that would be run in-house for you, right? We might come consult and set it up for you and then you can have CI as a service in-house on your own environment, right? It depends on your needs, right? I mean, if you're, let's say you're like just throw this at rack space or something, right? And they always wanna roll out the latest and greatest OpenStack, right? But they don't wanna wait until Liberty releases to start testing, right? They wanna be able to test earlier. And so, and there's a lot of folks running private clouds or public clouds that are under the same situation. You wanna get that new feature fast, right? So that's one solution, that's one scenario. Another scenario is obviously vendors. Vendors wanna know that their code is running with the reference architecture of Ubuntu in scalable multi-node deployments, you know? It's all about flushing bugs out as early as possible. And then once the stable release comes out, you can go right to it. Yeah, one other thing that we also do CI on is charms. So like Calico has a charm and we do the same type of thing. When there's an upstream Git commit, we can run all sorts of tests on the charms, whether they're lint tests or unit tests, or we have these amulet tests which are a juju testing harness. So that's another thing that we can do. All right, so it looks like I actually have a dashboard. So this is the juju GUI for the deployment that we just did. We'll look at that first. And so that's exactly the deployment that we just deployed. If I wanna scale, let's say Keystone, I wanna add three units, I can easily do that from right here. Typically I'm like a command line guy too, so I could also do that from the command line if that was responding. But so yeah, so that's how easy it is to scale, right? What else, let me just look at, let's say the Nova Compute charm. I can dive down into the config options for this and the OpenStack Origin Git config option is the one that specifies the Git repositories to deploy from. So we need the requirements repository to update from global requirements. But we also grab Neutron. So there's the Git repository for Neutron and the branch is master, all right? All right, so that's that. That's the juju GUI and let's look at the dashboard. So one thing I'll point out, if you go to system information you can usually see the version down here and by default when we do deploy from source with the charms we're doing a shallow Git clone, it's a lot faster and because of that your version ends up being 0.0.0. But if you were doing full Git clones it'd take a lot longer but it would be saying 2015.2.0, which is liberty, right? So for the moment of truth, let's see if, there, see that? So, yeah, that's pretty cool, right? I mean it's not DevStack, it's a real multi-node scalable architecture, you know? If I go back to the juju GUI I just wanna see if, how Keystone's looking. Oh, there it is. Okay, so, yeah, we're still installing three units. But they install pretty quick. They're exact clones of the unit that was right there. It's just your scaling out, you know? It's like if you need more Nova Compute just scale them out and you got more compute power. So that's really all that I have, I think. If anybody has any more questions, please feel free to ask. I think we have a few more minutes, probably about eight more minutes if we wanna use that time. Yeah, yeah. So if you want to run Chef or Puppet config in a deployed Nova Compute or whatever to do something, yeah, it works together. Yeah. Yeah, I mean, yeah, you'd get it into those units and run that config, yeah. So they kinda, there's similarities, right? But they can compliment each other, yeah. It does in a way. We add these things, it's kinda, you'd have to, at this point, at the state it's in, you'd have to log in, you can do a juju SSH into a unit. The git repositories are stored in slash mount slash open stack git and then the git repositories are there and so is the Python virtual environment. So all your bins and whatever in there. And so these terms all have support for what's called actions. What you would have to do for that, you'd have to re-git clone the repository at this point into that directory and then run a juju action which reinstalls everything on that node and it'll just pick up that new git repository. Yeah, and that's also, that's kinda something that Charms do too is like in that scenario, it reinstalls everything, restarts all your services. Another thing with Charms is if you wanna change a config, you change that config, it's obviously not gonna go into effect until you restart the services, right? But it does all that for you automatically when you change, when you do that. Yeah, it's a good question. I think that's something that we could take back and maybe make an even better solution for. Can you define your control planes? Yeah, yeah. Any other questions? Well, thank you. If you have any questions, that's my contact info. Questions are best answered on our juju list or you can find us in IRC in juju. My name's Corey C.B., my nickname. So thank you.