 So, everybody, my name is Paul Vocio. I work at Rackspace. I've been there six years, joined in 2007. And I worked on the Slicehouse product just shortly after the integration. And I've been working on, we started on open source Zen, and we transitioned to Zen server shortly after for primarily the Windows support and a lot of the performance stuff. So, what we're going to talk about is just kind of our internal infrastructure on how we kind of build and run a public cloud, talk about how we deploy and some of the challenges that we have in doing this at scale and kind of what's coming and what we're expecting and some things we hope to see out of the Zen project. So, the way we deploy our cloud is we actually have an internal cloud. We call it INOVA or internal or infrastructure. And this is a bunch of internal OpenStack services that we offer to systems and users inside our internal network. And so anyone, actually they authenticate with the Keystone auth system, get the token and then present that. And then it's really just an OpenStack deployment. So we don't do all the public facing features that we get, but it's more of a vanilla OpenStack install. So, for all the same reasons that everyone wants to do this on the public cloud, we offer it to our internal customer. So easy spin ups and down, load balancing, testing, load, spikes, stuff like that. And we also actually do this to boot our public cloud. So I'm gonna go into this and show how it works. So we actually have this internal cloud. We boot all these services that become our public cloud. So when you're talking to an API endpoint at Rackspace, we're actually talking to a VM that's running on Zen, which is controlled by OpenStack. So I'll kind of show how we do that. And a lot of other reasons we want to do this. So how do we react to down hosts? We can migrate services around. It's just all the same benefits. So we have what we call just kind of a seed environment. It's just a generic term. But what we do is we take a Zen server and we boot a single VM on it and we install all the OpenStack services. So here's the API, the scheduler, the kind of non-scalable things, the database, the queue, and then we kind of package this up into kind of a golden RPM. And we take that RPM and then we kind of shove that into a Zen server. So we download that and boot it up. And this kind of one box and this one VM becomes the way we interact with it. And what we do is that one VM actually then we install the compute using that same mechanism onto a bunch of other Zen servers. So each of those Zen server hosts is running a VM that runs the OpenStack Nova compute service that does an interaction. So that single box up top is kind of our seed host. All the ones underneath that are running are now become part of the cluster. So this is kind of what we, when we bootstrap, we have a five to 10 nodes that get us off the ground. And then from there, we actually start provisioning more hosts. So from here, we're interacting with us just like an OpenStack cloud. And so my engineers and the admins and everyone who's building these just start provisioning services, configuring Puppet, kind of doing everything you would for normal kind of web app deployment. From here, these actually become our kind of just generic, what we call units of capacity. A unit is what we kind of quantify as 256 megs of RAM. So this becomes our capacity to go boot things. And so we actually split this up. Our I know environment is shared by all our internal projects. So everyone that's kind of runs an OpenStack or a public service or even a lot of internal dev and test services run in our I know environment which sits next to the same VMs that run our public cloud. So a public cloud API endpoint may be running next to someone's internal test rig. So we use our public cloud like we do our internal cloud like people use our public cloud. So you have dev and test and kind of mixed everything up in the same environment. And we do our best to kind of do performance isolation in the segment. So if you see somebody swapping, we'll start moving stuff around like we normally would on our public cloud. So those units of capacity become our total available capacity. And then from there what we do, I mentioned that we can kind of blur our dev, pre-prod and test environment. So we actually start provisioning more of those API nodes and this is what it ends up looking like. So this available capacity box right here is what we would call I know that no one ever sees except us. And then the public actually talks to our production API endpoints down here. So that's just a little bit of at a very high level on how we manage and deploy this. And then like I said, these become our dev staging and production nodes. So one thing I learned coming to Rackspace. And I kind of grew up in the small startups where at scale for me coming out of a startup was a couple hundred machines. And I thought that when you manage a couple hundred machines with two or three guys, we were being very efficient. We were doing things really well. Rackspace kind of changed my version of scale. And so when we came together and realized that we really wanted to build OpenStack and we really didn't want to do this again, we kind of set our goal at, our internal goal is a million hypervisors. We want the system to, when you move forward and you really want to think about, I never wanted to design this again, what do we want to think about? So we set our kind of internal goal at a million. So as a global cloud, we want to be able to manage hundreds of thousands of hypervisors. And so as we realized, we have to break these down. So we break these down to the region and then our cell into an individual hypervisor. So we deploy things in chunks of hundreds and thousands of individual hypervisors, which kind of gave birth to the cell concept, which we can cover shortly here, but it's something that we kind of had been running internally and then kind of brought it into the OpenStack community as a way to manage clusters of machines for whatever boundaries. So layer two network boundaries is really what we kind of designed ours around, but they could go as wide or as small as you wanted. There's really no constraint on how big a cell is. So it's whatever your, people do it can do it by cabinet or by hardware clash or whatever. We chose layer two boundaries within our network. So our normal deploying release strategy isn't that different than anyone else. So we pull trunk down, we kind of package it, we push it out to a test environment and then we iterate over that. The difference is we do this from trunk. So we don't wait till the major milestone releases or even the big six month releases. We do this every day. And there's a lot of challenges involved in that. And so I kind of equate this to the faster you wanna get to the speed of light, the more energy you have to get to do there. So we have to devote a lot of engineering people and time and resources into keeping us close to trunk. So we pull trunk down, we get all those new bugs and new things that people have fixed that go break us. And so when we pull these down, something that we thought was fixed or something was working is now broken. And now this pushes our deploy window out until this gets fixed or we have to decide is this a blocker or not for us to do this. But I think the advantages for this is we get features and we get fixes very early. And so we are usually a couple weeks behind trunk and that's how long it takes to kind of wash through our system. But I think the advantages that we've seen are good for us and good for our customers. So but usually by the time we hit the summit and the code has been cut and released, we're usually already on it. We put a lot of effort in trying to get the most current release out. So we're gonna do our best to be on the Ice House or at least when we hit the November summit. So when we talk about deploying and again we talked about scale because a lot of the OpenStack components don't work well if they're out of sync. You can't run a compute on a forward version and some of the stuff going forward and backward. We have to get everything else out as fast as possible. So what we kind of see is the number of deploys that we were able to do over our time period. And so we kind of hit a stopping point in February of this year when our scale got too big and the time it took to deploy ran too long. So because, and we used Puppet and we couldn't update the Puppet manifest and it kind of lazily checked in and updated its package, which meant it's now out of sync with the messaging library that's using the RPC go to the far side. So it was kind of an all or nothing type upgrade. So what we ended up doing was moving to a, we were doing Debbie invert packages. So we realized one of the ways we could get around this is just by doing a small virtual environment. So we actually started shipping the libraries and the code and the Puppet manifest all with it. So this kind of cut down on some of our pre-stuff because what we actually do is more like a fabric type deploy. So we'll push everything out and then just resim link it when it comes time to it. So we're actually moving packages at deployed time which saved us a ton of time. And we actually use BitTorrent to deliver a lot of these. So this was kind of a no brainer when you go back and look at it. But at the time, you know, we'd built everything using PSSH and SCP and kind of custom scripts to move everything around and the Torrent actually cut it down by a number of hours. So we're actually able to pre-stage using Torrents. This, an interesting challenge for us was also because of our, we have a fairly flat network and it goes very wide. And so what we ended up having to do was kind of segment this up into multiple seed Torrents coming out of cloud files. So we're able to see serve Torrents out of cloud files, push this across our clusters as long as you're not saturating firewalls. This works very, very, very well. And again, we moved away from a centralized Puppet Master to a Masterless Puppet because I said our updates happen very fast. When everything would check in over Puppet, we're basically DDoSing these Puppet Masters. So we would have to scale these very wide for a very short amount of time, assuming it all worked. If it didn't, then you'd happen to go back and kind of hot patch these certain particular ones that didn't check in. So moving to the Masterless Puppet, coupled with Torrents, coupled with our being able to push this ahead of time made these things go a lot easier for us. So how do we adapt for scale issues? So upstream OpenStack, what we're trying to do is work on more specific use cases on how we do this. So more particular testing around Zen coverage and Rackspace use cases and more production like Dev environment. So the DevStack environment is what's used to kind of gate a lot of the check-ins and that's not necessarily, sometimes the best way to gate how we check-in. So we're looking to get more production type loads either at scale and then again, talking about database and code manager. So something that works really well for 60 lines in a database, 60 rows, doesn't work well for us when we have 60 million. So if you're tracking all your images or all your snapshots or all your VMs that have run, we ended up having to do a lot of pruning. So something that ran very quickly when you checked it in through Garrett migration for us will take hours. And so we have, by staying up on trunk, we actually realize these are coming and then have to kind of mitigate. What we really like to do is move that step, one more thing, one more process upstream to where if you have large data sets, you run a migration, you know that it's going to work or it's not ahead of time. So we're not having to kind of scram on the backside and kind of struggle with once we've already had the code land. And then we still want to stay close to trunk. I think that's a big benefit to the community because we catch things, hopefully before they go into production, sometimes they slip through, but by having a large system running trunk at the edge of what we do this, I think is very beneficial. So talk about some of these. So we get early detection of codes and issues and we actually found it much easier to do smaller releases. So this is agile-ish in that we do this every couple of weeks. But by not waiting six months before releases, we're not getting these huge three-day never sleep deploys, which nobody really likes doing. So we try to do deploys in all the regions every couple of weeks. We'll take our smallest DC, roll that one first, let it bake, that one looks good, then we'll roll this across the others. What we'd really like to do, and I think some work is underway to do this, is upgrade ourselves at a time because we have dozens of cells with tens of thousands of hypervisors. It really becomes difficult. It's going to get difficult for us to roll an entire region in a night. What we'd like to do is be able to deploy small sets of machines throughout the night or have these roll through to where they can handle different message formats or if a database migration is run, being able to take the data from one column and migrate it to the second one. And on the next deploy, then you deprecate the first column and roll forward. But database migrations are really what holds us back from a lot of this. So some of the things where we'd really like to see this go. So IO scheduling for some of the VHDs, I know this is being able to detect swaps. And so instances that are swapping affect other tenants in a multi-tenant environment. And so when we first got on Zen server, a lot of the multi-tenant protections that we see now really didn't exist. I mean, a lot of the stuff that we had to go bake into it is doing macros spoofing and a lot of tenant isolation and a lot of things that it, I think the product was built for mostly use in a particular, let's say corporate environment where you semi-trust each other and you had a single administrator doing this. And that makes sense. But when you kind of give it to the public and tell them to go knock themselves out, they start doing really stupid stuff that you never thought they would do. And so we had to buckle this down really hard. And so one of the things that causes a lot of users, a lot of grief is someone would buy the smallest instance possible and really try to squeeze the most performance out of this. And what you would see is that disc would start swapping really badly because we haven't allocated a lot of CPU or a lot of RAM to them. And so that swapping would therefore affect other users on that same box, which is having to take up CPU time and IO time. And so being able to isolate those tenants on a disk partition level would be really key for us. And getting better VM resource usage. So I know that there's some R&D graphs kind of baked in under the hood, down in Zen server, and we'd get very kind of weird results from it. Sometimes we would pull for bandwidth and we'd be doing a couple of megs a minute and it would spike two gigabits and then spike back down. And because we were testing, we knew we had never sent this. So we ended up having to kind of reinvent our own kind of metering and billing around that. So that was kind of a hard point for us. So just a little better kind of monitoring and introspection on what's going down, kind of under the hood would be good for us. Zappi calls in Zen store, which has been a big one for us. So I think KeterBear was talking about people SSHing in and XMing or XE or XL commands. Yeah, that was good, we did a lot of that just because some of those commands aren't available over Zappi. So we would kind of look at the documentation and realize you can do some stuff on XM but you can't do it on Zappi. And so you're gonna have to kind of split our API calls. Some we would do over Zappi, which we wanted to for the session persistence, but somebody would have to fall back to SSH, come in and twiddle something and come out. So we ended up having this kind of fractured library, which was kind of a mess to maintain. And the upgrade experience has been a battle for us. So we're running probably a, I don't know, half a dozen different Zen server versions from 5.5 all the way up to 6.2. And I think 6.2 is the one where we finally get the live migration. So we're very, very excited about that. Just for the idea that we'll be able to distribute a lot of, you know, we see a hot tenant or someone doing a lot of load, we can move them, isolate them on tour of their own box or upgrade them where they don't notice. And so with this coming out, we've got some new DCs rolling out, we're looking forward to making use of this feature and kind of experimenting with it. Seeing how much it will improve the experience for the customer and that we're able to move them around or give them better isolation or better experience and they won't know that they're being moved. VDI introduction for new VMs. So this was something that we noticed instead of how to do an SR scan, we can just tell it, hey, it's there. And I think incremental backups we talked about. And I think some of this stuff either, so we're on Zen server 6.2. We've kind of been experimenting with going to or towards just OS and taking the packaging from the Zen project and building on top of that. So we've been working with these guys and some of the guys over in Cambridge on how this and we're very interested in how this is gonna work. Something we've been trying to work with for a while, but it seems now with the foundation coming out, we're finally had a good time to move forward. It's kind of been the open instead of trying to figure out how to do this just by ourselves. And I think we've been working with them as well and trying to virtualize the whole open stack experience to make this work better well. So we've been working with open stack guys and with the Citrix guys on how it can virtualize this and move the sword. Push the testing from inside the walls of specific companies actually out to the open to where use cases that we see in the real world using this under heavy load get tested before they kind of slip in downstream. So that's all I had. Anybody have any questions? I just want to thank you for your time. Anyone have questions? Well, thank you Paul. Sweet.