 to do development on. If yours do, then you should come talk to me afterwards so that maybe we can figure out where you work, and no, I love Hewlett Packard, they're great. Wow, I get in trouble in new and interesting ways every single time I stand up in front of people, and I think I just found a new one. So that's great. So anyways, you can do all of those things, which is ultimately beneficial to your business, to your business goals, really. It's about making things easier. So that's the nice side of things. That's the pretty marketing picture of how this works. This is also not a picture drawn by Hewlett Packard, and it was also not Copyright 2013 Hewlett Packard Development Corporation. So again, sorry for my inability to use open office, or Libre office, whichever one it is that I'm using. I'm pretty sure it's Libre office. But this is sort of more what an open stack installation looks like. It doesn't really look like this. This is very nice and clean and pretty and understandable. This is actually an old and simple version of what a current open stack deployment would look like. The camel circuits, what? The family circus diagram, yes. There should totally be a little dude. There should totally be a little dude like running around this, that would be, yeah. I love that, that makes me really happy, yes. This is, I want to see that emanated version of that slide. That just makes me exceedingly happy. So in any case, this is what an open stack deployment looks like, and as you might be able to tell, this is a little bit complicated. There's a lot of pieces here with a lot of relationships to each other. So many so that we have many people whose sort of sole purpose in life is trying to make this easier and better. It's kind of indicative of, and this is sort of where we're going, it's kind of indicative potentially also though of applications that you might want to deploy in a cloud. Because with something that's this complicated, you might want to be able to iterate on that easily to develop it. In any case, with an application like this, you wind up with some typical issues that crop up in your, wow, I have, we're being invaded by something. That's kind of amazing. One of the most interesting sounds I've ever heard. So you've got some issues that will crop up in something as complex as that previous thing that can cause you some problems. Probably there's other ones, but a few of the biggie ones that we've been sort of trying to deal with are bugs in the software. Turns out when you've got that many components, some of them might not work properly. I know it might be far fetched to believe in open source software, but we do have bugs. A million eyes or over 2,000 pairs of eyes don't catch all bugs. Also in the installation, Croft and Entropy can come up. You've got thousands of machines. Things happen. Admins log in and make changes somewhere and all of a sudden you've now got a divergent fleet of things and then you do a new deploy and then something doesn't work on one of the nodes and you don't really know why and then eventually you figure it out. It's because somebody installed some sort of lolcat software on the server. I don't even know what that mean, but something happens. The server drifts from the state that you thought it was in so that your application of a new state of the server changes. And then the thing that we all know and love is hardware failure. Things can die and that can also make your very complicated application unhappy. So in thinking about the complication of an OpenStack deployment and in thinking about these problems that tend to arise is where we came up with what we're calling triple O these days, which was this is what happens when you let me come up with a code name for something. It stands for OpenStack on OpenStack. It's sort of the word has stuck around and gotten a little bit more prevalent than one might have expected. And nobody seems to be able to spell it. So just so you know, it is spelled out with a word and not as three O's, it's not that important. But what it is is looking at a number of things from the perspective of treating OpenStack like a cloud application. So that we can do things that we sort of know how to do with other cloud applications such as continuous integration and delivery. We can keep the install and maintenance costs of running one of these clouds down because a lot of this, in addition to being about velocity, it also certainly isn't not about margins. And allows us to use all of the fun tools that we've written for our cloud end users to encapsulate and describe the installation of the cloud itself. And then sort of finally the common API part. We've got this API that allows us to manipulate infrastructure resources kind of neat if the guys in the data centers running the cloud themselves got the benefit of being able to use that really nice API for manipulating and controlling infrastructure resources. So we're really trying to be friendly to our ops guys, even though it causes them to want to go a little crazy at first. So there's a few sort of specific, along in the process of this, to deal with the problems that I was talking about before, there's a couple of different ways that we're approaching trying to mitigate the issues of the complexity. From the bugs perspective, and this is sort of something open stack wide, the more we can do continuous integration and continuous deployment is the larger chance we've got of catching bugs before they get deployed. Right, so rather than sort of deploying things and then going, oh, that broke, we try and do rampant amounts of testing. This isn't specific to Triple O, but it's certainly a philosophical point that the Triple O team ardently supports as an approach, and is sort of part and parcel to how we're thinking about the deployments. The cruft and entropy side of things, we are attacking from a golden images perspective, and typically when we say that, there's the people that are like, oh gosh, that doesn't work, we tried that before. So I'll talk a little bit more about the specifics of why I think you're wrong, but not why you're necessarily wrong, but why things might have changed, let me put it that way. Okay, why you're wrong, just why you're wrong is the way I'm gonna think about that. And then this is no news to anybody, but one of the best ways to deal with hardware failure is to actually develop an HA deployment of your thing so that if the hardware fails you don't have a service go down, not rocket science there, but sort of underpinnings of how we're thinking about what happens. So this is a diagram that there's absolutely no chance that you can read the words in the boxes, or possibly if your eyesight is better than mine, I'm gonna say I can't read the words in the boxes. That might be because I'm getting old and need glasses and haven't bothered to go get them yet. This is sort of the workflow of managing and deploying an open stack based cloud in the triple O sort of mindset, which is that everything in a deployment starts with the change the developer makes in their local code repositories, right? Turns out you can't deploy new changes if somebody did not write them, unless you've got a magical code generating bot somewhere, but I guess even that would be writing the code. So somebody developer writes the change and uploads it into the CI system, whereas it's tested, and at this point we generate an image using some of the tools I'm about to talk about, generate an image and then spin that up in a VM and test the image itself, that the set of software that comprises the things touched by that change, test it and go, hey, does that work at all? And if it does, then we can take that image and combine it with the other images that comprise the service over here, and then we can actually deploy that onto a set of bare metal machines, because we have a bare metal cloud that I'm gonna talk about in a second. Deploy the set of those and test that, and all of this is completely, like a completely automated workflow at this point, other than the developer writing the change, we still haven't automated that. But deploys that onto the bare metal nodes and tests that. So now you've got a multi machine, multi image deployment being tested, and then if that works, then you have a general sense of ease and pleasure about the change that you just produced. Especially if down here you're actually properly deploying a sort of a multi node situation and testing multi version, I've got some of my nodes in the older version, I've upgraded some of them to the new version, et cetera, but those are all automatable. So you do that, and then at that point you can publish the images into your public server, which can then feed the next iteration of this, because probably you're doing more than one change a day. If you're not doing more than one change a day, I don't know, well you probably are. But you can publish the images so that both they can be consumed by your production infrastructure, or they can be consumed by the next iteration of the next changes that are gonna come after it for testing. So to do this, we've written a number of small tools. We're big fans of writing small special purpose tools that can interact with each other, but also that don't require each other, so that you can use some of the stack or not, and other parts of the stack or not. And just for comparison's sake, tried to sort of group these into sort of sets of sort of areas of concern, and sort of done some indication here of where some of the other tools in the space take care of one or more of these tasks as well. So like in the puppet world for instance, configuration, state management, and the software installation are all handled by Puppet, and then Razer from the PuppetLabs guys is the thing that they've written to handle bare metal provisioning. In our world, we've tried to split each one of those up into its own tool. So that if you like our tool for software installation, but you still want to use our disk image builder tool, but you still want to use Puppet or Chef on top of it, that should be completely fine. And if it's not fine, then we've done our job wrong in terms of separation of concern and making sure that everything is modular. So the tools and the things across the top are the sort of the main focal point tools. Nova Disk Image Builder, I actually need a new column for this, I just realized I've fixed the rest of the slides and not this one. We have a set of three tools actually, OS Apply Config, OS Refresh Config, and OS Collect Config that work together to do things in terms of configuration and state. And then Heat is the thing that sits in the position that knows how to do multi-node orchestration. Absolutely, you'll hear no words from me about anything that do not involve completely open source things, and actually all of these are constituent projects of open stack. So, and I'll dig into that in just a second here. So, sort of directly following up that, these are the sort of, this is sort of the list of the components themselves that are important. The list has grown a couple of times since the last time I've talked about this. Underlying all of this for this to work is what is currently, what's currently sort of production usable is the bare metal drive-off for Nova. So Nova Bare Metal, Nova's the open stack compute component. That is in process of being split out into a project called Ironic. Ironic is not ready for production yet and will not be ready for production in this next release. So Nova Bare Metal is still the thing that you would want to use in an actual deployment today. Ironic is taking what was, a lot of the things that were sort of hand hard-coded in Nova Bare Metal to make the sensible one and make it more modular so that we can interface with more vendors, ILO systems and, you know, because we were starting to get the point where it was a plug-in to Nova which itself needed plugins and that's when you start going, okay, maybe this wants to be its own thing. So that's underneath that. To drive the Nova operations itself, we use Heat which is the open stack orchestration engine and then those are both tools that sort of exist in open stack and have sort of separate life cycles. The Triple O team itself is responsible for the next collection of things. There's a tool called Disk Image Builder which amazingly enough builds disk images, very inventive with names. There's a tool called OS Apply Config which is a very, you could almost say that it's config management but it's not. It's really not trying to be a chef or a puppet replacement. It's trying to be the minimal required management of just config files and not other state. Partially so that we can drive installation and testing of these things in a pure open stack environment without having to just write specific shell scripts or punt out to Chef or Puppet. OS Refresh Config works in contact with it. It's actually the thing that orchestrates the running of a thing that is going to muck with your config files because there's things that you might need to do like, say, queues a service before applying the new config file and then start the service again after dealing with the config file. That level of in machine orchestration is handled by OS Refresh Config. There's also a lovely little tool which is actually just useful in and of itself without any context of this called OS Collect Config which knows how to get your config data from all of the various cloud metadata sources that might exist. So if you've had a machine that's been spun up by heat there will be metadata in the heat metadata service. There will also be metadata in the cloud in it whether it's the local network interface, the EC2 metadata server, or whether you've got a config drive or whatever like that. There's a bunch of different places that something might have passed some metadata to your server. And OS Collect Config will go to all of them and get them and present them to you as sort of one single pile of JSON which is not particularly all that complex but winds up being kind of handy. And you can just pop on to any, pretty much any server and just run OS Collect Config and it'll show you all of the cloud metadata that is available to you on that server. Which I find handy just because I can never remember how to curl the fancy internet address that's there to get that. So anyway, so that's that and it's used by the other two. And then we have, so disk image builder is comprised of series of elements. So if you're going to build a disk image of software you need to describe what software you want to go into that disk image. And the building blocks that we use, we call elements, they've been called at least five different things since the beginning of the process. And all of those terms were overloaded. So we think, we think, we've been on elements long enough that we think that we're okay with that as a long running term. And there's a repository of disk image elements that are the ones that we use to actually deploy an open stack. And that's in the triple O image elements repo. And all of these, these are all the, you can look at it URLs. These are not the clone from this, no, sorry, these are the, these are actually a, dang it, I did it backwards. Yeah, yeah well, these are the clone URLs. These are not the look at it URLs. I realized earlier today that we have a new Seagit server in the open stack infrastructure and it does take putting URLs here, interesting in terms of indicating which activity you're expecting to happen from those. In any case, so I've been babbling a lot about the components. The short version of the story for a triple O deployment is that you have a heat stack which defines your cluster. Heat then drives an over API to deliver images to the machine. So this is sort of a typical cloud activity. Hey, cloud, spin me up a one of these. So heat drives the orchestration of that. And it does these things in a sort of dev test environment on virtual machines, but because we've got this Nova bare metal thing, it does that on full bare metal for actual production deployments. And if you've got the bare metal, you could do that in your CI CD environment as well because it turns out it's clouds all the way down. So Nova bare metal is the thing that takes possibly the weirdest amount of mental bending to get your head around. Nova has the idea of pluggable virtualization drivers. So you can use Nova to spin up KVM VMs, Zen server VMs. You can actually use it to spin up VMware VMs. If you have the ability to do these things, you could do Hyper-V. I don't know anything about doing that myself, but I hear that it is a thing that Nova can do. But in general, it's a pluggable backend that knows how to talk to different things to get VMs. So the Nova bare metal driver took the vert layer interface and put an implementation of it that speaks Pixie and IPMI. So rather than spinning up VMs on the machine on which the Nova compute instance is running, it talks over the network via IPMI and Pixie calls to other machines in your rack and manipulates them as if they were VMs. So we've gone from the place where we have a cloud that we did to give us a virtualized abstraction to get us things that behaved like real machines and now we're taking real machines and making them behave like the VMs that we then want to act like real machines. So, and the best part about this, which is really fun, is if you want to hack on Nova bare metal or Ironic, there is a virtual bare metal power driver. So you can use VMs to behave as if they are bare metal machines so that you can boot them so that you can get bare metal machines that behave like VMs. And thus you can Pixie boot VMs on your laptop so that you can just do development of, anyway. Yeah, but the fun thing about that is actually, for what it's worth, there's virtual box and VMware desktop drivers and stuff like that in there as well, so if you've got various ways of, I don't know, if you're like a vagrant fan or something like that and like doing that, you could actually do that with Nova commands instead to drive the VMs on your local workstation. Might be a little bit overkill, but you know. Why? What's that, yeah. Just, yeah, the easy way is just install OpenStack on your laptop and that'll make it really easy to manage VMs on it. But you'll treat them like bare metal machines so that it's not, anyway. So that's Nova. So once we've got that, once we've got a Nova API that actually can control the hardware and you can think of this, right? Nova already knows how to sort of scale horizontally out by adding more Nova compute nodes. You can have a data center's worth of hardware that's just basically controlled by a Nova API calls and we're talking like a couple of machines of overhead in the data center to manage all of your bare metal like it's a cloud, which is sort of cool. But then you've gotta be able to do something with that. Yay, I can, oh, that's weird. So the next thing is when you've got a data center worth of gear, you might not wanna be thinking of it one machine at a time because you might quickly go mad. So Heat is the OpenStack orchestration tool and the thing that it's focusing on is the multi-machine orchestration level. Heat's job isn't to manage the MySQL inside of your machine. Heat's job is to say I need a machine that looks like a MySQL server that's been defined in some particular way and I'm gonna make one of you there. And so then the things that happen in the machines are basically a handoff from metadata that a combination of Heat spinning up the right base image from the cloud provider that you want plus Heat handing some metadata to that spun up image. So it delivers that metadata and also Heat accepts return values from the thing. So when Heat spins up one machine, it says, hey, here's what I know about you. You go do whatever it's you want to when that machine is done handling that data, it actually calls back to you and says, I'm done. And also you might not have known this piece of information about me before this moment because it wasn't possible to know. So here, here's some more information about it which can then be parceled out to other people in the cluster. So that's sort of the space in which Heat sits and the position that it sits. You can use it with something like Chef or Puppet. Chef and Puppet are really good at managing the configuration of a machine. They're really, when you start using them for the multi-machine coordination of things, things get a little bit weird. It's honestly not what they were designed to do. You can do it, but it's not the underlying thing of that. So the idea here is to decouple that so that the thing that knows about the machine then just tells the machine, you go do whatever it is that you do on that machine and I'm fine with it. You're a MySQL server now, go for it, yes. Salt, yeah, I could, so I apologize to various people. I've kept most of the references in here to just be Puppet and Chef rather than Puppet Chef. Salt, you could, salt, same thing. Salt, because of how it was written, has a little bit more understanding of sequencing and relationship and orchestration between servers. But in this context, Heat can certainly drive salt states on a machine as well. So certainly absolutely possible. The thought here is basically that there's two concerns. One of them is what is the relationship between your machines and the other one is what's happening inside of the machine itself. There's a set of heat templates that describe an open stack installation which are at, you can clone them from get.openstack.org slash openstack slash triple o heat templates. You'll notice that we tend to name things after what they are. So that's a place you can go get those and see what those look like. So if Nova Bare Metal is a way to boot, to boot Bare Metal machines, boot images onto Bare Metal machines and heat is a way to pass metadata to those images that have been booted, you probably kind of need a way to actually make the images that you're going to boot onto the Bare Metal machines in the first place. Or the VMs because it turns out it's cloud software and so you can actually use the exact same image to do both of them. And so this is where we wrote the tool, Disc Image Builder. And the idea here is to encapsulate and describe the sort of set of software. These are intended to be static images essentially, not to contain lots of specific specified metadata. So if you've got a thousand machines in Iraq you probably aren't going to wind up with a thousand images. You probably wind up with an image for each type of thing and then configuration data from heat gets passed into the machine at boot time or at cluster stack reconfiguration time, which then can be specialized by your config management system. This is sort of another way to think about this is the equivalent of packages except at the machine level, which is sort of what the Juju guys like to say about Juju. But I like to steal as much wording from the Juju guys as I can because it's fun. But the idea is that you've got a description of a machine that's repeatable rather than just the description of one of the building blocks of a machine that's reusable. And especially if you're thinking about thousands and thousands of machines then thinking about the individual, the multiplication of the packages plus the machine. Anyway, it's intended to be building blocks at that level. One of the things that's nice about this is that because the images themselves are sort of self-contained and accept a sort of a known and reasonably small set of metadata inputs, you can test the images and then when you go to deploy you can actually deploy the actual image that you tested, not just the instructions for creating that image but actually the actual binary bits themselves, which is kind of fun. It also means that you're a little bit more, you have possibilities for efficiency because you're not running the build steps to create what should be pretty much identical a thousand times, a thousand times. You're running them once and then you're just splatting out the results of that. Which is again sort of why we talk about that being sort of like packages except at the larger level because that's one of the things that you get out of packages, you don't rebuild Emacs every time that you install it. You just, well except for the elist files which you have to buy to compile, but I'm digressing. I think you get the point there. So there's again small tool chains. We've got this image builder which I mentioned earlier and triple low image elements which is a collection of elements which describe the software. So we've got the image elements which describe the images that we want to deploy with heat and we've got the heat templates which describe the collection and relationship between those in a running cluster. And then I talked about these a little bit earlier. The OS, the OS splat config utilities, the three of them. I've sort of talked about what they do but basically the interaction between them is that you have OS collect config, gets the metadata from heat or from cloud in it. One of the reasons for that is that we would sort of like the first boot and the first boot configuration action and the subsequent configuration actions to sort of go through the same mechanism rather than to go through wildly different mechanisms. OS refresh config then knows how to quiesce any fragile services that you might need it to do. Upgrade software as needed from Glantz. Trigger OS apply config to actually manifest any of the metadata into config files that might need to happen if needed to reboot the machine. Although that's a, there's another set of information about that later. And then after that has been done, OS refresh config ensures that any of the required services that it might have mucked with are running again. And then after you've done that, you might need to do migrations. If you've applied an update, you might have to do a data migration in a database or something like that which needs to get applied. And then the fun part is that once it's done with that, it sends heat back a message saying, hey, I'm done, which means that you can actually describe things like rolling upgrades in the heat structure because you can say I need you to do, I've got five of these guys, I need you one at a time and need you to not start the second one until you're finished with the first one. So heat actually has the context to know that the operation it's attempting to do on the first one has finished before going on to the next one and doing that one. So sort of managed orchestrated rollouts of that sort of thing is possible. Although I should be clear that some of that is theory, the heat guys are currently still working on rolling out the orchestrated rollout thing, but the structure of how we're describing that is there. And then the theory of getting that to go, yes. So in our version of the world here, we're talking about using Puppet and Chef to, potentially if you're into using Puppet and Chef, using those to actually manage configuration and not using them to manage software installation. Because running app to get install new package version 10,000 times across your 10,000 servers is actually not the quickest thing in the world to do, whereas spotting out the results of that is actually pretty repeatable and you've got 10,000 new points of potential failure each time you run the code that does that. So we're trying to split that heat in, excuse me, Chef and Puppet are actually extremely good at describing configuration changes because they are config management systems. So it's a little bit, you can use those if you want to. In our particular case, the OS apply config thing we've got here is sort of a special purpose one for an open stack installation. I would imagine that if you're doing a non open stack installation, you would probably use Puppet or Chef rather than OS apply config to actually manage the config files on your server. But the main thing that in looking at this is what you can get to is having heat be the one that triggers the Puppet or Chef run, except it's triggering it on a per server basis in a phased manner. And so you're actually able at that point to do things like either use Chef solo or Puppet apply rather than needing a Chef server or a Puppet master because the input, the sort of the things that you would normally get outside of your recipes or modules in a Puppet or Chef world are gonna be the specific pieces of information, the metadata about the server that those recipes would need to know to be able to do their job. And heat is the one with the view of the sort of whole collection of software. So it's able to say, hey, Puppet run with this file of JSON, it's sort of like from the Chef world, I think that'd be like a data bag, right? They'd be like, here, run with this. Run with this piece of input data and apply these input parameters to your recipe or your module. And so then you also don't get into the scaling problems that you see with things like a Puppet master and Chef server at tens of thousands of nodes trying to manage them, although I'm sure that, I don't know, I was about to say bad things. But yeah, so that's the, it's not necessarily a let's kill Puppet and Chef as much as it's maybe let's rearrange things a little bit. And in fact, that sort of dovetails into a really interesting sort of point here is that building some images and then deploying them using sort of your cloud tools, it's kind of really fast. And speed is not everything. Correctness is actually way more important than speed. But when you're talking about deploying tons and tons of things, if it takes four hours for you to actually deploy an upgrade because of the time that it takes for Chef to calculate the graph and then for it to partial out all the things and then on each machine it's gonna take 20 minutes to run all of the upgrades or whatever that is, that stuff adds up and it adds up to making you a little bit more worried about rolling out one of the upgrades because of the impact that it might have on your system. We can go from a rack of bare metal machines that's completely powered off to a completely functional working rack of software and with the software installed in six minutes. Which again, you don't do this very often, right? That's to be really clear, the times in your life when you do that are kind of small, but that's sort of to illustrate the sort of efficiency of just splatting out the changes that you need rather than having all that stuff run every time and across your days, yes. There's a what? Yeah, so that's not, so there's another, sort of dive a little bit further into the, into sort of ironic and Nova bare metal and how those work. The question was, assuming there's a system to be able to basically take a node out of production and put it back into the pool of things. So in the way that the Nova bare metal system works, you have say your Nova compute controller that with the bare metal driver in it and then I don't know, say a thousand bare metal machines that have registered with that. Each of those machines is basically a resource that can either be spun up or spun down and is like a static pool of allocated VMs. So if you've got a compute node that's got a certain amount of RAM and disk then there's a certain amount of VMs that that compute node can spin up. And in this case, this compute node has a certain number of bare metal instances that it can respond, that it can, it can spin up. So it can delete those just as easily. It's kind of like what? Yeah, yeah, it's a little bit lower level in this. This sort of assumes that basically just sort of letting you know that we've got this thing which lets us use your bare metal like a cloud but then basically most of the cloud functionality we'd expect to happen is present. So actually really what it is is to just mark it off maybe power it off with the IPI things and the next time you wanna use it you're gonna be splatting a whole new image on top of it. There's an interesting thing that somebody should find me at a bar at if you're interested in talking about this or actually you should find, thank you. Matthew Garrett who's sitting over there and talked to him about it. There's some interesting ramifications of a multi-tenant bare metal public-facing thing in terms of what you can do to the BIOS on machines and things like that that need to be sorted out before people start doing actual public-facing bare metal NOVA which may or may not be solvable even but some of them are really interesting. Anyway so that's, it turns out I babble a bit too much. I'm running out of the time so that is I'm gonna sort of talk faster and I even added more slides for Jim this time and I might have to show him those at a bar. So the general structure of this is that we have a couple things. We have what we call the under cloud and we have the over cloud because once you start talking about using clouds to run clouds things get tricky really quickly. And so in this case we have an under cloud which opens up with bare metal and then we have one or more tenants in that cloud that are themselves KVM based clouds that are running on the bare metal machines that are part of the resource pool of the under cloud. Sort of have to say that four or five times to yourself and your brain before it starts to really make sense but we're trying to invent some terminology there so that we don't get quite as confused with talking about this cloud and that other cloud and this cloud that's inside of that cloud and whatnot. But so you can at this point since you've got a cloud under the underhand you can add additional tenants and have sort of additional clouds. So if you wanted a production cloud and a dev test cloud those could just both be tenants in the exact same pool of bare metal so that you can elastic your clouds as easily as you can elastic the resources inside of the cloud which is kind of a neat idea. The under cloud itself needs to be a fully highly available bare metal open stack cloud. It's self-hosted because it turns out this is a cloud that knows how to operate bare metal that's running on bare metal so is absolutely possible for it to pixie boot itself after you get past the first stage which is a little bit neared but if you've got two nodes that are in an HA failover scenario you tell one of them to take the other one out of commission and pixie install it with a new version of itself and then when that one comes up you orchestrate the other thing to happen on the other node. That one hurts people's heads a little bit but then you can basically have as few as two machines for the control plane of that and then the rest of them are just bare metal resources to be the source of your tenants for the other things. The over cloud then is again fully HA because a lot of these things work you don't really wanna run a cloud that's not HA because then you're just everything's gonna break. There's a fully highly available in our case KVM based open stack that's hosted as a tenant on the under cloud. So you've got a KVM cloud and it's running inside of the machines that are inside of the bare metal cloud. This is orchestrated by the heat instance that's running in the under cloud. You can also have a heat in the over cloud but there's a heat in the under cloud that's managing the bare metal resources on the under cloud that manages that. And one of the neat things about this is that the disk images that you build for your services you can use the exact same images on bare metal machines as you can in VMs because of the way that all of the Nova stuff works. So you can actually if you've got services that want to live both in the under and over cloud you can deploy them from the exact same software source should you choose to. The initial installation is a sort of a special case which I won't go into too much detail but the general version of it is install a cloud all in one on your laptop, plug the laptop into the network card of the rack, register one of the nodes in the rack with the cloud on your laptop and tell it to HA expand itself out to the second node that it now knows about and then take your laptop away. Now you've got a, you had what was an HA pair now it just got broken and it needs to heal itself so then it'll expand out to one more node and now you've got a two node HA cloud in your rack which you can then use to install the rest of your KVM clouds. That's the short version of that because I've run out of time. There's two versions of the upgrade story which again I'm gonna blow through in three seconds. The simple version is all of this is HA so for all of your scale out things you just shoot one of them in the head and reinstall it and reboot it and you do that in a rolling fashion. This is the sort of standard scale out how you deal with your Apache web farm. You just kill one of them and reinstall it and then you do that to the next one and if all your HA stuff works and you have the machinery to do that then score. Like it's the same machinery you use to deal with machine failures and it's not a problem. However, that doesn't work if you've got actual things that have precious data. I mean you could in theory do that but it starts to get really costly and even an HA failover situation the cost of dealing with that might be such that doing that say 12 times a day might be a little bit, you might basically be running in a degraded mode all the time and that wouldn't be exactly great. So if we talked earlier about having, I sort of blew through the slide because I talked about something else. The, so part of the idea here is if we think about these things as cloud apps the general theory is that your cloud compute nodes are kind of ephemeral and that you have volumes that are less ephemeral that you attach to those nodes which is where you would put precious data because in theory, so it doesn't always work out in practice, but in theory the volumes are the place where the cloud provider is telling you this data isn't gonna go away but these VMs might go away. And so in the sort of VM cloud world you'd have a sender volume and in the open stack world that has your data on it and you can then spin up a new node and spot them out. You can sort of do the same thing here if you think about partitioning your bare metal machine in such a way that you've got the root file system, the thing that is ephemeral as it were and then the place where your precious data goes. And so then when you have a new disk image that you want to upgrade to you can actually just rsync-delete the contents of the disk image over the root file system and either do a full reboot or a kexec hop of that if a kernel update was needed or otherwise you can just manage restarting services as needed, but you should know in that case that your root file system isn't something that you're changing outside of the context of the tooling and the data that is being changed by your application is going into the long-lived volume. There's some gotchas to that but I have babbled about other things for too long so I would have to defer to that. There's a couple things we need to do to make that better. One of them is local sender volumes which is the sender is the volume API service inside of OpenStack and we need a way to tell the system that when you redeploy a machine that has requested a particular sender volume that that actually needs to attach to the bare-metal machine that it was running on before because what it actually is is a partition on the machine and it's not actually something that's being connected over the network via iSCSI, waving hands but it's a thing that you need to be able to reattach to and also in some of these things we're actually net-booting kernels so to do kernel upgrades, you actually need to be able to tell Nova, hey, reboot this machine with a new kernel but please don't mess with the disk image that's there. I just want you to reboot it into a new kernel and so those are a couple of the to do those tasks and I only babbled over time for just a little bit. That's sort of what I've got for today. We're out of time for the questions, right? I have two minutes for questions. Woohoo! I just want to thank Monty. Absolutely, thank you. Yeah, I'm always, I'm always available for... A couple of questions. So, the, the Nova Group Metal stuff is in now and is functional as of Havana. The, the specific triple-O tools, OS Star config and disk image builder and stuff like that, they all, they all work right now. They are, they are a official program currently. I'm not sure that we've, we've defined how utilities produced by a program such as triple-O like OS can apply config might relate to release cycle. So, they're, consider them a little bit orthogonal from the, from the open sacrileases at least right now as we're figuring that part out. But those all work at the moment. Ironic probably won't land as, as usable until, until Ice House. But that's the, the goal is to get that landed for Ice House. And, and then yes, the other ones are more about, about trying to orchestrate this rather than being, most of those are utilities rather than, than things. The, the, the features we need from Neutron in order to do the, the DHCP and Pixie booting appropriately in the bare metal context has landed for, for Havana. So there's, there's no changes needed in the core open stack projects that I'm currently aware that we're blocked on in terms of being able to use this for, for real deployment as of the Havana releases. What's that? Heat, I don't, I believe there's one more upgrade thing that needs to happen to heat before you can do the non-destructive upgrade version of it. For destructive, blow one away and give me another one upgrade heat can, heat can already do that and, and knows how to do that. So it's, it's a little bit, there's works and then we'll get better is sort of the, you know, as those features land. Yes, these, these will be up on a slide share somewhere and then all the code is, you know, on the online. So, absolutely. Oh gosh, should really learn to stop saying things about my employer on these things. Great, thanks a lot.