 So, just as a note, a lot of the stuff we're going to be talking about today, in the early parts of the slide, we're going to tell you how some things work. If we're talking about on-metal decommissioning, this is what we're running, not what we're doing upstream, and then a little later we'll talk about what we're going to do upstream. So first of all, who are we? Well, we're the Rackspace Teeth team. Our team motto is to destroy proprietary control of the data center. And for us, that means using open software and open hardware wherever possible to commoditize the data center. So all the hardware we're going to be talking about today is open compute hardware designs. All the software that runs this, in so much as we can, is completely open source. Even code that we may talk about today that we're running downstream that is still in Garret, you can go look at it today, you can run it today, come into OpenStack Ironic Channel will help you run it. What does the Rackspace Teeth team do? Well, we built Rackspace on metal. It's built on top of OpenStack Ironic, and it integrates Ironic with our public cloud. So if you're a Rackspace customer, you can call our API the exact same one you would to get a virtual machine, use a different flavor, and instead you get a piece of hardware provision to you in minutes. So what is Ironic? Well, it's not 90-spot music, much like, you know, it's not rain on your wedding day or anything like that. It's actually the bare metal program for OpenStack. So we kind of want to bring a little rock and roll to this rather than kind of going with the nostalgia. We want to move forward, talk about new stuff. So how should you think about Ironic? Well, I like to think of Ironic as a bit of a bare metal hypervisor. On almost any Nova hypervisor you use today, there's going to be an API there. You have, in our example here, you see that Nova with Xen, you have Nova API, receives a call that makes its way through Nova to a Nova compute. That Nova compute calls a Xen API, which has all kinds of internal state, just like Ironic would, except for it's hidden from you, and it builds a VM. With Ironic, it's the exact same model. You've got an API request to a Nova API that eventually gets to a Nova compute, and instead of having a Xen driver loaded, it has an Ironic driver loaded, and instead of knowing how to talk to the Xen API, it knows how to talk to the Ironic API. So this does fit a general model of what you might have thought about from the way Nova works today. One of the big notable differences, though, is if you see we have a box over here around our Xen setup. That's because if you're using a traditional hypervisor, generally your Nova compute, whatever hypervisor API you're calling, and your virtual machines are all on the same box. However, with Ironic, you're going to have independent Nova computes. You're going to have a completely independent Ironic environment, and then completely separate bare metal nodes to deploy to. So there's a little bit more orchestration involved. And obviously with a hypervisor, you sort of have God access over the VM. You can do whatever you want. You can set up things. You have that kind of in-band access to do things. But how does Ironic do this? If you've got a bare metal node, how would Ironic control it? And there's two primary methods. We generally refer to them as in-band and out-of-band. When we're talking about in-band methods, we're talking about a RAM disk that we booted on the server that is actually running from the perspective of an operating system. And some of the things that are done almost exclusively in-band today would be disk imaging, whether it be actually writing the image to the disk or exposing the disk as a nice fuzzy target to be imaged, and disk erasure. There's also out-of-band, like your typical, your BMC, your management controller, and ILO, a DRAC. There are many, many synonyms for this. I'm going to be calling it the out-of-band during this talk. And the out-of-band generally is used to handle power control, what device you're booting from, getting a serial console working. And then there's sort of this place that's a gray area where you can do some of these things in-band or out-of-band depending on hardware. Those would be like firmware flashing, BIOS updates, and BIOS settings. So these are the tools in Ironic's arsenal for getting a tenant workload on a machine, getting that off, and then cleaning it up. So we'll be talking about these tools as we go throughout and how we're using them to clean up the nodes. So as I said before, this is kind of a continuation of the story that Jim was telling yesterday of us wanting to build on metal and looking at Ironic. We originally went and did our own thing. We called it Teeth Overlord and Teeth Aged. We pretty quickly realized with the urging of some of the more senior NOVA people at Rackspace that maybe wasn't the right path to go. So instead we went to Ironic. But we saw there was a lot of stuff missing that we really needed. And so I want to talk about this a little bit, and then we'll dig deeper into one of those a little later. So what was missing? Well, we need a boot from disk. The existing driver when on metal was created was predicated on pixie-booting the instance every time, which actually is kind of an HA problem, because if you have a control plane outage of one of your conductors, that instance, if it's rebooted during that time, might not be able to come back up, which is not a good customer experience. We wanted to support whole disk images. We wanted to be able to pre-boot those nodes to avoid a post-it deploy time, whereas Ironic today, nodes are expected to be power off when they're deployed to, and that saves us three or four minutes of provisioning time on every server that's provisioned. We also wanted a more complete RAM disk that included some fleet management tooling. The RAM disk that comes with what today in the Ironic code is known as the pixie driver is actually built using disk image builder, and it's very slimmed down. But if we're going to be managing a large fleet of nodes, we need a way for our ops people to get in, diagnose hardware problems, et cetera, so we wanted those fleet management tools in there. And finally, and what's going to be the focus of our talk today, is cleaning up after an instance is deleted. Ironic didn't do this. So what did we do to kind of try to solve all these issues? We built IPA, and then no, not the beer, the Ironic Python agent, which is a REST API for managing bare metal nodes. So we boot a RAM disk, that RAM disk does a look up in Ironic to see what node it is, and then exposes a REST API that Ironic can call to do those in-band control actions we were talking about before. And along with that agent, you also have to have a driver that's partnered with it, so we created the agent deploy driver for Ironic, which is what tells Ironic how to communicate to the agent and use it to deploy. So why IPA? What's so good about this that made us want to write it? Well, the big thing for us is that IPA is pluggable. Hardware is wildly variant. You might be running hardware in your data center today that I've never even heard of. So we wanted a model that would allow you to integrate your own needs into an agent that Ironic talks to. And so what are the two key parts of this? One is that we have a hardware manager interface, and you can override almost any stock IPA function that touches the hardware with your own in a downstream hardware manager if you choose to. And also this was designed with custom RAM disk images in mind, and today we have one pixie bootable RAM disk and an ISO, and we have another one coming. So let's talk about those RAM disks. The one that exists today, and we actually have a job that builds this. So if you go look at it, if you want to play around with this, we do have a job built on every commit for IPA and publishes the tarwalls at OpenStack. And that RAM disk runs under CoreOS. We find that CoreOS is very easy to customize. It has a lot of tools for OEM-ing things into your own images, which we needed to get some tools in there. And it does have a complete user space. We can SSH in there. We can do things to the operating system, diagnose hardware problems, troubleshoot deployments, et cetera. And also it has a real-init system, menu system D, which would respawn any agent should they crash that we don't have nodes left in a weird state. There is a disk imageable to RAM disk that's in progress. The review is linked on the slide, and you can also find our slides online. I'll put that back up at the end of the talk. The main thing that the disk imageable to RAM disk is going to be a lot better for is people who have memory constraints. So the key place here is the gate. So today, the agent driver CI runs with one gig VMs, whereas the normal ironic CI runs with 512. So we do want to get that disk imageable to RAM disk. We can kind of slim it up a little bit and get that working a little better. And I mentioned before we supported ISOs. Well, I don't mention an ISO on here because the ISO support can take an arbitrary kernel and RAM disk, put those into an ISO that's suitable for booting on virtual media. And that support exists for the Pixie driver today as well. So I've told you a little bit about the background, what we've been doing, why we've been doing it. But let's talk about what this means for on-metal today. Why do we need it, first of all? And just as a note, you might see or hear us say DECOM or decommissioning interchangeably here. That's primary from one of the major lessons we learned, but we didn't make a slide about it because it's a little embarrassing. It's being very hard to spell. Very hard to spell. So we use DECOM as to avoid the squiggly lines that makes PowerPoint unhappy. So why do we need this DECOM support? Well, deploying to the cloud is all about having your users have a stable, consistent platform. And you can't guarantee that consistency unless you've made sure that nothing's happened while that server was provisioned because your hardware state may have changed or failed while it was provisioned to the last tenant. And you also want to make sure that any remnants of that tenant are gone. And so I have a reference up here to a bug from Grizzly that was filed against the Nova Bear Metal Driver about 18 months ago, saying that tenant data was left on disks. We're here to save you. We're fixing your bug. So what should DECOM do? Well, DECOM could do firmware upgrades. And in fact, pictured here is one of the IO cards from our IO flavor, which every time we DECOM, we put a firmware update on that to ensure consistency. It should secure erase devices, devices such as the 32-gig data stated on pictured. Binding anomalous conditions and components. And this is actually one of the more interesting things we've used DECOM for so far. Josh actually built out a step in decommissioning that actually verifies that the ports that our servers are plugged into are accurate based on what Ironic thinks, which is very important to compare the real world to what Ironic thinks. Because obviously Ironic only has those in-band and out-of-band tools, whereas the data center ops people have 10 fingers and two hands. Much better tools. So it could also flash or configure a BIOS. And this is again one of those things that kind of falls over to the out-of-band as well. But there's also some stuff that DECOM shouldn't do. We really don't think DECOM is suited to doing burn-in testing of hardware that that might be something better done with a heat template. It's not for tenant-specific configuration. We had a design summit session about that this morning on hardware capabilities and so that would be the right to do for that. Or really discovery of newer changed components or any sort of hardware interrogation to find out what's going on. That again is being handled outside of DECOM as a part of another effort inside Ironic. So what we're going to do now is Josh is going to sort of zoom in here and talk about how Ironic and the agent collaborate to decommission nodes. Thank you, Jay. All right, so this is the same slide that Jay showed earlier with how Novel works with Ironic. But I'm really going to zoom into the bottom three boxes with Ironic and the nodes. So how does Ironic work? Ironic's your basic distributed system. You've got APIs and conductors in a hashing. The basis for Ironic is that there are drivers. You need a different driver for the different types of hardware that you have. So you have ILO drivers, DRAC drivers, stuff like that. And then there's deploy drivers which manage how you do the deployment onto the disks. And it also works with decommissioning. So in this example, if you called Nova Boot, the Nova Boot goes through Nova and then it calls down through the Nova driver to the Ironic API. The request then, the API will look up the node in the database, figures out which driver is working for it, and then finds a conductor that can handle that driver and passes it down to the conductor, or finds a conductor that can handle that driver, sorry. And then that conductor shells out to the drivers and the drivers are what actually talk to the nodes. So in this case, I have IPA and IPMI would be one type of driver. That's what we use in non-metal. But you could also have like a Pixie and ILO driver talking to some HP hardware. So how does a node actually get provisioned in Ironic? So here's a state diagram. Nodes that are powered off, ready to be provisioned to, start in a no state. When you call Nova Boot, they move into a deploy state. That's where the driver writes the image to the disk, gets a config drive on there, stuff like that. Once that's completed, then it moves to an active state. That's where the tenant is on the node. They can do whatever they want with it. They can keep it for as long as they want. And then when they call Nova Delete, the node will move into the teardown state. The teardown state doesn't really do much. It does some internal cleanup on the conductor. But after that, there's no verification or anything. It goes directly into a no state. And in this instance, new nodes like if you rolled into new rack and wanted to get it on Ironic goes directly into that no state. So we want to do decommissioning. So we felt the easiest way is to add a new state in here. So we added the decomm state after the teardown. So same as before, you have a tenant in an active state. They call delete, it goes into teardown. Teardown does a very minimal set of steps. And at that point, the node is removed from the user's list of active instances. We don't want decomm to take like four hours and your node's saying deleting for four hours. Customers might not appreciate that. So after that, it moves into a decomm state. Decomm is where we do all the work that we're going to talk about here. And then after decommissioning is done, it moves back into the no state. And it's ready for provisioning again. At that point, we know the node is working, the hardware's working. It's ready to be deployed to its network and it's working. And that's the reason we think for a new node, it might sound kind of kind of retuitive, but you should put a new node into decommissioning first. Because that'll really verify that the node's working. It could also be called ready state, and that's what we might call it upstream, but. So how does IPA fit in here? Like Jay talked about, we have a hardware manager. The hardware manager is a thin, is a small set of Python. It's got a Python API, basically just functions like write to disk, do decommissioning, stuff like that. It has a list of decommissioning steps. So each hardware manager can determine how you want to decommission something. And then we have vendor tools. So we have a tool that flashes the BIOS. We're obviously not going to put that in the upstream because you're not using the same hardware as we are, or you could be. Or tools to erase our IO cards, stuff like that. And then we also include some versioning in there. This is kind of important, you don't want to flash an older BIOS. And then the next step, you boot a new agent with a newer version of the decommissioning, and all of a sudden you're applying settings for a new BIOS that you don't actually have and everything goes crazy. So we had versioning in there to avoid that. And these hardware managers are broken up into pieces. Right now there's a generic hardware manager, and this is just all the things that we expect any piece of hardware should be able to do. So it's like write something to a disk, erase that disk using secure ATA erase, choose which disk you want to write the image to. So like in our example, or in our case it's any node bigger than like four gigabytes, we don't want to write to like a floppy drive or something. And then we have our own hardware manager called the Unmetal hardware manager. This is a big monolithic piece of Python that does all the steps that we need to provision and decommission nodes. It's monolithic, it's not great, but it works. So you may be asking how does IPA and Ironic work together with hardware manager to do DCOM? Here's a decent diagram. Basically, once you get into that DCOM state, Ironic is going to call DCOM, that comes down to the driver again. The driver is going to talk directly to IPA and tell it to start DCOMing. It's going to pass a target state to the agent. And in the first step it'll be none, which signifies that it's the first state. IPA is going to list all of its DCOM steps, find the first one, execute it. And then IPA is going to report that it's finished that. It's going to report what the next state is. And optionally, it can ask for a reboot. So IPA is running in-band. And the only way you can reboot the node is like pseudo-reboot. That's not good if you're doing something like upgrading a BIOS or a firmware or something. You want a nice hard power off from the out-of-band. So Ironic has to do that. And this is one of the problems that we ran into. Anyway, so it passes that target state back up. And this is really key because we're running a RAM disk for IPA. If it reboots, it loses all of its state. It doesn't know which step it was on. You get into an infinite loop, doesn't really work very well. So Ironic runs DCOM again if there's a step. It tells it what the next state is. It runs, continues, until Ironic says it's done. And those steps that I was talking about look something like this. This is what we have, this is just a basic version. The key part here is that there's a priority. We really want to run these steps in a certain order. So we order them by priority. Like you don't want to apply the settings for a BIOS. If you have a BIOS update coming. So you want to do the update BIOS first. So that would have a lower priority or higher priority, whichever way you want to look at it. And then the settings would be after that. And then they can also request a reboot each step. So that's the basics of how decommissioning works. Sounds pretty good. Jay is going to tell you how it actually worked. So thanks Josh. And that's what's running today. If you deleted an all metal server right now, you're going to get a decommissioning that looks somewhat like that. We haven't listed every step we've done because quite frankly some of them are quite boring. But this is open source. And I believe on a later version of that slide, there's a link to our hardware manager we use. But let's talk a little bit about the real world. This isn't a dream. This is something we're doing today and we're doing it successfully. In fact, on metal runs DCOM over 1000 times every day. And we do this with great success. We see a less than 0.1% failure rate when running these. And those failures are usually environmentally related and are simple to clean up. This is really important because it has to be reliable. Humans aren't reliable. If you give your ops person a set of steps to clean up a server, they might not always happen reliably. But ironic today with IPA is doing this and it's successful. But let's talk about when it's not so successful. So we have a few hardware shortfalls we've fallen into. And one interesting thing to note is that we didn't see this in our hardware lab. We didn't see this in our small staging environment. Only when we started to scale up did we see these issues because they occur at such a small percentage rate that they just never emerged in any other environment. So like for instance on our hardware, updating BIOS automatically has caused some issues. Anything from freezing the system up long enough that the agent was restarted by system D to sometimes even making our out of bands go away and require a hard power cycle. And honestly, we even had a few nodes that we were forced to RMA due to these BIOS update failures. And so it kind of reinforces the idea that if you're going to run something in DCOM, the very first step is not try to run DCOM and Ironic. It's try to run DCOM yourself. You have to know your hardware. You have to understand your hardware in order to successfully clean it up. So what else? Well, we kind of tried. I talked a little bit about our verify ports. We attempted to implement a verify properties as well which would make sure that the amount of RAM, the number of CPUs in the system is consistent with what Ironic's reporting. But we ran into a bit of a problem. We found that different manufacturers, even for the exact same spec memory ship, report subtly different values for the amount of actual bytes of RAM on the node. So the fix for that would be to allow a small percentage variance in reported memory when you have a large cluster that might have hydrogenous memory manufacturers in it. But we have not yet implemented it with that fudge factor. That's something that we still have ahead of us. So the other one and anyone who's had to do some of this will sympathize is that vendor utilities don't work well on a RAM disk. Now I mentioned our RAM disk is based on CoreOS. CoreOS generally runs pretty close to vanilla kernel. So you're talking about, we're running utilities. So for instance, your BIOS flash utility, according to the open compute spec which I've linked here and I'm citing, the OS standard is CentOS version 5.2. And as you know, if you have a piece of software that's written by a vendor to run on CentOS 5.2, it can be pretty fun to get that to run on anything else. In our case, we actually had to do some extensive testing. We found that just simlinking the newer version of the library into the older version's location was sufficient in our case for getting this resolved. But this is just kind of a warning. Your environments are not always gonna be the same as maybe your vendor feels like. So those are some of the hardware problems we've fallen into. We have a couple of other states which come up rarely but we tend to believe that that has more to do with environmental issues than hardware issues. So what are some of the shortfalls of the actual software today? And this is what we did in on Metal. These are some of the lessons we learned as to what we wanna change. Right now what we're doing downstream it only supports the IPA driver. So that means if you have an ILO or a DRAC or something that allows you out of band access to do more interesting things during DECOM, that's not supported if you were running our downstream version. And there's actually some really interesting stuff you might wanna do with that, like rotating your BMC passwords. Given that it all happens in band right now, we can't do any of that. And as a patch that we're using downstream, we're not sharing it with the community. We really want, we think decommissioning is important to Ironic. And so we wanna have a solution that's gonna work across every driver in many people's use cases, not just what on Metal needs today. So let's talk about what DECOM's gonna look like in Kilo and for that, Josh is gonna come back. Thank you, Jay. Yeah, so obviously there's a few problems with what we did, it's very specific to our environment. We need to do something that'll support more types of hardware, more types of decommissioning. So we're gonna change it up a little bit. So this is the slide that I showed before that shows how hardware managers work. You've got a generic and a monolithic on Metal hardware manager. The easiest way to fix this would be to break it up into multiple type, multiple hardware managers that get mixed together. So you still got your generic that does all the basic stuff. And then maybe you'll have a hardware manager for your NIC to do some upgrades, your storage devices, maybe a JBOT or something. You'll have a BIOS, so you'll have like an HP BIOS flashing utility, a Quanted BIOS flashing utility. And then different types of verification, whether you wanna verify properties, ports. The stuff that we were using to do the verify ports is very specific to the switches that we use. They use some vendor extensions for LLDP packets. Yeah, won't work across the board. So you'll need something specific for your own hardware. And we think there could be a decent ecosystem for this kind of stuff, like pre-built RAM disks with all the hardware stuff that you need for your specific platform. The other main thing that we need to do is change how the steps are created for decommissioning. Before it's almost completely driven by the Ironic Python agent. Obviously that won't work if you're not running IPA. It also doesn't give you access to the out of band stuff. So our proposal would be to do all of this in Ironic itself, push it up the stack a little bit. So each of your drivers will be able to submit a list of steps that it wants to run. And with the same priority ordering that you had before. So in this case, for our hardware, you would have the IPMI management interface, submitting a list of steps like rotating passwords. You'd have IPA listing all the steps from your hardware managers, like erasing disks and updating firmware is there. Or if you had the DRAC and the Pixie driver together, you could also do any steps that those require. So in Ironic, how it's gonna work is you'll get the list of steps from each of the drivers. Exactly the same as before, you're gonna sort them by priority and then you execute them in order. And then you're gonna pass those executions down to each driver so that they can manage how they wanna do it. For example, IPA is gonna have to talk to the Ironic Python agent. ILO has to know how to talk to an ILO. And it'll go through the same step with the same, like I said, the same priority. In this example, the first thing that you're doing is updating a firmware on your ILO. And that'll happen out of band. And then you are asking the IPA to upgrade your BIOS and erase your devices. So now we can support in-band, out-of-band, we can support pretty much any hardware that you can write a driver for. There's already a pretty rich ecosystem of drivers. So this should be a pretty simple fix. And that pretty much covers what we wanna change up in ILO. Help us? We don't wanna do this all on our own. And we have a design session tomorrow at 9 a.m.? Yeah. So one of the nice things is that I really think is important to revisit is we said help us. Well, first of all, you've gotta help yourself. And so at the beginning of this talk, we talked about all the things that Ironic didn't do that we wanted it to do. Well, it doesn't now. Everything on this list, including the two that aren't merged, are things that on metal is doing in production today. We do it all the time, it's stable, it works. But boot from disk, support for whole disk images. A RAM disk that includes fleet management tools, that all shipped with Juno. The IPA driver is there, it exists today, you can use it. So the only two things left off of that initial list is a spec we call long running deploy RAM disks, which Jim has been working on, that is targeted to Kilo, as well as the spec for the decommissioning that we've been talking to you today. That has a spec up, we do have the code up for some of that as well, and that is also targeted to Kilo. But how can you help? We do have the design summit session for the decommissioning we're talking about. Today is tomorrow. So if you think our ideas are good, come, learn more about it, help us design it. If you think our ideas are bad, come tomorrow, help us design it. That's really what you gotta do to help is you have to participate in the process. And this includes operators. I have a strong system in an operations background and because it's so important to understand how your hardware works, especially the input of operators who've managed fleets before, you'd be very appreciated if you have an interest in this to come to our design summit session and help us understand maybe some of the pitfalls that we haven't run into yet. And also a general plug, the specs repose tend to scare people sometimes and feel like under this process. I've been reviewing specs and Ironic for about one cycle now. I find it great to know what's coming up next, to be able to have input into things and it's really great. And again, you don't have to be a Python pro to go look at those specs. If you're an operator, go look at them. If we have bad ideas, tell us and let's find something that works for everyone. So the other thing you can do to help is ask good questions, which that part is about to start now, but I do have a quick cross promotion. We actually discovered today that Peter from Bloomberg is giving a talk immediately after this on this same floor. It's not in this room. I believe it's on the big room at the end of the hallway where they have open compute hardware platforms and everybody talking a little bit about the nitty gritty of managing that. So if you're curious about that, you should go see his talk. We had a great conversation today and it sounds like a lot of his experiences mirror what we've seen as well. So if anyone has any questions, we'd love to fill it all now. It's having kind of like an alternate Nova boot right after you. In order to do that, you would have to prevent that node from being scheduled too. So for instance, in an environment where you have high churn, like in a typical public cloud scenario, you're going, there's no way to really guarantee that a node, once it's deleted, it goes back into no state, you would basically be racing to deploy your image to that node before someone else's image to that node. Also again, that would limit you to just the in-band tooling rather than the full suite of in-band and out-of-band tooling which as we talked about, we learned is really necessary to have a complete decommissioning experience. That's an interesting question. In fact, it's something I meant to mention earlier. Generally speaking, we've been using our vendor utilities that have verification function. So for instance, our BIOS flasher is verifying that the BIOS matches what we expect it to be and we don't flash it if that's the case. We do that with almost all of our firmwares. We do a verification of the firmware if it's already at the correct version for exactly what you're talking about. We have concerns about flash exhaustion. I don't know the answer to that question and I don't want to learn it the hard way. We've talked to our vendors about it and they advise that what we're doing is smart to not reflash it every time. No, no, like, yes, we're actually, it takes the, so the typical BIOS update would be verify, flash, verify. This does the verify if it matches the BIOS we expect it to be, it stops and it doesn't rewrite it. Our devices don't have TPMs. In fact, part of the benefit of using open compute hardware is that it does trim out a lot of unnecessary components which gives us a smaller surface staff to secure. A lot of that stuff is under NDA with us and our BIOS vendor. That's kind of what I was talking about when we can't authenticate it. We do have signed firmwares though. Every firmware that's on an all metal server is signed and when we do use that, we do also have some additional security mechanisms but like I said, I'm not really at liberty to talk specifically about those modifications. That's something you get to work out with your hardware. Absolutely, that's why, like we said, knowing your hardware is very, very important and it's doubly so for us running a public cloud with it. We worked very closely with the internal people at Rackspace as well as the vendors to make sure that we were able to do this in a way that we felt was secure. And in addition, this mirrors processes that happens at managed hosting providers throughout the world today already except for instead of having a human do some of these steps, we're having a computer do some of these steps which makes it more reliable and allows us to add more steps to it which we appreciate. Yes, in the back. We have an experience that problem, exactly. I mean, quite frankly, if you give us a credit card to spin up a very, very expensive server, we have a very good fraud team. That's not something that we've run into yet. I'm sure we will and when we do, we'll continue to work with our hardware vendors and we'll recover it. Like part of the reason for Decom and if any of these steps fail, the notice kicked into a Decom failed state and stays there until we've inspected it, we've interrogated it, we've seen what's going on and we've actually had situations we've specifically called hardware failures. So if you were running Ironic without Decom and you had that type of hardware failure, the node would go directly into no state and the next tenant that got scheduled to it would get a broken server. Instead, we get a node that fails decommissioning with a message as to how it failed. We also specifically today do not power off the node after that type of failure so the logs are still available on the agent for us to go see what happened. And this is basically the process we had to follow when we had the issues upgrading BIOS with our initial deployment. Oh, I think he got you first. It takes about 30 minutes. Most of that is secure race of the say-to-dom. We want to work more closely with our hardware vendors in a revision of hardware to help take that time down a little bit but we find it takes about 30 minutes that can vary based on how long posting time takes and between our three different flavor types there actually is a pretty significant difference in how long each one of them takes to post. These are cookie cutter instances. This is a cloud. We have nodes that look like every other node. Customers can't really change that. I do think that there's a clear need for some type of network attachment, attached storage there, kind of like you would typically have in a cloud using something like Cinder but we don't allow our customers to modify the hardware at all. That's the exact point of providing hardware in a cloud is we want it to be a consistent experience which means you have to trade in as I said in the keynote yesterday, you have to trade in your precious servers to get the speed and the API. And in fact, that sort of reflects how we expect people to use these servers. Yeah, we're looking forward to that support. To be honest, when we were originally planning this product, we almost wanted to not put disks in there at all and instead have them be completely ephemeral but we found that that was pretty difficult to implement. So the 32 gig say to Dom was what we did as a compromise at that point. Most of the trust to boot stuff I've seen so far is about a testing that an image on a specific piece of hardware behaves a certain way. We have our customers the option of using many different images as well as we eventually hope one day just like we do on our cloud to our custom images as well. So I think that that maybe doesn't fit very well with the technology that exists right now in the cloud but I can tell you that we're doing research every day to try to implement things that anything we can do to add extra layers of security. Yeah, that's actually true. We do have trusted boot support that's going on, Intel's working on that right now in Ironic downstream, we aren't doing it right now. So upstream. All right, well, that is time for our session. I'd like to thank you all for coming. Again, please come to our design summit session tomorrow if you have an interest in this.