 I'm going to talk about provisioning bare metal resources, not using Razor, Cobbler, Mass, things like that, but actually using OpenStack itself. My name is Dave Ananda of Andriven. I work at HP. That's the blue slides. And you probably heard. You probably heard. OK, you all probably saw on the keynote this morning Mark McLaughlin talking about Triple-O. Triple-O is based on the idea, the functionality of bare metal provisioning. So I'm also the PTL for the Incubated Ironic Project. I'll tell you a little more about that later. But for those of you who are part of the community or were at the summit last summit or the one before, that's what I used to look like. I cut my hair off. You might remember me like that and I've had a lot of people be surprised and not recognize me. I'm mentioning this to say that times change. People change. Software changes. We're developing a lot of new things. We're breaking a lot of new ground with OpenStack in data center management, in operations DevOps, but everything we're doing with bare metal provisioning, with Ironic, with Triple-O, this is based on technology that's existed for 10, 20, 30 years. About 15 years ago, NASA published a paper titled Bootstrapping and Infrastructure. This is about the time of the 12th USENIX conference. About the same time, IBM published the specifications for IPMI 1.0 and Pixie, network booting. These in turn were based on much even older means of network booting, remotely controlling machines called RIPL for booting DOS workstations in about 1990. That in turn even based on older work. So we're not really using new technology, we're just adapting it in new ways. People have always wanted to manage large amounts of machines. What is new that we're doing? Clouds seem new. Everyone's talking about clouds these days. It's a lot of hi-end, but maybe there's really something to all this cloud stuff. What is interesting about clouds? Why are we also interested in it? Clouds give us an API to programmatically allocate resources, virtual resources, you know, compute, storage network, et cetera. I'm gonna borrow a phrase from the keynote yesterday, but actually this isn't a virtualization, this is an abstraction. We're abstracting resources. Whether they're virtual or physical, doesn't matter. Now on top of this abstraction layer, everyone's written various tools for deploying applications, for describing whatever your application is and then repeatedly programmatically deploying it. In a test environment, in a production environment, in one data center, somewhere else in the world, you can be guaranteed it's the same result because it's a programmatic API. You're not doing things by hand with small variations here and there. That's the real benefit clouds are giving us. Less errors, more automation, less skew over time. But it turns out, actually building a cloud, if you're building your own cloud, it looks more like that. It's much more complicated under the hood. And so a lot of people are still using homegrown environments to build their cloud and then all the benefits, DevOps, automation, good testing is on top of the cloud, not in the cloud, not the cloud itself. The cloud itself is not abstracted. And so a while ago, maybe 18 months ago, we all had an idea. What if we could simplify deploying a cloud? What if we could use the same tooling, the same automation that we use for DevOps, running things on a cloud and deploy a cloud, repeatably, testably, programmatically. What was missing from that was a way to control physical resources with the same API. NTTDocomo, USCISI, Virtual Tech, Japan, I have some familiar names around here, had proposed a driver for Nova at that point in time to control physical resources, bare metal resources. They were using it for high performance compute workloads. They didn't have, I don't know, we all got together in the Grizzly summit, I think it was, and kind of had a pow-wow, and so I took over, and some of the guys I worked with at HP took over for them in getting this into Nova. It landed in Grizzly as the bare metal driver, and that's sort of how it was born. And then the triple O team started around the same time with this concept of leveraging the bare metal driver to do something else, to automate deploying a cloud. In order to get this to work, we had to change several things in Nova. So we had to create a separate database to store the physical information. The instances aren't VMs anymore, they're real machines, and so the real characteristics of those machines had a separate database schema for that. We had to make some changes to the scheduler to make it aware that a resource is not partially allocated. You can't slice up a physical machine, it's all or nothing. You don't get to, you know, just like with VMs, you can get small VMs, big VMs, with bare metal, it's the whole machine. There is no hypervisor. Oh, flavorX respects. So we had to export some other data about the machines such as what CPU architecture they have to match the image that we're gonna deploy on the machine. Store some extra image about metadata, sorry, extra metadata about images, and we added a new helper service for the actual writing of the image onto the machine hard disk, and I'll talk more about that a little bit later. The process for this Nova bare metal deployment, it's still all done through Nova, and command goes to Nova API, Nova boot this thing, Nova scheduler picks a node, the command goes down to Nova compute, Nova compute pulls the image from glance, issues commands via IPMI to turn the machine on, issues that the machine turns on, it's DHCP boots, gets a response, pixie, and the image gets written to the hard disk, machine reboots, there you go. The machine is now running the image pulled from glance, no hypervisor. Simple version. Limit. So there's a bootable RAM disk. This is an extra component one has to build using the disk image builder, or if you want something else, but really it's easier with the disk image builder project, build a small bootable RAM disk that is fed to the machine as the first response to pixie, that exposes the local disks over Iskasi. This extra service I mentioned, Nova bare metal deploy helper, mounts that Iskasi target, copies the image onto it, partitions it and so on, the total time for this process is approximately two power cycles of your machine, whatever that is, and two times the network transfer for that image. Slow network, slow disks, a little bit longer, but what we've found in production really is that the image transfer time is very, very negligible compared to the power cycle time of hardware. Post, it can be three to five minutes for some machine. So looking at about 10 minutes per deploy, which is all just the hardware power on. And again, if you're in the back, I see people standing in the hall, so try and scoot up a bit if you can. So everything I just described, for those who are more visually inclined, looks like that. I don't think you can see my little laser pointer, but in the top right, step number one, Nova boot, that's just a regular command to Nova API, that trickles all the way through Nova scheduler, which goes out to Nova compute, which talks to Neutron, talks to Glance, writes in the Nova bare metal database, and then uses IPmine pixie to control the physical machine with the help of the Nova bare metal deploy helper service. Now, all of that process I just described is what happens for a single instance, when you're deploying a single bare metal instance. Orchestration lets you do many things at once, like you describe your application broken down into many different resources in heat. Because there's a cloud API on top of bare metal now, you can have heat orchestrate a deployment of many bare metal nodes in parallel. The metadata for each node, whatever it takes to parameterize that node, its host name, its IP address, other endpoints it needs to talk to, maybe it's a WordPress website and you want it to turn on and know where the minuscule server is it has to talk to. All of this can be done with heat passed in through the cloud metadata service, and we have, you know, you can use Shrapp or Puppet or Salt to do this. You can also use some tools that are part of triple O called OS config apply. And the triple O guys will go into way more detail about that than I will right now. The point is because we have now automated deployment on a single node, you can orchestrate deployment on many nodes. And so what kind of workload can you orchestrate? You can orchestrate a high performance compute workload, which is sort of what the original purpose of the bare metal driver was. You can do metal as a service. You can have a cloud where the instances that are being booted are physical instances with no hypervisor. However, the caveat only do that for trusted tenants. People inside your company, different divisions. Because there's no hypervisor, there is many different avenues for attack that don't exist in a virtualized cloud. So do not please do not use bare metal to provide a metal as a service cloud for public users. This is bad right now. We may add better security to prevent firmware attacks down the road, but today, firmware is vulnerable. Of course also, you can use this to deploy open stack itself, which is triple O, which I think is really, really cool. I'm not gonna talk too much about it though. Louder? Okay, sorry, I keep turning my head. Is that better? Okay, so all of everything I just talked about is basically in the Grizzly release. But really, the Grizzly release was just the first cut of this driver. It used DNS mask instead of neutron, which runs into some problems if you wanna do anything besides a flat network. It had a lot of bugs that we ran into as we use this in our test environments, but it was really sufficient to prove triple O is viable. Havana, which just came out, is much better. We fixed a lot of the bugs. We've been running this now on hardware in our test lab for six months. It uses neutron instead of DNS mask. So that simplifies the use of this. It also opens up much better support for software defined networking. You have a switch, an open flow enabled switch, and you're using neutron now with the code around Havana timeframe. You can have some better security and more options for your bare metal deployments. And basic functionality of a bare metal deployment is there in Havana release. However, is it really ready? Well, I mean, like I said, we are deploying this and triple O is actually deploying this on their test cluster about 24 times a day on bare metal. And they're not seeing that many failures, except when they break things, the orchestration layer breaks things. Bare metal driver's working. Some companies, I'm told, are using this in production or in tests. I can't say who, but they poke me every so often to say, hey, look at what we're doing. But all of that, I'm still gonna say no, it's not really ready. Why? Because the Nova compute process itself, running the bare metal driver, that is a single point of failure. There is no way to recover if that were to go away. You'd have to destroy the instances that you've provisioned on bare metal and build them again. You lose control of them. There is really not good support for different disk layouts right now. Whatever the flavor in Nova is, it specifies here's your root gigabytes, here's your swap gigabytes. That determines the disk layout right now. We have plans to change that. Very rudimentary right now. There's no hardware configuration built in at the moment. Again, something we want to do, but right now, if you wanted to use this in production, you'd have to have some other out-of-band mechanism for firmer updates, changing bio-settings, building a RAID config. Mixing different types of hardware gets a little complicated with schedulers, sorry, with exporting properties to the Nova scheduler in just the right way. And it's easy to mess that up right now. Again, things we're working on. Basically, the bare metal driver in Nova is very limited in what it can grow into, because it's part of Nova. And Nova wasn't really meant for non-virtual things. So last summit, I proposed that we rip this out of Nova and make it its own top-level service, part of OpenSack. Between then and now, the technical committee approved and the Ironic project was born. So again, I think kind of a fitting name. It was Portland, Portland was sort of the headquarters of hipsters to love irony. So where are we at today with Ironic? So that commit is actually where Ironic started. It's the very first commit in the project. You have a date there that's roughly six months ago. In those six months, we've grown from 8,000 lines that were copied out of Nova to over 32,000 lines of code today. We've got 28 developers, about half of which have more than five commits. So we're not just seeing people drive by and we're seeing people more than 10, about a dozen developers now jump on this project and contribute. And I'd love to see more. A very sizable portion of this work has not been coming from me. Coming from not just HP, but many other companies are interested in this. Morantis, Red Hat, IBM, Yahoo, a couple others that are all jumping in and helping out. And I would love more people to come help out with this. Particularly any of the hardware vendors, HP, of course I work for them and talking with them, Dell, IBM, AMD I think is here and has some interesting work they have mentioned to me. And I'm really grateful to everyone who's come and helped out and continues to help make Ironic a better project, make it more fully featured for bare metal provisioning. The architecture for Ironic now can really grow. It's been decoupled from Nova. It can grow to be a fully featured bare metal provisioning service for all the things that one might need in the data center. It's not gonna be a CMDB. It's not gonna store a long-term state and things like that. But hopefully it will also be able to do auto discovery and firmware updates down the road. Why does it have to be split out so that it can interface with other open-sack services? Cinder, Glance, Neutron. We're just talking with the Neutron guys upstairs about how Neutron and Ironic can interact to manage the networking for physical machines. So the plan for its interaction with Nova is that Nova will still do all the scheduling. Ironic is just an API to the hardware. You'll define the properties of the node in Ironic but it will export those to Nova so that Nova's scheduler can make the decisions about placement of instances. There's a driver in Nova that is underway right now to bind the two services together. That will talk to the Ironic API service and that'll, for all the regular Nova calls, spawn, destroy, reboot, et cetera. All the things which are not part of Nova's understanding, Nova's understanding, such as interrogation, discovery, et cetera, those will not be exposed to Nova but there will be a separate API or the same API but you'll be able to use those out of band. The diagram I showed earlier for Ironic looks more like this. You can see Nova off on the side and how Nova Compute will talk to the Ironic API which will have its own RPC layer to the Ironic conductor service with its own database controlling the hardware and this service separation will also allow us to do, to make the service, the Ironic service, highly available whereas with Nova you have a single point of failure in the Nova Compute process that is sort of proxying the bare metal commands. Our plans for Ironic is to make the Ironic conductor any conductor able to take over for another conductor if it failed. So reads and writes to the database to enroll in your hardware. These will go through the API to get passed to the conductor for privilege separation. The conductor will use a distributed locking model such that you could run many conductors and have many nodes and each one would be determining at any given moment what action to take and not to step on each other's toes. And of course we have some growth to do these are all in the plans. It already has a pluggable driver interface internally for different hardware vendors so we have basic drivers right now for IPMI and Pixie. And I'm encouraging hardware vendors to come and add additional functionality besides the basics of power on, power off, deploy, console that we have to support for Nova today. There are other functionality that hardware can do like discovery firmware. I know I keep saying those but they're the most requested features for Ironic. Autodiscover hardware and be able to update the firmware. So, the mechanism for doing that is a little bit different for each vendor. I would like to standardize the API that Ironic has but allow each vendor to use their drivers which are different, some may be faster than others. I don't know exactly how to do all of those. I'd like vendors to come help out with that. Each driver itself, there's a contract between the conductor and the drivers internally that they all must adhere to. And there's also a pass through mode for where vendors want to explore features that they have that maybe other vendors don't have instead of requiring Ironic to make everyone standardize the same way and preventing drivers, preventing vendors from taking advantage of their own features. There is a special bypass area for that kind of thing. What else is on the roadmap? Like I said, hardware discovery. This is really, really strongly requested feature. Same thing for firmware updates. Very strongly requested feature that we probably won't get to either of those six months or more from now. We need to get a stable service first and the real roadmap is hopefully get Ironic stable, bring it out of incubation and have it replace the Novogram metal driver. After that, begin adding many of these new features. I know I've said many times I would like more involvement. I'm inviting everyone who's interested to come help out with this project, add drivers, here's some links that help. Everything's like any Opensack product on Launchpad, on GitHub, come join our IRC channel, either Opensack Ironic or Triple-O. And I think I went through my slides much faster than I meant to. So, thank you. All I have time for questions. Any questions? Silence. Yeah, yeah, just. So I was talking with, Oh, sorry, thank you. The question is what's missing to make this work on Windows? And the answer is I don't know how to network boot Windows and not an installer. But I've been talking with that fellow over there who says it's actually much easier than I thought and he's gonna come help us do it. So it could happen, Windows support could happen pretty soon. Yes. Okay, so I think the question is if you know what servers you have and when you're deploying an instance, how do you target a specific server instead of letting the scheduler pick one? Yes? The answer is we already added that functionality to Nova. It's in Havana, it's called dash dash force host and we extended the host definition from what used to just be availability zone colon host. Now it's also colon node, which is a UUID that identifies that machine. So you can do that today and we'll preserve that functionality as we move to Ironic. The question was when I say we did that already, what am I referring to? I'm referring to when I said we already did this. So when I said we already did this, what I mean is it's in the bear, it's in the Havana release supported with bare metal to target a specific node. Yes? Yes. The question was the conductor drivers in Ironic, what is that meant to target? It is meant to target the variations between hardware implementations of remote management. So IPMI is common, but not everyone adheres to standards in the same way and everyone deviates to add their own functionality like remote mounting virtual media. It's not part of IPMI spec, but everyone does it. So the driver is meant to encapsulate those differences and Ironic is the abstraction layer on top of those hardware differences. Yes? I didn't quite understand. Logical partitioning of the disks or of the system itself. Okay, so yeah, I think other vendors have similar things and so we should all talk about how to support this in the Ironic API. Yeah, no, so how do you install if there's no RAID configuration on the machine? Currently you have to have some out-of-band method that has pre-configured the machine, right? And this is one of the limitations of the Nova Bear Metal Driver is it doesn't have any way to model that. And so with the first iteration of Ironic we're just gonna have future parity. You'll still need some out-of-band RAID provisioning. The goal is for Cinder, and I need to start, I'm talking with the Cinder guys tomorrow about this, for you to be able to specify a volume of specification in Cinder. I describe it with the Cinder API. Pass that to Ironic and then Ironic would either through the out-of-band management or through a bootable RAM disk with, I don't know, make a CLI in it or something. Go set up the RAID for you before the image is put down. Turn-based provisioning? Okay, so I think what you're asking about is how do we optimize the image distribution? We'll talk about that tomorrow. I don't have a plan. I want people to come together and figure out what the solution should be. And it's the same thing with the RAID specification that I just mentioned. That's my idea, but it isn't implemented yet. And if people have different ideas or wanna come contribute, we're gonna discuss that tomorrow. The Ironic design track is roughly two to six tomorrow upstairs. Question over there. Not yet. Question is, do we support booting from Cinder volumes? The answer is definitely not yet. It won't fit in November metal, but I'd love to support that in Ironic in time if people wanna come at that. Over there. So the question was for monitoring the nodes. Do I see integration with Cilometer and Ironic? The answer is yes, and I spoke to them yesterday about that and we all are on the same page and it's awesome. And if you wanna come again. Yeah, yeah. Yeah, yeah. Yeah. What does Ironic do if the hardware doesn't behave the way you expect it to? Very good question. We have to make sure something reasonable happens in that case. Right now we're working on better representation of state change failures and exposing that to the API. Like you asked a node to turn on, but it didn't turn on. How we are exposing that exploitation mismatch, those errors. Ultimately it's hardware, you know? Sometimes it doesn't do what you want. Is there any community interest and ARM support? Anyone here have interest in Ironic supporting ARM? A few hands went up. Yes, it seems there is. Well, if there are no more questions. Thank you all very much.