 Good afternoon, everyone. This is about that time. I'm going to give another minute for folks to trickle in. I guess it's pretty full in the back. So I'm not going to give another minute, and I'll just start early. So I'm going to talk about Ironic. I assume that's why you're all here. My name is Dev Ananda Vanderveen. I work at HP Cloud. I started on working on OpenStack about three years ago. Yeah, springtime three years ago. I run the Ironic service team at HP and have been on the technical committee, though I just stepped down this last election because I needed more time to work on other things like Ironic. It's about it. What I want to talk about is, first off, just virtualization and OpenStack and some conceptions of those two things. And then I'll go into Ironic's architecture. If you want to deploy it, some config choices you have to make, what operations looks like, and what limitations we currently have in the KiloCycle, what's coming up in Liberty. And then if there's time, I can either go through, just talking through the process of a deploy, or I might actually be able to demo one here with one of our new tools we just built in this last cycle. First off, I just want to point out that OpenStack is not a virtualization layer. It has, yes, there's VMs. You can get VMs with it. But it is more than that. OpenStack is fundamentally an abstraction layer. That is what makes it so powerful. That abstraction layer includes physical machines and containers under the same API. That is my vision for it. The same things should apply to storage and networking. And folks are working on that as well. So a common abstraction layer for physical and virtual compute storage networking. I haven't started a talk with graphs before. But this is new for me. But I thought this was kind of an interesting correlation. Virtualization and cloud computing as Google trends, both already peaked. There is not, that is just a thing people understand now. People aren't searching for it. We understand it. It's in the common collective mind share. However, DevOps is not. DevOps, OpenStack, both peaking around the same time, or sorry, starting around the same time and still rising. I don't know that that's causative, but it's interesting. What do developers really want? What is it that people in the DevOps mindset, or who are trying to do that, really want? Well, they want to separate application delivery from hardware procurement. That's moving from a CAPEX model, where you have to buy hardware, wait a while for it, rack it, the ops guys go, set it up. Then you can deploy your application and debug your application. We really want to separate these two things. That's really powerful and enabling all sorts of fun stuff. We want an API for resources, whether they're compute resources, networking, storage. I'm going to talk about the compute side today. But we want that API regardless of what type of resource it is. We want to be able to describe through that API, I need a GPU today to do some rendering. I need some fast disks to run a database. I need a whole bunch of VMs that I'm going to use just to rapidly scale out this application tier because it's Black Friday and I'm a web, a sales property. Whatever it is, we want a single API for that. Really, we just want control and flexibility of our infrastructure. And then from that, we can build on top of that. So what is this ironic thing? It's a set of Python services, libraries, APIs, that abstract hardware management. That means it abstracts the tasks of provisioning hardware, managing the firmware. We're getting there, turning it on and turning it off. By abstract, I mean you can plug in many different vendors. And I'll get to that in a bit. It's a consistent API across all the vendors. So we have strong contracts in the code for other vendors or if you want to write your own hardware driver, you don't have to upstream it to have a guarantee from us that it's going to be not broken when we upgrade the service. There is, it's well-defined hardware API abstraction inside the code base. And then there's the API on top of that. It integrates with OpenStack, so you can run it as a service and use Nova for scheduling and all of that, and I'll talk much more about that. Or you can run Ironic all by itself, sort of like you can run Swift over here in SwiftStack, you can run Ironic all by itself. And get many of the same functionality, provisioning hardware through an API, but there is really a lot of benefit in running it in OpenStack if you need multi-tenancy or quotas and resource management, all of that. What's it break down to? There's an API, API service, there's a conductor service. That scales out horizontally to manage your hardware if you have lots and lots of hardware. There's an Ironic Python agent. This is a temporary service, runs in a RAM disk. You might wanna build this yourself to add in some custom bits for your hardware for that RAM disk, like managing the rate array. Your rates different than someone else's rate array, different vendor stuff, so build that yourself. And that's booted temporarily, does work on the machine, and then is taken down. There is no persistent agent inside the tenant instance, the tenant operating system. It's a Nova driver, right? If you wanna use Nova with Ironic, you use a different driver, not the liver driver, the Ironic driver. There's also DiscoverD, it's a new project, for discovering hardware. Maybe you just inherited a bunch of hardware, you kind of figure out the management credentials, but you don't know what's in the box. You wonder when DMID code, build an inventory, compare it to what you think it is, that'll help with that. And Bifrost is a set of Ansible modules for getting Ironic working all by itself outside of OpenStack. Just sort of as a utility to go poke at your hardware real quick, or lay down an operating system, and then take your hands off. That's roughly what it looks like as an architecture. The API run in Python, run it under Apache, set of conductors, scale that out. There's a database, a message queue, and plugable drivers. It's based on Open Technologies. So all of our reference implementations are done with Open Technologies. And just to check, does everyone have some sense of what these things are? Because I don't really want to go into them too much. Essentially, it's open standards for managing hardware remotely. IPMI for power, DHCP for passing information to a machine about its IP address, and even redirecting its boot process through Pixi or Ipixi. TFTP to copy the network boot program if it needs it. I SCSI to attach the hard drive remotely and write an image to it. You might ask, what about vendor enhancements? OpenStacks ecosystem is based on vendors and specialization. Well, yes. So we have a whole bunch of vendors that have already contributed drivers. There's a minimum set of functionality that every hardware vendor has to meet. And then through the rest API, you can discover what additional functionality that vendor has enabled, whether it's things like a serial console remotely or UEFI secure boot mode, things like that. Or potentially entirely vendor-specific extensions that are not really part of our API that the vendors might add to their driver and expose. You can discover all those. And we're working towards, as each vendor adds a little bit of this and a little bit of that, trying to bring them all up to a common standard. So you have options. If you don't want to use IPMI, you might use ILO or DRAC or iBoot or SNMP for rack-mounted power distribution unit or AMT if you're playing with nooks, like I have up on stage here. They're fun little developer boxes, by the way. If you don't want to use DHCP because of network restrictions, you can inject static IPs into the machines. If you don't want to use Pixi, again, maybe because of network restrictions, some of the hardware drivers, you can use the virtual media channel to attach a disk remotely and boot from that. And potentially pass in cryptographic keys over the out-of-band channel, so it's more secure. Things like that are being adopted by more vendors. And if you don't want to use iSCSI, I'll give more into the architecture there. There's an architectural choice, like performance versus flexibility. The agent driver also supports fetching the images directly from any HTTP source. You can even be Amazon, no matter. You have more options, too. If your hardware is all the same, it's really simple. If it's not, maybe you have Sundell mixed with some HP, mixed with some Cisco, and you have some servers with lots of disks and some servers with almost no disks but lots of RAM, you can use a Nova scheduler. You describe properties in Ironic and define flavors in Nova with those properties or additional metadata, and the Nova scheduler can match them together for your user's requests. So you can describe the kind of experience you want your application to get when it is deployed on a physical machine. If you're a single tenant, maybe you're an internal service provider for a private cloud, maybe you just have a flat network and all your, you know, there's no isolation, and that's probably fine. If you're a service provider for multiple tenants, either internal or public, you probably only use Keystone for off, use quota management from Nova, use Neutron for network isolation, use Glance for storing different images, use all of OpenStack, integrates with Ironic. However, if your tenants are not trustworthy, that is you don't trust them and they don't trust each other, there's limitations there. There's some big caveats worth pointing out here. The network isolation via Neutron is not all that well upstreamed yet. There's some stuff up from Rackspace, there's some stuff that HP is working on, there's a couple other vendors that are putting up proof of concepts and making proposals. I expect that to be well solved in this cycle. All the frameworks there, there's a lot of support for it. So right now if you really wanted to do that, you could pull down some stuff and put it together yourself. We did just add in the killer release secure erase disks. So between each deployment, there's a delete process that also goes in secure erases the hard drives. The ability to also wipe all the firmware to protect you from the firmware exploits or protect the next tenant from a poisoned SATA controller or poisoned BIOS or whatever. Some of that support is there, not so much upstream, it's sort of a roll your own right now. But that ironic Python agent, RAND disk I mentioned earlier, it supports pluggable hardware managers for those tasks. So it's all part of the cleaning phase, which I'll get to in a bit, and you can add that yourself if you want to. That's what I mean by some assembly required. New and Kilo, ah, we had some issues in the last couple of releases that some drivers did or did not support booting from local disk. That's solved. You can do that as a description now. Do you want this instance to boot from local disk or not? If you don't want to use a cloud-based metadata service, you can use a config drive and write to the metadata that that instance needs to get on your network and get some SSH keys directly on the disk at the end of the disk and it finds it on first boot. I mentioned secure erase. We added version headers. So Nova did some of this, we did some of this in the Kilo cycle that lets clients or other services talking to Ironic discover new features or recognize and upgrade or request an old version of the API. So you can choose when you upgrade your clients. We'll guarantee the API won't change for some significant period of time. As a convenient thing for operators, you can add a logical name, to reference a note instead of just a UID. And if you're developing your own drivers, you can do more cool things now with your own drivers. I'm gonna do it on time. I'm going pretty quick, okay. So by the way, I also love it when people interject and ask questions. We love an interactive audience. So feel free to raise a hand if something jumps out at you. Configuration choices you're gonna need to make. There's a whole lot of information up on our documentation page. It looks like developer documentation. It's not in the opensack manuals, but if you scroll down to the bottom, there's a link to operator documentation, service installation and configuration docs. I'm not gonna go through all of it, but short version is, you do have to reconfigure opensack a bit. You have to configure NOVA to tell NOVA, use the computer driver for ironic instead of KVM. Stop firewalling, because nothing's local on the compute host. Change the scheduler manager, though there's some folks who've been mucking with that and changing that too. Ram allocation ratio and things like that don't really mean much when there's no local hypervisor. Someone gets a whole machine, you don't get to subdivide it. They get a machine or they don't get a machine. And then lastly to point out, another limitation we currently have is around this compute manager. Running more than one NOVA compute process to manage to talk with an ironic cluster runs into some issues right now. So something else we're gonna work on solving in Liberty is the HA story for NOVA compute, which acts as a proxy to ironic. A little more configuration you have to do, you add a section of the NOVA config file for ironic, basically tell it where to talk to ironic. That's about it. You might wanna do some stuff in neutrons config, in ironics config. I'm not gonna go into all those here. It's pretty generic, mostly, just link the services together. I did mention that there are machine images to build. So whether it's the deploy image with ironic Python agent, maybe you wanna customize the actual workload image, pre-bake Hadoop into it, or pre-bake whatever application into it you want, or customize the drivers available in that instance image. We mostly use a tool called Disk Image Builder. There's another one that's started in the kilo cycle called CoreOS Image Builder. It's pretty much the same sort of thing. It's a framework for building a disk image. One just uses a bunch of bash and purl. Sorry, Python, and the other uses CoreOS. The result is a QCOW or a TARGZ-formatted image that can be used with ironic. So you make the image, you load it to glance, or stick it in the patchy somewhere. Somewhere you can reference it. That's about it. As I said, you can give hints to the NOVA scheduler. So this is roughly what that might look like. First, you create a flavor in NOVA, and you describe the basic properties, RAM, CPU, and disk. These are NOVA's primitives for virtual machines. Same thing applies to physical machines. And then you have to tell, if you want to do additional properties, or additional hints, you tell ironic like that. You add a capability, and then you add a flavor key on the NOVA flavor to match. We're really getting to what it looks like to operate this. So there's a separation in the API. The ironic API should never be exposed to end users, untrusted users. You want users to talk to NOVA to request a compute instance. Even if you're only providing bare metal, even if it's only internally, you still want them to talk to NOVA. The ironic API, ironic doesn't have a concept of users and tenants in quotas. All of that is offloaded to NOVA. So if someone can talk to ironic, they can do everything. Really, that is meant for operators to enroll hardware or to manage hardware. Once the hardware is enrolled in ironic, the conductor through the hardware driver will continuously talk to the server to make sure it's there and it's healthy and expose that to NOVA to make it available before scheduling a workload. But the user just says, you know, NOVA boot, some flavor, some image gets passed down to ironic. And ironic does most of the work from there. Limitations, firmware and RAID. As I mentioned, we've got a plugin framework in ironic Python agent, but you really have to bring your own plugins to it. And that may or may not be easy depending on whether or not your plugins run in a CoreOS or say, Ubuntu or Fedora-based image. If your RAID management tool only runs in Windows NT-based RAM disk, I'm not sure how to integrate that yet. We're gonna work on that. Also a lot I'm working with hardware vendors to make that better, yes. So the question was, if I had a group of DBAs, I wanted them to be able to provision bare metal but not do anything else. I would say, give them access to NOVA, right? Set up a tenant in NOVA that has access to a certain flavor or a certain image, certain pool of hardware, but don't give them access to ironic. And you can set those policies through Keystone. NICs and networks. So again, in the mapping between NOVA and ironic and neutron, there are some inherent limitations from the assumptions around virtual machines, one being that a single virtual NIC maps to a single network. That's not always true on physical NICs. You might want multiple VLANs on a NIC. This is something we need to solve and we're working with the neutron team in this cycle hopefully to resolve that. There are some workarounds, but yeah, it's a bit unfortunate right now. Provisioning network and tenant network separation. So that is, during the provisioning process, images are transferred to the machine. After the provisioning is done, you want to move it into the tenant network. Even if that's just one network for all tenants or one network per tenant, right now the support for that swap isn't upstream. There's some stuff Rackspace has up on the Racklabs GitHub. Again, this is the area we're focusing during Kilo to improve that, I'm sorry, liberty to improve the network layer. It's, so the question was, could I go into a little more detail about the plans to fix that? There's multiple proof of concepts up and the work, the existing ones that folks have done are, and jump in those who know neutron better than I do. Neutron mechanism driver and people have done a couple different ones and so the work right now is to instead use the Neutron ML2 extension framework to talk to top Rack switches. And what I'm told is that's basically already done in Neutron. It's a matter of sorting out the APIs between Nova, Ironic, and Neutron. Which one calls which to initiate the change? And that's most of the work and then adding a bit of glue and Ironic to tell whichever servers, whether it's Neutron or Nova, switch these things around at the right times. And it's mostly just glue code. Yeah, we have two design sessions. There's one in the Neutron track on Wednesday and one in the Ironic track on Thursday. Yeah, and there's a couple of specs up in Ironic. There's proof of concept code up in Garrett. So this is really not far away. At this point, I can either walk through some code samples I have to show what it looks like or I can attempt a live demo and I want your input on which one you want. So who wants walk through sample code? Okay, live demo? Okay, okay. So I mentioned the Bifrost project at the beginning. Is that at all readable to you guys? Cool, okay, so I mentioned the Bifrost project. I put something together that's letting me run Bifrost in a VM on my laptop just through Vagrant. So it's as simple as Vagrant up. I'm not gonna try and do it all now because I have no internet access up here. And it wants to do things like app to get update. But just trust me that I have run a few simple commands which were Vagrant up that sets up all the Bifrost stuff in the VM. Oh, and that's it, yeah. It is up on my GitHub because I haven't made it clean yet. It's going to be proposed to the Bifrost project which is now part of OpenStack. As soon as we sort out the CI problem that that's having. But I promise that all of this code that I'm using right here is already up on GitHub and will be proposed to OpenStack as soon as we can. Okay, so what I've got is a little nook. Those of you who are familiar with Intel's little boxes. This is the model that has AMT. And I've got some quick commands here. You can see in the window down here that this window is, oops, not what I'm supposed to do. It's just watching, ironic node list and ironic port list. I'm pointing to the ironic service running in my Vagrant VM which is in this window, okay? There's two ironic services, API and conductor. There's no Nova. There's no Keystone, right? I'm not running any other parts of OpenStack. This is just ironic. There's nothing in the inventory database. I've got, let's see. Okay, so there is a CSV file. Don't ask me why it's CSV, it's historical reasons. That has the MAC address, the credentials and an IP address and some properties of this machine. I'm going to enroll that with Ansible. This is gonna make some calls to ironic and we should see it pop up in the lower right in just a minute, any day now. The network between these is iffy. Yep, I just registered the node and now something is, oh, ironic is probably trying to talk to it. Yeah, this is fun. There we go, okay. So we've got the machine is there, provision state's available, power state's unknown yet, not in maintenance mode. I could just turn it on and off, but that's not interesting. Let's see if I can do a deploy and I'm gonna switch the display as soon as that finishes and move the cable over here. Or, well, we should see the power state change when the deploy is done or when the deploy kicks in. There we go, deploy to hardware step. So what that was all doing is, if this had been the first run, downloading some things, building the RAM disk it's gonna use, building the config drive. Hey, the light turned on, it's blue. I'm gonna switch the cable now. How do we make the video switch? Video? Okay. So that would download stuff from the web, modify the image a bit to make it work on this machine, build a config drive because I'm not using a metadata service, and package it all up and tell everyone what to do with it. Still waiting on the video. I'm plugged into the new guy, yeah. It hasn't switched. Yeah, that's the HDMI cable, I moved it. I mean, I can do that. Yeah, okay, well, the video is not switching. So I'm gonna move it back. There we go. So we can see now, it broke the watch. It's ironically showing that it's in power state on and provision state is wait callback, meaning the RAM disk has booted and ironic is waiting for that RAM disk to phone home and the process to continue. And that'll take, it should take a minute or two and I'm having network issues because, you know, conference, there we go. That at least shows power on wait callback. Let's let that sit for a minute. I'll go back to the slides and check on it in a bit. Let's see. So just to show some of the details, if you were enrolling your own hardware, this is an example of the command line call you'd make. I had that all scripted through byfrost, but maybe you wanna write your own script to do that. And create a port for each machine and validate it so you can validate that the drivers have all the info they need. Byfrost, again, already did all that for us. In this case, I forgot some stuff, so I added validate again, things work, show details, maintenance mode. I should talk about that. So if a machine is misbehaving like this guy is, I can set it to maintenance mode, which tells ironic to take its hands off while you, the operator, go and fix it, like plugging in a new network card or something. And then when that's done, let's turn maintenance mode off and I run up like over again. There is a, I'm gonna skip this so this is boring stuff. Yeah, this is all boring stuff. Oh, that's the end. Oh, no it's not, here's a slide. Okay, so there, I mentioned that, hey it turned on. There are two modes for deploying to different boot drivers. Do we get graphics this time? Do we get video? Yes, no, yes, no. I'm back in the box. Yeah, HDMI is not swapping. Okay, well, so there's two, two deploy modes, the pixie drivers and the agent drivers and I'd recommend folks use the agent drivers and we're gonna work on migrating various vendors code over from the pixie stuff to the agent stuff because that enables all the cool stuff that I talked about, you know, plucking in through the hardware managers, et cetera. I mean, if the box is on, so I don't know what's going on, the video doesn't switch. Yeah, okay. Well, then I'll just reboot it. Turn it off and on again, right? Maybe that'll make the video work. Okay, so the next slide that you're not seeing is actually a flowchart of that. Ooh, look at that. So yeah, the video only works if the video cable's plugged in and the nook turns on. Good to know. Hey, I have a log in. I bet I can SSH into that box, too. Not that you can see that from here. I need two monitors. Ah, here it explains. Right, so at this point, Ironic is showing that the deploy is active and the power state is on. And you know, I could do things like turn it off or I can SSH to it. The address was really now. Well, that was interesting. I don't know why that's happening. Good enough. Okay, deploy process. I'm gonna skip that one because this one's much more interesting. The agent deploy process, right? Nova says set instance info to prep the instance what image it's gonna use. Set provision state active, like Bifrost did. Goes to the deploy, builds the pixie or TFTP config or the ILO config for virtual media or whatever the driver does. Updates DHCP boot parameters with Neutron if it's using that or doesn't if it's the driver saying, you know, use ILO instead. Blah, blah, agent or driver-specific stuff if using DHCP, the machine boots. The agent starts up and does a lookup request and says, here are the MAC addresses I found in this machine, who am I? Am I a thing? Yes, okay, keep going. Pass the UID and it's a heartbeat. That heartbeat tells Ironic things are alive on that machine while the deploy continues and the agent downloads the image when it's done. Ironic clears the DHCP boot flag or doesn't if the driver's different and reboots the machine from the hard drive and then also depending on configuration it might tell the machine through the out-of-band channel boot from your local disk or boot from network or set it to a secure boot mode. It all depends on the drivers with the drivers doing at that stage. Reboots into the user instance. The user should be able to SSH in even though I couldn't and things should work. Any questions? Now the question is what kind of images are deployable? A machine image, right? So it has to have the right drivers for that hardware. It could either be a partition image, like a QCOW format partition image or a whole disk image. If it's a whole disk image, it has to have a boot block and a partition table and all that. So the question was how many releases are we away from having virtual machines and physical machines through OpenStack with the same API and have the networking all working? I would say there are folks who are doing that today and I'd pointed the back of the room. And I would say there are other folks doing that downstream in their own companies today using flat networks. And then I would say if you really want the whole fully featured all the fancy stuff, hopefully Liberty, like I really think we can do it in Liberty. If you want also like a cinder volume attach, there's a lot of interest. We're having more discussions there. I don't know that that'll be done in Liberty but I would like it to be. So the question is can I talk more about the network stuff and I'm gonna say no because I'm not actually a network expert. I'll punt that to Neutron guys in some other session. Other questions? Is there a callback after the images copied to make it more secure? Like what? The images have been written. So if you mean like are we key signing the image and then testing, validating the kernel signature on boot that is possible through UEFI and secure boot. There's interest in doing this through TPMs. Not every vendor supports that today. So maybe it depends. There's a question over there. You were first. I actually could you speak up? The one-to-one mapping limitation I mentioned was actually Nova and Neutron's model of a virtual port to a physical port is one-to-one which limitations took out of that. You had a question. It doesn't or sorry, the agent does verify that. The agent does if you set that property in glance or in the driver, you just tell it the Shah sum. Like as part of the node's instance info you say here's the reference to an image. Here's its Shah. Make sure it has that Shah every time you download it. Yep. I'm gonna jump to this side because there's questions all over. Ah, for which use case is what I recommend to bear metal over virtualization? For example, video rendering. Things that do not pass through KVM very well. Specialized GPUs or specialized hardware appliances or very high IO bound workloads whether it's MySQL or Oracle or SAP or Hadoop or something like that. That running that in a VM is going to significantly slow down the application. And you just wanna use all the resources in that physical machine for one application. You don't need to pack a lot onto one machine. Maybe just don't use virtualization. Yes. I missed the first part of the question. A disk wipe. Yes. Can I punt that to Jay in the back? The question was with regards to erasing the disks during the deletion and cleaning phase is there some sort of an audit trail for regulatory purposes that proved that this happened? Take a hammer to it. Quick call-outs, I think we're getting close on time. There was a last-minute change in the schedule for the ops summit. So those of you who are operators and maybe have used Ironic or begun playing with it a bit please come give us feedback, Wednesday 9.50. There's a question over there? Yeah. If your hardware inventory doesn't align well to flavors let me ask why. Is it because there's lots of little differences and you wanna abstract all those and have maybe just a few flavors or what? Oh. So if there's a sort of a hard hardware selection with hundreds of options but you don't want to expose hundreds of flavors how would you expect users to choose what they're getting? But I mean how do they indicate through an API what they want if not through the Nova Flavor API? You could do that. You could run Ironic alone. Ironic's API exposes all those properties and in the same way that Nova picks them up you could write some scheduler outside of Nova that picks that up and does that. Yes. Quick time check in the back. How are we over or are we just at the right spot? Good. Okay. Question here. Funny thing about maybe using this to install other components of OpenStack. Yes. I could go into that for a long time because that is originally how this project one of the motivations in birthing this project was let's deploy OpenStack. And in that space I also saw from previous employers lots of companies build their own homegrown Pixie based deployment solutions and I wanted to just build one in open source within the OpenStack ecosystem to abstract away all the differences that every Pixie system has. But again the original motivation for that was let's deploy OpenStack on some hardware. So yeah. Any more questions? Yes. The question was can you have Nova manage both bare metal and virtual? The answer is yes. I covered that at the beginning of the talk. Many more questions. Over there there's actually we're at time. So let's go ahead and call it.