 I figured we'd start a little bit early, so here you go. Hello, my name's Scott Jensen. I'm the Director of Engineering with Dell, and what I'm going to be talking about today is Crowbar for OpenStack Deployments. So a couple of quick questions. How many people in here have used Crowbar before? Okay, good. So I have a number of things to go over, in particular to Crowbar, so we'll go through the agenda. So we'll go through a basic overview so that you guys can understand what Crowbar is and how it might be able to help you out with your OpenStack Deployments. So we'll talk about a couple of concepts inside of Crowbar, particularly bar clamps, which are the modules that are inside of the product that you can modify or add or remove, depending on what it is you want to do. The concept of late binding, which is one of the key concepts of how it works. We'll go through the workflow on how Crowbar actually goes through and instantiates the systems and brings up the product. Networking, which is one of the interesting things inside of Crowbar, and it's actually fairly complicated, so I was going to spend a little bit of time on that so that you guys have a basic understanding. Some of the supporting services that Crowbar provides as part of the infrastructure. Community, and finally, Crowbar 2.0, which is a refactor that we're in the process of working on. So we'll talk about some of the features that are going to be as part of that and call it for help. Anybody wants to help out with the project? So without any further ado. So what is Crowbar? Basically, Crowbar is a modular open source framework that helps you bring up your cloud infrastructure. It does everything from the networking infrastructure to the operating system lay down, to the BIOS, to the RAID configuration, to laying down the workload framework, in this case, OpenStack, so it can deploy Nova, Glance, Keystone, all of those various components and puts them all together in a single component. It's based off of a DevOps model, which is basically you codify all of your operations platforms. By doing it in code, you can have a repeatable deployment that you can keep deploying, tearing down, bringing it up, tearing down. So it's really good for a test environment, but also if you're going into a larger scale or a production environment, having the BIOS and the RAID sets all at a standard set just as a basics as well as the operating system really helps with the deployment. It has these things called bar clamps, and I'll get into those a little bit more. So it's effectively a modular product where each one of the components has its own modular. So Nova itself is a bar clamp. Keystone would be a bar clamp. The BIOS is its own bar clamp. So each one can be added, modified, and changed based off of how you want to do your deployment. So there's some information about it. It is an open source project. The only things which are not open sourced are the RAID and BIOS configurations, mostly because they're very specific to the hardware platforms, even the specific chipsets of those hardware platforms. So we have an open source that code. There is a list community, and we'll go into some of the other online resources. Rob Hirschfeld is one of the guys who works with me, has a blog, talks about this stuff all the time. So I encourage you to take a look at that. So what were we trying to do? When we built Crowbar, we decided that what we needed was an installer that could deploy on the hardware from bare metal, basically unracking the systems to a fully deployed cloud in less than two hours. So that's the goal, and we achieved that. Basically, we started it off as an open stack installer, but we've turned it into a little bit more. It's more of a generic installer. So we actually use it. My team actually has two different offerings. So we build an open stack private cloud, but we also have a data analytics Hadoop cloud, and we use Crowbar to deploy that as well. So we do that with a distribution. So you can use it to deploy anything it is that you want in those components. Basically, what we wanted to do is get customers up running very quickly, have a way to cope with the frequent updates, basically updating the bar clamps online, and be able to connect all the different components together, so it has an understanding of what's going to occur in the environment, what the customer wants to use these systems for, and then provisions them appropriately. For example, if you have a Nova compute node, there's certain settings inside of the BIOS that you want to turn on, obviously virtualization, otherwise you're not going to get very far with a KVM instantiation. On the Swift side, you actually don't want that on because it actually has a performance impact. For example, the RAID sets are particular to what it is that you're going to do, and I'll talk about that a little bit more in the late binding concept. But those are the pieces that we're trying to get through that. And we get a lot of uplift from the community. They do a lot of bug fixes and some feature development for us, which is just awesome. So it's great to have a project that's like that. So when you're deploying OpenStack, when you take a look at it, there's a whole bunch of interlocking parts. You have the RabbitMQ, compute nodes, controller, API, storage nodes, and it really matters in what order these things are deployed, not just that they're deployed. So it matters how they are, but they reuse a lot of different components. I believe that's the end of the build, but it's not. So these different components can be reused on the compute nodes, so we keep those all contained inside of the same bar clamp. Underneath the covers, what we actually do is we use a CMDB to actually drive the deployment of the system. In this case, we're using Chef, but I'll talk about some other stuff that we're doing as part of Crowbar 2.0 to make the system agnostic to the CMDB that's under there. Right now, we're pretty tied to it. So each of those things actually turns into a series of recipes that get deployed to the system, but it's orchestrated through Crowbar, which understands what it is the user is trying to deploy in the environment. So in the DevOps model, what we do is we build up the system in a series of layers, as opposed to building like a gold image and doing a ghosting component on there where you have like a gold operating system with a lot of the components that you want to deploy and you image that on the system. The way that we're doing it is slightly different in the fact that we're layering the different components on there. So we'll do a hardware layer, so the RAID and BIOS stuff that I talked about before. We'll then deploy the appropriate operating system depending on what it is that you want to do. In the current 1.0 version of Crowbar, when you build it, it has one bundled into it. So if you go to GitHub and build it or you download the ISO that we provide, everything is all contained in that component. And when you install it, it'll actually have the Kickstart files and all the stuff for the operating systems. For OpenStack, we deploy Ubuntu at the moment. For Hadoop, we'll do a Red Hat deployment. There's nothing that says you have to stick with one or the other. The cookbooks are very specific to the deployments. After that, we'll deploy the infrastructure. So we'll deploy the Nova Swift, Keystone. All of those components will make the appropriate entries in the databases. We will set up the MySQL database in the back end and get all of those things going. That's nice. And then on top of that, what it allows us to do is have a framework to put ecosystem components on top of it. So if you wanted a orchestration software to be deployed with it, something like Instratus, that'll also be part of the system. And you can deploy it all from the same infrastructure over and over and over again. Make sense? Cool. So provision the bare metal servers without user intervention. And I'll get into the workflow and how the state machine and those components work inside of the system. The core operating systems can integrate with other existing products, deliver basic data center services such as DNS, NTP, and other components like that are all part of the crowbar server when you bring it up. So effectively, when you get the ISO from the system, you install it on what we call an admin server. It's a server that runs crowbar. It'll also support a series of supporting services like DNS, NTP, and Pixie Boot server, TFTP boot. All of those components are all part of the system so that the cloud itself can be self-contained. It's nice for developers in the fact that because it's all self-contained, you can do it inside of a VMware workstation. So you can build an entire cloud inside of your VMware workstation, emulate a multiple system deployment, tear it down and bring it up, tear it down and bring it up, fully self-contained. So you don't have to worry about any of those other ports. We do have the ability to attach to an upstream DNS server, stratum 3 o'clock, however you want. That's part of the deployment. Cloud infrastructure is, again, pluggable and it's designed in a way so that the community can expand it and change those components as necessary for your deployment or to support something different. For example, Cinder would become its own bar clamp going forward. Celiometer for metering as well would become its own modular component. They have dependencies between them where the components can be dependent on each other and it's basically set up in a run list and a dependency list inside of the system. So you can set that stuff up and modulate it in any way that makes sense for your deployment. And then, of course, the APIs and the entire cloud infrastructure gets put in at the very top. So the bar clamps, they all fall into these basic areas. On the physical resources, there's a provisioner bar clamp which actually deals with the rollout of the state machine, Pixie Boots, who gets brought up what and what area. The network one, which I'll go into a little bit further. Raiden BIOS, I've talked about those a little bit. We also have an IPMI where if you have an IPMI enabled system, we can set that up, send reboot commands, reset, identify yourself, a couple other functions. Core components. There's a crowbar bar clamp. So when you install crowbar, it actually runs Chef on itself and configures itself so that it follows the same paradigm as the rest of the system. Again, a DevOps code, your deployment. The deployer, DNS, NTP, MySQL, there's a logging bar clamp that'll collect all your syslogs and shove them off wherever you want or allow you to collect them for a support case if somebody wanted to analyze the logs off-site. Then the cloud infrastructure portions. I've talked about those. I have a couple in here for the Hadoop side. They all work the same way, Hadoop Hive, but Nova Swift, Keystone. Chef is one of the ones. We're working with the Ink Tank guys to build a bar clamp which would deploy a distributed block storage system as part of the product. Force 10 is the hardware platform. I don't necessarily have a bar clamp for that, but that's where it would be if I built one at this point in time. And then on top of it, we also deployed Nagios and Ganglia for some basic monitoring capabilities. I'll talk a little bit more about that when I get to the slide on that component. The Dashboard, so the Nova Dashboard horizon, and this is where we put things like Instratus or Cloud Foundry, which are a couple of the other bar clamps, Cloud Foundry ones, the VMware guys wrote for us. Cool. So, late binding. What we mean by late binding is that when you're going to deploy one of these clouds, effectively what we do is we hold off to the last possible second before enacting changes on the system. So the customer goes through the crowbar user interface and they say what they want to deploy and effectively what they want to use the system for. It's in a method we call a proposal mechanism. So in the UI, what you do is you say, I want to deploy a MySQL database. It'll ask the customer a couple of questions. Do you want a singleton, multi? How do you want to deploy the database? I want to deploy Nova and I want, the system will give you a proposal to say, okay, here's a series of systems that you can deploy Nova on. Which ones would you like to deploy it on? And basically we go through all of those components without touching anything in the systems. The systems, when I go through the next slide, which is how the state goes, we'll pick up some basic information about the systems, how many drives, what CPUs are, how much memory they have, basic things that you need in order to make those decisions. The main point in the late binding is at the very end when the customer says, yes, this is the configuration that I want, at that point we will begin setting up the network, we'll just begin setting up the appropriate operating system, we'll set up the right BIOS, the right RAID settings, and all of those components. They all happen at the very end. Minimizes churn in the environment, effectively. As opposed to starting off with a gold image that you splatter around a whole bunch of systems and then modify it to make the changes. So that's what we're talking about with late binding. So how does crowbar work? So basically, as I mentioned before, you end up with an ISO and you install the component on a single server. This brings up a number of services and a number of basic services that can be used to support the infrastructure. And then what you do is you unbox and you configure your network switch, put your systems online, and boot them up. And effectively what happens is they boot themselves up and pixie boot into an image that we call sledgehammer. It's a dissolvable sent OS image that runs just in memory. We then run a series of Chef clients against it in order to get an inventory of the system. And once we have that inventory and we determine that the system is something that we want to bring on, it goes into a waiting state. So in the user interface, it kind of blinks this yellow light. And that basically means the system's sitting there, there's nothing deployed on the hard drives yet, but you now know some information the system can be used as a target. So we run the Chef client on that thing to make sure that it's okay. When you finally decide what it is that you want to deploy on it, we then run another series of Chef clients against it and that's where it'll enact the RAID and BIOSes. This way we have an operating system that's a well-known operating system that we can run a series of tools against in order to enact that stuff regardless of what the final operating system is going to be on the system. And we'll use this sledgehammer image in other times if you wanted to do a hardware update, do a BIOS update, or you wanted to change the RAID array if you're going to do it in a non-destructive manner later on in the environment. So then what we do is we boot the system again, remotely boot it, and then we lay down the operating system using your typical Kickstarter jumpstart platforms. We lay down the appropriate operating system with all of the different components that are in there. We then lay down the network configurations. So depending on whether you're putting this stuff on Swift or whether it's a Nova, it'll have different networking characteristics which are defined in the networking bar clamp. It'll determine whether it has a storage VLAN or not, whether it has a public VLAN or not, admin and VLAN and all those different components. So the network configuration is done in there, which can be tricky in Linux given the enumeration issues that you can run into. So the next series will... The next thing I'll go into is the networking components and how we try to deal with the enumeration problems. And then effectively we do another run of the Chef runs, applying the appropriate cookbooks and roles in order to deploy the application the way that the customer has determined that they want the app to be deployed. Okay, does all of that make sense, guys, or does anybody have any questions about the basic running of it? It seems simple. Yeah, sure. Why do we... So the question was, why do we need Crowbar? So what Crowbar does on top of what Chef is doing is it orchestrates the different cookbooks. So we actually looked at using Chef. We would have much rather have just used an off-the-shelf tool rather than writing a new one. But what we found is that it was really good at making a system, a specific thing. But if you have multiple independencies like you do in a cloud environment, the orchestration for that just wasn't there. So we needed to write an orchestration layer or what we call a meta-layer, which is what Crowbar does. So it orchestrates that stuff and then makes a cleaner multi-system deployment by doing it that way. Sure, go ahead. The question was, do we have our own versions of the cookbooks or are we using the standard upstream ones? At the moment in Crowbar 1.0, we have a fork of those cookbooks. Effectively, actually, we worked with the ops code guys and did a lot of the writing of that. The rack space guys did a ton of stuff as well. So we've taken advantage of that. Currently, the cookbooks have a lot of what we call Crowbar-isms in them to make Crowbar work correctly. That's not ideal. And inside of Crowbar 2.0, when I get to that section, one of the things... One of the features that we're putting to the product is CMDB agnostic. So basically, you can use any CMDB. And in order to do that, we need to take those tags that are in the recipes and make them not destructive in the recipe and pull them back up. That way, we can upstream those portions. Yes? What about quantum? So quantum would be one of its own bar clamps. At the moment, it's not in there. So that would be part of the Folsom deploy. Yes? So the question was, are we already using it in production? Yeah, we have a number of customers who are using Crowbar. And we also use it internally for a lot of things. A lot of the customers who use an open stack at this point are in kind of a dev test mode. So what they're doing is they're testing the deployment or they're testing their application in it. But yes, they've used Crowbar in live environments. Okay. I'll move on. So networking. Oh, last one. And then the apps right back into the orchestration machine. So the state machine is actually really important inside of Crowbar. So each of the systems are running through a state. And as each of these phases are occurring, the system moves from a ready, an installing to a ready, to a hardware installing to another ready state. The state machine keeps track of those things and allows us to trip it back into different states. These different bar clamps actually have the ability to talk to each other via the RabbitMQ server. So when the Nova, for example, goes to deploy something, it actually talks to the networking bar clamp and says, okay, I need a series of networks. What do you have? And it'll give them back. When it's done, what it'll do is it will post onto the system that I've done a deployment. And that will kick the Nagios and Ganglia bar clamps to wake up and say, there's a new system that I need to monitor. Here's what was deployed on it. And these are the new monitors that I need to add. So the state machine actually runs a lot of those components. So networking. So the networking in Crowbar is actually fairly advanced. We handle all of the different networking modes, flat DHCP, tended to DHCP that OpenStack supports. We also support bonded NICs. And you can specify which type of bond it is that you want, type five, type six. Depending on what you have for your switch, you may want a more active bond or you may not. The problem with it is in Linux and specifically Ubuntu, depending on where you've put the network cards in the system, the systems will enumerate the NICs in a different order, whether it's bus one, bus two. So what we've done is we've built these series of what we call conduits. And the conduits are a abstraction layer from the hardware, the physical hardware deployment to what it is that you're actually writing your bar clamp code against. This allows you to have standardized code to say, you know, use the network where you don't need to know that it's bonded, not bonded on this network, that network, on those components. The conduits themselves can also represent a bonded stream. So when you set up the system to paste off of the conduits, they tie then into the physical components of the system. So there's a file in there called the network JSON file that describes all of this stuff. Again, we have a crowbar 2.0 feature that I'll talk about a little bit later about how we're trying to simplify this a little bit, but it's fairly complicated to get set up. The basic one that's in there is usually sufficient for what it is that you would want to do, but usually there's some modifications to paste off of the deployment that you'd want to do there in order to handle the appropriate networking modes, types of switches that you have, how many networks you want, et cetera. Sure, go ahead. Okay, the question was, is there an ability to differentiate between different link speeds? Yes, there is. So what you can basically do is you can say, you know, 1G or 10G, and then you can preface like a certain network to a 10G if it exists in the system, if not fall down to a 1G. And then effectively what we'll do is provision the NIC cards in that order, a little timing thing going on here. So I thought I'd show a couple of screenshots of it. This is the proposal view. So basically in proposals, what you do as the user once the systems are brought up in there in the sledgehammer, you select one of the bar clamp that you actually want to deploy, and the system gives you a proposal of which systems you should deploy that with. You can typically just take it at face value or you can modify the settings underneath it. And when you apply that, that'll actually kick off the operating system install and all those other components via that late binding thing I was talking about. This is a really basic proposal. So there'll be a series of attributes on the top so you can set passwords, networking modes, whatever you have depending on the one that you're doing. Over on the right, there's a raw mode that'll actually show you the basic JSON that we're doing. We expose effectively three different interfaces. This thing's going to keep advancing on me. We expose the user interface. Obviously, there's a fully-functioned RESTful API so you can integrate with it that way and there's also a CLI and all of them perform the same functions. They actually all write the same APIs so you can integrate it with other systems in your workload if that's what you want to do in your workflow. And then Nagios and Ganglia. So I had mentioned that Nagios and Ganglia are actually modified at deploy time to monitor the appropriate things that are being deployed. So if you pick a non-standard port and one of the monitors here is an active port monitor, it will modify itself and the deployment to monitor that appropriate port, which is pretty cool. I talked a little bit about the supporting services, DNS, NTP, logging service, and then the CLI and APIs. The community is really important to us. Some of these statistics are a little out of date, but we have over 500 followers on GitHub at the moment. Tons of hits on the crowbar sites and lots of downloads of the ISO. We also do a series of hack days, which we did before the last one. We had over 70 participants worldwide in the hack day. We did another deploy day before that to work on these things, so that was fun. We have a number of different contributors, different companies contributing, either as a company or as individuals in those companies. We also host a series of meetups, not specifically on crowbars, more for OpenStack. My team is actually broken pretty evenly between the Boston area and Austin, Texas, so we host meetups in those. There's a couple hundred people in each one of the meetups, so those are pretty cool. And then there's a series of webpages to find some information there. So the last topic is crowbar 2.0. So we, like I had mentioned before, we're in the process of a refactor. So I wanted to go over quickly some of the key things that we're working on. So the first one is attribute-driven recipes. This goes to one of the questions that was earlier where effectively what we're trying to do is take all of the crowbarisms out of the recipes that are currently in Chef and make it so that we can then upstream those packages and have them work whether you're using crowbar or not as a best practice deploy. So we're in our process of that, and what that will eventually allow us to do in the future is use any CMDB that the customer is interested in, effectively by adding certain tags into those components. Heterogeneous operating systems. This is currently what the system does is if you download the OpenStack one, it installs Ubuntu, otherwise Red Hat. There's nothing to say that the system can't deploy a mixed environment. So some Ubuntu, some Red Hat, some SUSE, some whatever your favorite flavor is, Windows even at some point. So that's the feature in there that we're adding is the ability to do that. What that allows us to do is to build clouds that have a mix. Some hardware components, if you needed a database server in particular or if you wanted Hadoop working in conjunction with your cloud for data analytics, that's what that feature will do. The networking model, so I had mentioned that there's enough stuff in the JSON. Effectively what we're going to do is strengthen the abstractions between that to allow the system to understand that there may be virtual systems. In other words, it's part of a HA deployment where you'd have two systems and then a virtual system on top of that. And also be able to change the networking model after deploy time. At the moment, what happens is you deploy the system and those are your networking models. There's some things that you can change but some of them are already been enacted. This will allow you to change the networking components. We find that in a lot of customer deploy, so they tell us, here's the networking I want and we go and install and they go, I've changed my mind, it's not that anymore. I want something else. So that allows that portion. Pull from source. So Andy, who's here, is doing a talk later on that one. At the moment, what we do is we're using canonicals packaging in order to do the deployments part of the recipes. There's nothing that says that we have to do that. So we've also are in the process of building a pull from source. It'll pull directly from trunk. On line mode, one of the base tenants that we had with Crowbar originally was that it should be able to work totally isolated. When you start adding things like heterogeneous operating system, then the ISO gets a little big to try to deploy it that way. So this would allow you to pull different components from the net as part of your deployment. Basically, all the different bar clamps would be able to be pulled as well as the operating systems. So those are those two. Other things that we're doing is we're improving our database and state machines to handle much larger deployments and allow migrations and upgrades to coordinate the migration and upgrade problem that everybody has at the moment. We've added in a documentation feature, basically a markdown documentation feature. This mostly will support people who are writing third-party bar clamps. That way, they can incorporate their documentation into the product and have that all be part of it. We've kicked up to Rails 3 from Rails 2, something or other. That allows us to simplify the code base and take advantage of the newer jams. We've spent a lot of effort working on the automated test and unit test systems. We've recently incorporated Tempest as part of it. That handles the API tests for OpenStack. We're also implementing a series of unit tests in the system. And that is effectively what we're doing at Crowbar 2.0. There's a number of other features. The stuff's all on the web. If you want to read the design docs, you want to help code stuff and do other portions. So they're all out there. We have a booth over on the other side if people want some additional information around Crowbar, want to see a product demo, or want to get these things because they're probably tough to write down. You can take a look at those. There's some YouTube feeds on what it is that we're trying to do and then the design docs for using EtherPad to coordinate the stuff we're having by weekly community meetings to work with the developers who are helping us out with this code. That's the bunny. So questions. 2.0, the schedule isn't hard yet. We're hoping to get it finished up by the end of the year. Crowbar 2.0 will be based off of Folsom. 1.0 stuff is currently Essex, but we have a Folsom community edition available now. Quantum? I don't know if Quantum's in there. I don't think it is. Not yet. It's something that we intend to work on but we're waiting for it to stabilize a little bit. Sure. Yes, sir. Okay. So the question was, what are the scalability issues? So in 1.0, we're abusing Chef quite badly. So we're using it as our data store, effectively. So we use data bags. If you're familiar with Chef, there's a data bag where you can just stuff stuff in there. That's cool. It works great. So we didn't need to deploy something. However, it causes a series of seek functions. So when you get a large deployment somewhere in the order of 60 systems or so, the seek times start becoming really slow and the UI becomes semi-unresponsive. So what we're doing is we're taking that off and using our own database and that way we can mess with it in our own object model. Does that make sense? Yeah, it's around 60. And then you start hitting UI problems. Sure, sure, sure, sure. So the question was, right now we've talked a lot about Chef. What about Puppet? So we've been working with the Puppet guys. The Qobar 2.0 CMDB agnostic feature would allow us to use either Puppet, Chef, or whatever, right? It would be a series of calls that would be made and would have to be written for a different CMDB as an abstraction layer. So we intend on doing a Puppet and Chef capability mostly because our customers sometimes have a preference for one or the other. Yes? Okay, two questions. So first, does it require any hardware, any specific hardware? Not really. It just needs enough memory, right? So you need a beefy enough system to handle it. The open source versions, again, where you start tying in with the hardware with the BIOS and RAID bar clamps, then you need specific hardware, but the one that's available on GitHub really doesn't matter. You can do it also in virtual as well. The second one, does it support IPv6? Not any differently than what OpenStack would do now, but we don't do an IPv6 deployment. Okay, so again, the same thing kind of... So the question is, for the clients that you're deploying, are there any specific hardware requirements? Again, the RAID and BIOS bar clamps are very specific to certain versions of BIOS and certain versions of RAID, namely LSI controllers. Other than that, the IPMI, I don't know if there's a specific version. Does that make sense? So effectively, it's mostly agnostic. We have people doing it on a number of different platforms. All right, I've run over by a couple minutes, so I'll take one more question. Two questions, so the first one was, do we have a commercial version? Yes, we do. It includes the RAID and BIOS stuff. We don't actually sell crowbar. It's still free in that mode, but it's part of the Dell cloud offering. So it's contained in it, and it's obviously high-level support for it, and it contains those other additional bar clamps. The second one is, do we have an HA deployment? Natively, we do not have an HA deployment, but as part of that offering, we have a services offering where we can deploy an HA system as well, but it's not baked into crowbar currently. I'm over time, so I want to thank you very much. Thanks, guys. I'll be over at the Dell booth if you guys have any questions or want to see a demo or anything like that. Thank you very much.