 So this is a session that's going to start off with a little theory, and we're going to show some cool toys at the end, about enterprise volumes, or really, I guess, kind of the journey that Cinder's going through now. Ironically, this session was scheduled at the same time as the Cinder Design Summit, but not part of the Cinder Design Summit. Most of the decisions are happening in the other room. So if you want to go over there, now's the time. Alright, so here's the recovery. First, we're going to talk about enterprise storage concepts and open stack, primarily. Kind of some of where Cinder's going as it grows up through Grizzly. So little DevOps in you, current design questions around Cinder, and then some toys that Rackspace is doing along with NetApp. Alright, so this is Robert Esker. Hello. I did not authorize you for that photo. It was on your LinkedIn, dude. It's on the internet. It's free for use, right? Alright. PS, all of the pictures in here are ripped off of Google Images. Yeah, this is Cody. Cody's a long time racker. We used to call him the grandfather of virtualization, but now he's the big daddy of cloud. Yeah. And then me. Okay. So when we think of big iron IT, and a lot of the reason that cloud is so compelling for enterprises, storage tends to come up as one of that. Not because it's too overly complicated or it requires extra specialization, because it's basically in use everywhere, right? But what we do say is that traditional IT models are either specialized or they're segregated or layered. You know, it kind of mimics the ISO model, where if you have an application, it doesn't really talk to anything below it. It's kind of agnostic of what it's running on. It might be aware of other members behind the load balancer, or it might know of other functions like caching servers. But it has no idea if it's on, let's say, a super fast tier of storage. It has really no idea if it's on aggregated ethernet links, right? It doesn't. And that's just kind of the way that applications were designed. So when we talk about, this is actually a picture, I don't know if you guys know from hackers, this is the Gibson mainframe. But yeah, so the traditional IT model is very strong, smart infrastructure, dumb apps. As we move from traditional IT into a cloud model, what you see is smarter apps that make the decisions for the infrastructure. Now that's different. There is going to be some people in the Cinder community who would say that there is no place for enterprise storage in cloud. I actually disagree with that. I think that that is a bit naive of a perspective. I think, what is it, $200 billion or whatever across the storage industry can't go wrong. But I do think that there is a place in that it is transforming and that it's going to transform us as IT consumers and as IT specialists. It's going to change developers the way you write applications. In cloud, it's not that things get dumber, it's that decisions are made in different layers, right? When you talk about giving your storage components, the actual physical shelves of disks and the controller arrays. When you start talking about giving those APIs that can be consumed directly by the application living on it, you move the decision making power into the app. This is the magic of cloud. This is where we're all going. So it's not that, if you want to, I guess I could have done it. You've got cats and dogs for DevOps. I forgot where it was. I've got smarter apps. Thanks. So you're talking about smarter apps that have the ability to make sense of the underlying technology. So I've got three pictures here. This is like what we think of. Most of us, this is probably what our developers and our engineers look like. Cats and dogs fighting. This of course over here is DevOps if you guys don't know what that is. What's the fastest way to get results out of two groups that hate each other? Give them the same goals. Put them on the same team. Make them have to work together. Then there's certain freaky individuals in your team that are going to end up like this. This is where DevOps ends up. You guys have any experience with it? You get weird cat-dog hybrids. And these are the guys in your groups. You have people here who are both operationally experienced and are pretty good developers or vice versa, whatever their focus is. These are the guys that you want to take care of. Because they're going to be the ones who end up designing the infrastructure and application together. Who designs your storage platform in cloud? These guys. Who designs the caching players? These guys. It won't be devs or operations, but guys who understand both. Okay, so without further ado, let's talk about storage design questions in Cinder. This here is shamefully ripped off of Google Image Search. This is a SQL server cluster from some Microsoft knowledge base. So we have here centralized storage, which is I guess where this is going. A lot of current clustering design mechanisms such as like Quorum disks or Active Passive, they rely not on multiple copies of data everywhere. Sometimes that's really inappropriate. It's difficult to write, as you guys know. And there's not too much that can be done there when you start talking about the need for atomicity in data and whatnot. Shared storage, however, gives you that. In fact, this design paradigm is currently impossible unless you do terrible things, like try and keep your own data in sync through something like DRVD or some really weird, you know, crazy witchcraft. The reason being is that you can't mount volumes on two instances. Okay, so there's a few things here. These are what we chose to talk about, the three types of, I guess you could call it, storage designs that Cinder doesn't really have an analog port right now. First, shared volumes, which I was just talking about. Second, is storage tiering. Sometimes it's not appropriate to have all of your file archives on SSDs, but you do want to keep maybe some Oracle database on SSD. You don't really get that option in Cinder. Okay, you kind of get one or the other. And then third is shared file systems. These are for more of your network file systems, NFS or SIFs. And actually, if you want to maybe get your voices running out, if you want to start talking. All right. Well, yeah, so it's interesting timing, interesting chronology, because as Paul had mentioned, the design summit is continuing elsewhere at the same time. So some of these, the approaches to having solved some of these problems are being contemplated, as I speak. That said, there are ideas, there are blueprints that approach each one of these. So shared volumes, actually, I think is fairly simple to accommodate, fairly simple to deal with. And it's a fortunate consequence of some of the early design decisions when Nova and the subcomponent Nova volume were originally created. It's not explicitly provided for that across instances you could have access to one volume, but it's not an immense amount of code. But I think that's something that we expect to sail through fairly easily. I can't promise it for grizzly, but I also have a hard time imagining why it wouldn't make it. Storage cheering is somewhat more interesting. Really what we, you know, there are different approaches to it. I'll just very briefly speak about some of NetApp's thought process and design what we did. For Essex originally and then again in Folsom and a couple of different variations on that theme. We ended up taking a very infrequently used parameter, vault type, and we used that to align to things that you can arbitrarily create on the back end, a storage service catalog. You create your own tiers and you can select from them. But to be clear, OpenStack actually doesn't understand what those tiers are. We're just basically doing a string compare, hey, it's the right one, good. So we'll serve it up. The gold-silver Platinum. Yeah, and gold-silver Platinum has no specific meaning to OpenStack, nor does any other similar construct or definition. So that was a way of dealing with what was there. It was expedient and I think there's a tremendous amount of effort right now to take a table that's not currently used. It's provided for a volume type of extra specs to inform what a given back end actually provides where you would have multiple back ends for different tiers and then allow Cinder, the scheduler thereof, to make the right decision. We ended up doing that on the back end independently of that because OpenStack itself doesn't actually provide for it. The promises that OpenStack will and we can get out of that business. Again, that's something that's being discussed right now. And then shared file systems is definitely not provided by Cinder. Indeed, the name Cinder kind of sounds like a block, something, right? Cinder blocks. That said, we've heard from a number of folks saying, okay, well, how do I coordinate cross-instance shared access? You know, shared file systems. There's lots of application types in the world that care about that and it's just actually a certain level of collaboration between inhabitants of an instance. And so we've prototyped some work, which we'll talk about in further detail tomorrow morning at 9.50. We'll show off the prototype. The Blueprint exists. There's a draft functional spec on the OpenStack Wiki that adds shared file system support to Cinder specifically. Now, this being a community, we'll see if indeed the wider Cinder constituents agree to take it. If not, then we'll pursue it slightly differently, perhaps, as a different incubated service. The point is we're bringing shared file system support to OpenStack. So a little bit of what's not there and what's on its way. Cool. Are you going to ask questions on that? Sure. Is that multi-tenant? Yeah. Is that multi-tenant? Well, how are you defining tenant? So rather than answer that specifically with what we've prototyped, I think what probably makes the most sense is to point out that we're providing the basic plumbing, the expansion of the Cinder API itself to request and receive shared file systems across instances. But there are a lot of other considerations, and by the way, this support is sufficiently abstracted that we don't care about what the actual shared file system is. It could be a SIFS or in a variety of NFS or any number of other things. I presume that we can go on. The point is there are other considerations, like permission to masks and what is the authority for username space. These are things that, frankly, aren't going to be coordinated entirely in Cinder. There'll probably be some interaction with Keystone or there may be a presumption that you have an existing user, like an open direct or an LDAP of some sort that exists to accommodate that. So tenancy, that's another phase, I think. You know, doing cross-tenant, that gets way more complicated and there's implications on what that looks like with Quantum as well. So we're starting with the plumbing and I think over time, the full realization, something like a cross-tenant coordination would be realized. I wouldn't expect that to be adversely planned. Can you put that another way? Can you take an existing file system that has a whole bunch of data on it that you want in the cloud now? Yeah, the way we built it would be a short question. Oh, would one expect to be able to take an existing file system, of which there are lots in the world populated with lots of useful stuff, and then event that amongst instances and the way we prototyped it, that should be a possibility. What's the name of that session tomorrow? I think it's called the Unified Stories Infrastructure for OpenStack. I know that's done a lot of work for kind of filling the gaps where Cinder doesn't have maturity yet. A lot of that's behind the scenes. You only get one assertive behavior. So hopefully as these grizzly designs are kicking in, we'll see more of those gaps being filled in officially, more APIs providing specs for functionality. So there's one question that comes out of all of this, is if you're looking to build a private cloud and you want to take a particular vendor, say like NetApp or UBC, how do you know what you're getting is going to work with Cinder, with OpenStack? How do you know if A, they have the functionality you want and B, if it actually is not just marketing speak or if it actually has substance there? Because there's a lot of that going around that you guys haven't noticed. So I want to talk to you about now with something Rackspace is doing. This is kind of the toy demo, which I think you guys will appreciate. Some of you Linux folks in here. So where Rackspace is starting is a vendor qualification lab. What that is is a way of answering that question. It says here are certified vendors and software, so both hardware and software, that we know works according to their design specs and works with OpenStack at large. We are, right now, it's off of the Ubuntu stable packages, but we're looking to extend that right off to trunk so that if you just yank down OpenStack, it's going to work on this model of NetApp Filer or it'll work on that metal of Dell Switch or whatever. So this extends for both storage and into quantum as well as compute. And this is a block diagram. Here, well, I didn't... almost illegible, I apologize. So what we want to show you now is just a cool tool. We actually thought it was so cool that you guys might want to use it too. It's called Razer. It's actually written by EMC. Razer is a bare metal provisioning thing. Cody's going to... Played with Razer so far. View over here. This side, not so much. Alright, so this will be a demo for the stage right. Stage right has fallen asleep. One of the two. Alright, well, we've got pictures of the slide going on. I don't want to interrupt. Come on now. Okay. So what Razer allows us to do is we have we have like 600 physical servers that we have cut up into various roles that we play with. It's almost impossible to manage because we're deploying OpenStack to it with each of those clusters like six times a day. Madness to try and do that. Read what I've got there. I can make it a little bigger. So we've got thumbs up. So what you're looking at here is... Actually, did you have Razer pictures in your desk as to how the process works? No, I didn't. So we'll just kind of add lib to the process. We have Tom here to correct me if I get anything wrong. Tom is one of the two authors of Razer so I will try not to screw it up. I mean, we've only spent the last two weeks together and it's all right. So what happens when a node boots into Razer? So Razer assumes a DHCP server on your network with a configured next server and beyond that it handles the rest, right? So it ships with a TFTP server. You boot up, you pixie. It pulls down a 20 megabyte or so microkernel, 40 megabyte tiny Linux distro that basically boots up and pulls all of your hardware information. That's pretty much what we're looking at there under the tags is a brief summary of the hardware information. So we've got VMs up with a single NIC, single CPU, some amount of memory, and because they're running in Fusion on my laptop here it identifies them as VM or VMs, right? If they were hardware, it would identify the piece of hardware. If they were Zen VMs or KVM VMs it would identify what hypervisors those came from. And that's actually really important as you get into some of the other aspects of Razer. So let me pull out my cheat sheet here. So we did... I was supposed to show off Razer imagery. So images in Razer are basically your ISOs. So what we have here, the bottom one is a debug of the microkernel. It just made things easier for the demonstration. And the Mbuntu 1204 standard ISO, right? We loaded these in last night, kind of thing. Excuse me. I didn't want that coming out over the speakers. It's never pleasant, right? All right. Next is we build a couple of models in Razer. So models are... they're what you apply in policies or so. So you take a policy, assign it a model. That model is also assigned an image. So what we have here is a basic Mbuntu deployment using the Linux Deploy template. We've given it a description and it's got a UUID, right? Fairly simple. But what I can do with this one image is compute nodes, controller nodes, Swift nodes, you name it. Anything that, you know, it's all a very basic Mbuntu install, right? Where that's important is as we get down into the weeds here. So we have a couple of policies. One for compute. The other for an all-in-one. So compute and controller, all in the same... same box. We've also tagged them. So my controller is tagged at one gigabyte of memory. So if you look up top where we have the Razer node list, there's one of those with one gigabyte of memory, right? If I had a hundred of those with one gigabyte of memory, I could all... when I apply that policy, they all reboot. They all become open stack all-in-ones, right? You can also set maxes and minimums on your policies. So the counter there, right? If I had a max, which is just to the left of it there, of, say, 10, I would get 10 all-in-ones, right? So let's go ahead and enable that policy and see what happens, right? So we went enable was true now, and I think we'll check in on this box, as well as it's always hard to type in about 150 people staring at you. So what you'll see here is we have A for active and B for bound. So what happened is my one with one gigabyte of memory that was running the microkernel checked in. Razer said, hey, I've got something for you. It bound that policy to that node and is rebooting. So TFTP booted, pulled up the menu, and is going to start pulling down the iPixie for... or an iPixie and then a boot to installer, and it's just gonna kick off. Before that gets too far, I wanted to... a couple other things we can do over here. So, whoop, active model, model. There we are. So we've got an active model, right? That's the third one to buy into it. What we're gonna do is we're going to simultaneously watch this thing install, as well as what's going on in the active model in the logs. So it init, it pulled up, it got a boot call, it pulled down the pre-seed file, acknowledged the pre-seed, and then it's gonna sit here at a purple screen for me. Right? Generally this takes about minute, minute and a half as that comes up, so... Yeah, I guess... So where this is going, and RAZR is a great tool. What I really like about it, RAZR, compared to other things, there's, like, I think, ten different suites of applications that do this similar bare model stuff. Yeah, the Y of RAZR, is that what we're talking about? So the Y of RAZR is... RAZR is very specific at what it does. It doesn't try to do everything. It doesn't try to be your DevOps framework and your provisioning framework. It is just the provisioning framework. It has a broker that will let you, once the node is installed, hand off to whatever that DevOps framework is. So you have your area of expertise. Like RAZR does provisioning, it does provisioning really, really well. DevOps does everything else really, really well once you have the OSBits laid down. That's the DevOps framework to be like... A Chef for a puppet. Rackspace is kind of a Chef shop. So ops code kind of, while Matt was drinking last night, agreed to write a broker for Chef. So if you see him, you should RAZR him. Be like, hey, dude. So that's the other one that embarrasses me. So we will... This is just because this is on his laptop. It has no... Yeah. Is this targeting Ubuntu mostly? It's not actually running anything. It actually even do Windows. Hyper-V and Ubuntu. You can do Red Hat, KVM. You name it. Yeah, through IPMI. You can register them. Actually, once you're booted in the microkernel, the microkernel occasionally checks in with RAZR at a configurable interval. I think it's 30 seconds by default. 60 seconds by default. So once every 60 seconds, that 20 meg image is booted in memory. Or 40 meg, sorry. 40 meg image is booted in memory. Your server says, hey, do you have a command for me? Hey, do you have a command for me? So is that fucking... You're talking about that? Yeah, you can... Yeah, you set your service to boot off of the network. So what it'll do is when it detects that you've deleted this server, it'll do an IPMI reboot. Yeah, it will cycle when you're already provisioned. So... Yes. If they're already provisioned in RAZR, you would delete the active model and reboot them however you would normally reboot servers. Yeah. There is one. So what we're doing is we have a little Chef role called Reboot that we apply to it. So... That's just because this is only one part of what you see. It's one part of a sequence of events that lets us... So it allows us... Yeah, you can't register the... You can't register a BMC. I screwed it up. You can register a BMC with RAZR and use RAZR Slice commands to interact with that BMC. The piece that we haven't... associated with the node itself. When we register a node, we need to check and see if there's an associated BMC. So where we're going with this is... This is kind of how we are doing our CI for... Well, I guess what's now grizzly code. It's going to say Folsom. So this gives us a base install in about three minutes. Then we punch down Rackspace Alamo which is the private cloud software where you go download it. Afterwards, that takes about 17 minutes. We won't show you all of it. Yeah, I'll just go back. You guys have seen a Nubuntu install, right? Yeah. If you're outside, it'll be more than happy to show you. Yeah. So in here, we do this perform a battery of tests and this is really where we ensure that OpenStack running on this hardware that we're certifying matches. So this includes our DevOps framework for Chef or whatever. We'll also do provisioning devices, things like that. We're... Internally, we just have a couple of vendors for storage that we're talking about because it's enterprise volume. We've been actually working really heavily with NetApp on this. They're kind of ahead of the pack, so to speak. Cool. So let's go... So the next thing I'll show you is these tests here take like six hours to run. They take forever. If they stress test, they make sure that performance of here is up to far. It makes sure that every little component that can be set is set. It's pretty thorough. So I don't want to show you that. I will show you the walkthrough of kind of how it does. So this is what V would be like, what NetApp would see from there. Oh, you did. This is... This is actually something that probably none of you will see unless you have some hardware you want to certify, open stack certification. So log in, click it. This is the part that would take forever. This is an overview. This particular one because glance, we make glance fail, only took like four hours to run. So you click the little details button and it'll bring you to... What did I say? Is that showing out? It looks different on the little screen. So here you can see here's failure to create volumes. Here's the list of volumes types failed. So what this is doing is saying, like, hey the driver that you provided so did not match the expected results that we set. So this could mean it could mean a number of things, right? Most likely it means that there is either an undocumented configuration that we need to add in and make sure it gets added into the installation document for open stack and this platform. Just a variety of things like that. Once it's done, this is a little... guys just threw this in last night. I included it in a little certified achievement. Yeah, so the... So when you get here, it's going to show up on Rackspace's certification page. So anyone can go see like, hey, I want open stack. Here's the vendors I have. Let me see what platforms they have that will work with what I've got, right? Some might even be hardware you already have in your data center, but you actually have no way of knowing that right now. I don't have anyone going right now. So this is what we're trying to build. The... We're kind of in the works a little bit. Like I said, NetApp is the furthest along. So we're working through those vendors. So that's all we have I guess if you guys have questions. No? Yeah. Yeah, Razor is cool, huh? Yeah. Yeah. We do have an M-collective. Can everybody hear me okay? Here we go. Razor... The microkernel has an M-collective channel built into it. So it should be possible to use M-collective, the Marionette collective for those of you who are not familiar with it to add custom facts to individual microkernel instances. I think you might be able to do it if you had a map of MAC addresses for where you are in the rack or serial numbers for system cards. We collect a lot more than just what's in factor actually. We run LSCPU and LSHW and microkernel as well. Cody could show you a list of the attributes that are there. It's fairly extensive. We actually filter it down to make it more. We can get down to the level of which nicks are in which slots in the chassis if we really wanted to. So I think probably most of the metadata is there that you need. But somewhere along the line someone is going to have to map it into a rack position. They're going to have to store that somewhere. And we would just have to work out how to interface with that system to map it to a location in a rack in a large data center. It's certainly something that needs to be done. Not there right now. But then again, Raker is pre-1.0 at this point. Yeah, there's the list right there. So you can see the the other facts are collected through factor. And factor would pick up a custom fact that was inserted with the M-collective. There we are. Well, I mean, the community develops stuff. Open source as a whole essentially raises the bar. And so it's contingent upon us and everyone else who wants to sell commercial storage capability that we are dramatically better than that bar. Not to get philosophical or anything. Well, I think they're all very shy. But they're still staring at us. Oh, we're out of time? Did we hit time? Alright guys, thank you very much. You're very brutal.