 So welcome everyone to the designing Cinder session. So we're going to go on through some introductions and then the agenda and then we'll walk through the session itself. So Jim, do you want to start? Sure, hi everyone, I'm Jim Ruddy. I'm from EMC, I work in the Office of the CTO in something called the Open Innovation Labs. And my name is John Griffith, I work at SolidFire. I'm the PTO for Cinder, I'm a software engineer at SolidFire and OpenStack is what I do. My name is Kenneth Hoyt, I am a technology evangelist of Rackspace and Rackspace is partners with both EMC and SolidFire, so my job is to keep them from fighting. Good luck. So quick agenda is we're going to, let me ask how many of you are familiar with Cinder or currently using it? Okay great, perfect. Let me ask another question, how many of you are using Cinder with commodity storage? How many are you using it with some kind of enterprise storage solution? Okay, that doesn't equal the number of people who said something out of here. It's ephemeral Cinder. Okay, so what we're going to do is we're making, we'll walk through a little bit what Cinder is, but the assumption here is that you guys know, have the basics down. The meat of this presentation really is going to be looking at real life examples of how to actually deploy Cinder in production environment with commodity storage and with also enterprise arrays. And then hopefully at the end of the session, depending on how much time we have left, we'll take some questions and then maybe we'll do some kind of a demo, but our main focus is going to be talking about actual use cases and lessons learned from deploying Cinder in an open sector production environment. So, I'm going to let John kind of do the introduction on Cinder. Okay, so for those of you that are already familiar with this, you'll have to excuse me, but I'll try to make this brief. One of the first questions I get asked a lot, I talk about Cinder a lot, and one of the first questions I get asked is always, what's the difference between Cinder and Swift? So on the Cinder side, we have block storage. On the Swift side, it's object storage. That's kind of the, I mean, that's the big thing, right? So significantly different use cases, significantly different models, how it's consumed, everything about it. So they are not the same thing. They definitely have their own unique use cases. If there's any questions or anything like that, I'm happy to answer them real quick. No, okay, good. So we can just blow by this. Oh, by the way, if you have questions, there's a mic in the middle. Yeah. So probably easier if you try to come up to the mic. So the whole point of Cinder is to give you your traditional block storage. And when I say traditional block storage, we're talking about things like disk drives. The way people like to think of it in an example that Ken uses a lot that I really like is the idea is you're taking a very specialized USB thumb drive and plugging into your OpenStack instance, right? So that's kind of the idea. It's your persistent block-level storage that can be used by Nova either as a secondary attached disk or even as the boot disk for the root disk inside of Nova. Cinder itself will manage all the provisioning, creation, attached, attached, everything else that you need to do with volumes, with block storage volumes. And the whole idea is, as an end user, you don't have to worry about things like what is the back end? What's the interface look like? How do I use that API, et cetera, et cetera? The base features in the Cinder API, it's really actually kind of simple. Of course you have create and delete. We have a thing called volume types inside of Cinder. And the whole idea of volume types is to give you the ability to expose custom functionality without breaking the API. You have the ability to do clones. Some back end devices allow you to do highly efficient fast cloning and things like that. You have the ability to put an image on a volume so that you can boot off of it. Snapshots, which are just point-in-time copies. Creating a volume from those snapshots is another thing you can do. One of the complaints people have is, I have the snapshot. I can't do anything with it. Well, sure you can. Create a volume from it. We give you the ability to back up Cinder volumes right now. So you can back up a Cinder volume to an object store, any S3-compatible object store. There's even a tape driver right now for TSM. You can transfer the ownership of a volume. So one tenant can create a volume and use it and then decide to give it to another tenant. You can set that up inside a Cinder right now. And then, of course, we have the scheduling filters and things like that that kind of dig into the guts a little bit more. John, do you want to talk a little bit about what's the use cases that the Cinder team envisioned when they were creating Cinder volumes? Yeah. So in terms of the use cases, the things that we look at is, initially you had, of course, object store, which is great for doing things like dumping media files or photos and things like that. But the reality is, you know, as the cloud evolves and things mature, you actually want to run real workloads in your cloud deployments. I mean, that's the whole point, is to change your IT department and use the cloud. The problem is you need persistent storage to do that. You need performant persistent storage to do things like run a MySQL database or an Oracle database or whatever it might be. So that's the idea. The idea is to satisfy the need for persistent, high rate of access block storage. You know, whether that be as a secondary attached model or as the root drive or whatever the case might be. So, stay on that. Okay, that's you. Yeah, we're good. Yeah, so at RackSpace, the other primary use case for using Cinder actually is the ability to do rapid recovery. So one of the things you may know if you're using OpenStack is prior to Cinder volumes, there was really just the concept of ephemeral storage for SCSI access. But the problem with ephemeral was if the node went down, well, first of all, if you destroy it, if you terminate the instance, all the data gets purged. But also, if the compute node went down, you lost access because typically that was direct attached storage. So one of the things that Cinder can do is since it is a storage that can be attached and then detach and then reattach another node, instance, if your compute node failed, the idea is you could bring up an instance on a separate compute node and then just basically reattach that volume to a new instance and you would retain all the data. Let's say that's running on a MySQL database. But anyway, so let me talk a bit about the commodity storage use case because in RackSpace in particular, we find that is probably 70% of the use case, especially in Greenfield type of deployments where there's no existing enterprise storage solution in place for an OpenStack deployment. And from a hardware standpoint, I'll have a picture later to kind of walk through this, but from a hardware standpoint, the components are really around a control node. So in OpenStack world, all the controlling services typically reside on two or more nodes that you've created and that's where you put your metadata database and your, say, your API endpoints. And then there's also storage networking, which we'll talk a bit about later, and then the storage itself. And then on those nodes, you're going to basically spread out your software components of OpenStack syndrome. So on the controller nodes, typically, it's where you put your sender APIs, endpoints, and then your sender scheduler. So the API endpoints are basically how you interact with OpenStack to say, hey, I want a volume created and I want that value, volume to be attached to an instance. And then on the scheduler, basically it's a process that runs... Okay. It's a process that runs on the control node that basically adds, when you are provisioning a new volume, it looks at all your storage backends and basically low bounds, across all your storage backends. So that sits on the controller node itself. And then because, typically, sender is running as an iSCSI protocol attachment, you want a iSCSI network. So, again, in RackSpace, we do not recommend trying to run your normal Ethernet control traffic on the same network as your storage traffic. So typically, you want a dedicated 10-gig iSCSI network that goes from your back-end array or commodity server to your compute nodes, which would basically then pull a volume, a virtual volume that's served out by your storage and then the instances would basically plug into those volumes. Next one. And then there's another software component that's called a sender volume. And that is the piece within sender that basically takes a volume and exploits it out to an instance to be attached to. So in a commodity storage type environment, the way to think about a commodity storage type of deployment within OpenStack for sender, think of that as typically a server that has some internal storage that essentially functions as a virtual storage array. And this is a recommended configuration for how much memory and how much CPU power you need to run that virtual storage array. So this is a high-level diagram of a typical OpenStack deployment using Rackspace Private Cloud. So you see here, we have two controller nodes for redundancy, and you see that's where we have the API service and the schedule service across both controller nodes for failure purposes. And then the compute nodes themselves really talk to the sender node volume using the iSCSI network on the bottom. And obviously we want redundant network switches if at all possible. And then the sender node itself, again, in a typical Rackspace deployment, is basically a Dell server that has anywhere from six to eight drives in it. And those drives can be SSDs, they can be Neoline SAS, they can be 15 or 10K SAS, depending on customer requirements. So why look at commodity storage? So obviously the initial cost is fairly low, especially if you only need, say, two terabytes, rather than buying a big storage array to start, you can just buy a Dell server or an HP server, or even a white box server, put sender volume service on it, and now you have a storage array, quote, unquote, to use. So obviously if the server team that's typically running the OpenStack deployment, they don't have to go to the storage team to get to procure the resource. They just have to procure a server with some amount of storage behind it. And again, the reference implementation for sender and commodity is using LVM. So if you have a Linux admin, who's typically the one running OpenStack, they can set up a sender volume fairly easily. Some of the limitations, and I'm going to kind of draw this out a little bit more in some of the next slides. But one issue is there is no redundancy. The sender volume is essentially a single point of failure. Also a storage, I mentioned using a Dell server with eight, let's say, eight drives, that's your capacity scaling limit for that one sender volume. And obviously Kamai Storage doesn't... It only provides services that sender itself has as part of OpenStack. There's no other services such as QoS and tiering or replication that you would normally expect from an enterprise storage system. So let me draw out some of the examples on the next slide. So you see here in a typical environment, you can have multiple sender volumes servers running in an OpenStack deployment. But note that the two are not... It's a true share-nothing architecture. They're not sharing anything at all. So if a sender volume fails, you've lost all access to that storage until that service is replaced. Which obviously creates HA problems if you're running that type of load where you don't have HA replication at the application level. If you're relying on the infrastructure to provide it, you will not get that with a Kamai Storage use case. And I talked about the limits of capacity. So again, you can use different drive types, different drive sizes in a sender deployment. We typically use 15 case apps because a lot of times we're running things like Postgres or MySQL which requires that kind of IO performance. So if you look at a typical Dell server that we would use for a single volume node, we're talking just over two terabytes usable. So as you can see, if I need eight terabytes, now you're looking at four sender volumes, none of which share information with each other. So over time, if you start getting into the multi-terabyte type of deployment, now you're looking at an entire form of additional storage service that you must procure, power, and manage. Before we move on to the enterprise storage piece of the presentation, are there any questions about Kamai Storage? Do you mind going to the mic if you can so people can hear the question? So in the previous figure, you mentioned about MySQL, ATB, right? So can you please go to the previous slide? I'm sorry? Can you please go to the previous slide? Yeah. So here MySQL database with ATB requirement, but the MySQL database will be on the API or the scheduler. Can you please elaborate on this? So I think there might be some confusion there. So this database is basically, so is this some kind of an application you're talking about? Yeah, so I'm sorry. Space consumed by the volumes that are created on the backend screen. I'm giving an example of a application that may have requirements for some amount of storage greater than one Cinder volume can hold. Okay. So since the scaling limit of a Kamai Storage is typically what the server can hold, you're forced to break up that application across multiple storage nodes. And my whole point really is, again, if you get into the multi-terabytes, now the cost of ownership becomes very high. Okay. There's a number of volume servers you have to manage. Sure. Thanks. Any other questions about Kamai Storage, Cinder? Okay. John, do you want to talk? Sure. Okay. So one of the cool things about Cinder is you actually have choices. Right? So Ken talked about the Kamai Storage. I call it the reference architecture or the LVM implementation. So you get that for free that's built in, but it has the caveats that Ken mentioned. Right? But the thing that's cool is what Cinder does is it uses a plug-in model and a plug-in architecture so that what you can do is you can use other storage platforms and whether it's EMC, SolidFire, NetApp, CIF, whatever it might be. And you can plug that in and use that just as effectively as you could anything else that's built inside of Cinder right now. So that's the cool thing. The other thing that's really cool about that is you don't have to choose just one. You can mix and match. You can start with one today. You can start with the LVM reference implementation today by a SolidFire tomorrow, by a NetApp, by whatever. You can just keep going. You can just keep adding. So it's true scale out and it's designed to work in that manner. So it's a huge advantage. So this is kind of another take on Ken's slide. It's the exact same thing. And this is just kind of, I wanted to display the fact that it's truly plug-in. Right? So the idea is we took the exact same architecture, the exact same principles and everything else that we used for the LVM driver. And instead now we just popped into SolidFire array. That's the only difference. The other thing is you notice on here that whereas in the commodity storage the Cinder volume service had to run on that storage node. In this case because we're not using a server storage node. We're using an enterprise array. That service actually co-exists on the controller nodes and just talk to the whatever enterprise away. I think through the IP network. Yeah. It's true. So in this particular example and in most examples we have what's called a management IP that is over the one gig network and you get to specify that in your configuration file and that's where the API calls are going through. So what happens is that volume service that Ken had talked about before that in his example is running LVM and doing all these fancy things and stuff like that. In the example when you plug in an enterprise storage device or some other open source whatever it might be the differences in those cases what happens is that volume service is actually not doing much. It's just basically turning into an API interface and that's it. It's a driver making calls to a device and that's about it. So this is cool but if you go to the next slide. Okay. So this is what this is cool but this is even better right and unfortunately in this slide I forgot to put the iSCSI connects but just but this is an example you know this isn't uncommon to see so this is an example where you see we've got a solid fire cluster an EMC cluster and the reference commodity storage implementation all inside of one single open stack deployment and that's pretty cool right because the idea is the end user the tenant that's actually consuming this they don't know anything about what's on that back end and they don't have to. That's the beauty of it. So Cinder will let you do a couple of things you can either have it set up so that a user can specify which back end to go to based on some sort of mapping arbitrary mapping that you set up or you can let it just go ahead and figure it out and do it on its own based on performance requirements capacity availability things like that. So. Yeah. I can and I'll touch on that in a second on my specific product because I know the most about that right because yeah yeah he asked if I could comment on the ease and functionality that it's available in terms of API in terms of managing these different back end devices right. Yeah. I think the point it's still that and you made this point in the panel is the way open Cinder works it's a fairly basic level functionality it doesn't quite get into allowing you to essentially set up the underlying array in a particular fashion so you basically the storage if there's a storage team that does that they would still have to lay out the volumes within where there's a NetApp EMC or SolidFi then hand that off essentially to the OpenStack team who then can present that as a So partially yeah so so Cinder and OpenStack do not take over the role of the storage admin so to speak the part that it takes over is instead of an end user saying hey I need a volume with these characteristics of this size and submit a ticket to the IT department that goes to the storage admin and he provisions it and then sends the info back the difference is the storage admin now just takes these different devices right and configures them into a pool inside of OpenStack and then they're ready to go they're available and they're ready to use and then the end user is the one that makes the calls and says hey give me a volume and the APIs automatically provision those volumes on those back ends and stuff like that the actual administration aspects and things like that yeah there are differences and so that's where we kind of get into when I talk about the enterprise advantages so there are definitely advantages things like simplified scaling so Ken pointed out you know scaling limitations and things like that that you run into and some of the isolation in terms of the max size and stuff like that on the nodes some of the enterprise devices are going to give you the ability to actually scale out horizontally without changing anything in your OpenStack deployment so there is no need for any admin interaction or anything like that the other big thing is is when you start talking about high availability when you want things like failover and HA and stuff like that you know the LVM implementation isn't necessarily going to cut it and when I say enterprise storage advantages that may mean another open source option or something like that I'm just using that as a general term and then that one last thing on that I did want to touch on was one of the biggest things one of the biggest advantages of having these drivers and these enterprise drivers and these plugin models is the ability to actually repurpose equipment that you have so you know people don't just all of a sudden start and have OpenStack right out of the gate right so they've got an existing infrastructure they've got existing equipment so the idea is it's great if they can actually use some stuff that they already have so that helps the on-ramp one of the things I like to talk about a lot is you know in the open source community people kind of like oh vendors, oh enterprise proprietary is bad I don't think it is, I think it's great there's a great platform because it gives you choices the more choices you have the better off you are in my opinion if you don't like proprietary you don't like vendors then don't use them but the reality is the more choices you have the more competition there is inside of OpenStack the better everybody inside of that community has to be so it's a win-win for everybody and then sometimes you know combining open source and proprietary things it actually gives you the best of both worlds and it's a win-win for everybody so the hard part is inside of Cinder there are 24 different drivers 24 different supported back ends that you can choose from that are in-tree so how do you choose which one to use that's a tough thing to do so the good thing is we strive for consistency inside of Cinder so the idea is all the base functionality should just work regardless of which one you pick up and which one you use the features and characteristics that's what sets them apart and that's where you get to start making your choices so one size does not fit all so the whole point of this is to figure out what do you want to do with your cloud what is your use case going to look like and then from there start making choices on what you need so a lot of the things that I run into and talk to people a lot about are how well is it integrated and supported inside of OpenStack how much work do you do inside of OpenStack how much to work does that vendor actually do in OpenStack and how much do they know about OpenStack so can they actually support the product how flexible is it can it scale what does scale mean to them is scale adding bigger drives is scale scaling out and adding nodes so on and so forth automation how easy is it to manage can I automate all of the management back to some of what you asked some products you can so it kind of depends predictable performance not just is it fast is it predictable is it steady is it consistent it really comes down to what am I going to be using it for and then the hard part is what might I be using it for in the future but the good thing is OpenStack it allows you to grow and scale out and change things as you grow and learn so and that's kind of this point you're not locked in the whole intent is to keep you from being locked in the whole intent of Cinder and its abstractions is to keep you from being locked into one particular device you can easily change the back end you can migrate the data if you need to and again like I said expect the same base level functionality so so my company SolidFire this is what we work on we have a scale out horizontal scale out clustered storage appliance I'm not going to go into a lot on this because we're going to run out of time and I want to make sure Jim gets a chance to talk here so feel free to grab me afterwards and I will talk to you more about quality of service and the things that we do but I think with that we should probably so do you want to see if we have questions I'll do the slides so how many storage administrators are here today? Anyone? so when we were putting this session together you know we started talking about like Cinder and how everything sort of plugs in but we also wanted to talk a little bit and this was to your question earlier is how does the storage administrator work in an open stack environment? so the first thing is that the old school storage administrator was sort of the master of his domain when he had to design his storage infrastructure the application owner came to him and said here's the Iops I need, here's the latency I need here's how I want to design my environment for my application to run correctly and he absolutely understood those requirements and how to build his SIN or his NAS infrastructure to give those services to the application owner but things have now changed, right? It's not the application owner who's coming and talking anymore directly to the storage administrator or cloud architect who's coming to the storage administrator and saying I don't necessarily know what the Iops are going to be I don't know what the latency of these applications are going to be but what I need is a certain type of profile of storage that you can provide to me and this is called a pool so the current thing is it's all about getting into the pool and in Cinder when we talk about all these things that Cinder is offering really all Cinder is looking for is from an array or some sort of SDS device the ability to create a pool of storage and create lones on that do all the masking and then ultimately give it to the end user and this is a big change for the storage administrator because the consumption of his resources is not by another administrator it's by some user out there who he has no idea what he's going to be actually using so when we design these Cinder environments we have to design profiles of storage that can actually support the workloads that are created by an end user who doesn't necessarily know what he's actually looking for and that's why you see the rise of all flash arrays or commodity storage working very well together in a Cinder environment so I can offer all of these different type of service through one API to be consumed by the end user so we have a demo we're going to run real quick here to sort of show you how that works you can start the demo I'm not an Apple guy I have no idea how to use a Mac I'm an old school I guess I don't know but we're going to show you this demo of how this works I'm not going to show you this demo of how this works but you can sort of ignore the fact that it's showing you EMC and sort of look beyond to the concepts that it's showing so if I'm a storage administrator how do I offer these data services in a pool functionality and provide the quality of service and the type of things I'm used to supplying as a storage administrator like DR, Snapshots, all the things that Cinder may not be able to provide it doesn't look like it's showing up you have to drag it over to the other screen there you go Apples are so tough people say they're so easy I just want to play it from here here we go so what you're going to see here is sort of a way of offering services to Cinder so again I can directly from an array offer a pool of services to Cinder and the volume services and everything can schedule against it or I can use something like SDS which you're seeing here is EMC Viper a software defined storage platform that allows me to take all sorts of different arrays and creates different types of pools of storage that I then offer to Cinder and in this demo what's going to happen is I'm using a couple different arrays that I have I'm doing just one-to-one mappings where I'm creating pools of storage on an array I'm offering it to Viper Viper is then offering it as a one-to-one up into OpenStack for Cinder to consume and we can see here in Unisphere I have these different tiers of storage and what Cinder does is that as I talk to this controller I then get directed to the correct pool of what I'm trying to consume based off the volume type that I created within the OpenStack and then it actually creates dynamically the LUNs and does all the masking on the back end and I do this through a flow so here what you're seeing is you know here's the typical horizon dashboard and you're going to see that I've already created a couple of volume types and I've already provisioned a couple of different LUNs and one of the really cool things that I really like about Cinder are different controllers and I can have controllers talk to a certain raid group or a certain volume type but then I can keep adding controllers and it's pretty simple, right? So you're going to see here is how simple it is to add a new controller within OpenStack is that this version, this controller here I've never installed Cinder on it it's not a Cinder controller so I go to YAM I do the install and then all I have to do is tell it is where the database is located and then I can become another controller within my environment and then I can actually go and consume those resources that I'm actually offering in this example through Viper Sure so Icelon from Cinder in this example we're consuming it through Viper we don't have a direct Cinder driver that we've released for Icelon so VNX we do have a direct driver or you can consume it there was a slide that I whipped through really quick that showed all that we can go back to that or we can get to the information but almost all of our storage platforms today have a direct driver that's either in trunk or working towards getting in trunk to have available to use against Cinder Excellent So you can see this just goes on where I start actually what I'm going to do now is I'm using GitHub to get the driver for Viper I do the install once the install is done I'll then start going through creating a volume type from one of the services that I see offered within Cinder and this is all done within Python I can get all these drivers for all of our arrays are available up in GitHub other than the ones that are available in trunk so we have sort of two things going on we're offering our drivers to get directly released through OpenStack and the ones that either we haven't gotten into the current Ice House district or working towards getting in did you know we offer it through GitHub for anything we won't email you we won't know that you have it you can go to GitHub and get it and it's yours to do with it what you please so cool that's it for those of you that aren't familiar with Cinder or anything this is definitely a little different model right now I think it's going to change as he said for drivers that are in trunk currently today typically the configuration in the process is a matter of going in and making an edit to the comp file in my case for example it might be three lines and then restarting the service and that's it until Viper is actually in trunk and stuff like that you're going to have to do some extra stuff so some of that doesn't necessarily apply if you're not familiar we can take one or two questions but these are some resources that talks about Private OpenStack Cloud and how can that be used with Cinder so I encourage you to I think these stacks will be made because it's available somewhere so don't worry if you don't see it if you can't capture it all now but before we end are there any questions that you may have you could try to walk to the mic if at all possible and if not we'll repeat the questions that's not true there is fibre channel support so iSCSI is predominant so when did the fibre channel go into it so fibre channel went in last release in Havana and then fibre channel's zone management just started to go in so so one of the things you'll notice if some of you guys are struggling with the whole open stack project versus a product so one of the things about a product that for example Rackspace Overhead does is we don't just take everything in trunk and go here you go we take things that we think are production ready so prior to this release since there was no zone management fibre channel zone management we decided it wasn't ready for production yet so we went and buy SCSI we will probably support fibre channel at some point soon so my question is about in enterprise sender deployment right in that case when do you recommend to have more than one sender volume node would there be any conflict that if there are multiple enterprise arrays if you have multiple drivers sitting on the same sender volume would you recommend a separate sender volume for each so currently sender gives you the choice you can actually do multiple sender volume services for each back end driver or you can do what we call multi back end which basically does a distraction and lets you actually just iterate through we make a fake host name for each back end name but it still all runs on the same node still but are there any scalability requirements or something to have more not really we've talked about and we're looking at different things that might come up in terms of HA and things like that that we might be interested in currently with most drivers like I said it's usually just a pass through interface so there's not a ton of requirements in terms of resource usage you're just talking about RPC calls usually thank you one more question that's it is there one more question now so you mean just the whole gamut so in general there are multiple protocols that are supported in sender today they include fiber channel iSCSI of course and there is NFS some NFS support that's being added in there as well but eventually that's going to well there's some implementations there's an NFS that actually then is used as a sender SCSI volume NetApp does that I think maybe in the center there's a few people that do that so NFS but it's not used as NFS it's used as SCSI it's used to pretend NFS is block I think our time is up thank you for coming we're available for questions after