 So, this is a session on open SDS, software defined storage, some work that's going on in that area. If you really don't care too much about storage, then feel free to zone out or, you know, find some other place and better time to be spent. My name is Alan Samuels, I'm an engineering fellow at Western Digital. I'll have to caution you, if you were a close follower of the schedule, you didn't see my name up there until about a week ago. Originally, this session was going to be given by a gentleman that's also part of this community that works for Huawei, and he had a conflict at the last minute, and he shouted out an emergency, can somebody do this for me? And I said, okay, I can do it. But I didn't have a lot of time, so he gave me the slides. So I'm going to walk through the slides, and yes, I've looked at them, but I can't claim to be intimately familiar with them. So there may be a question or two that pops up that I may be as bum-fuzzled as the rest of us are, and if that's the case, then I can connect you with Steve, and we can both figure it out a little later on. So if you bear with me, I think most of your questions will get answered, and we'll just dive right into that. So this is about storage, particularly in the cloud world, and if you sit back and you look at what's going on for storage in the cloud world, what happened? There's a simplified picture on the right of what's going on in storage in the cloud world, and I can tell you that that is a very simplified picture of what's going on, and to boil it all down, it's a mess, and it's been a mess for a long time, and a while ago, a couple of us got together and said, well, maybe it doesn't have to be quite as messy. So we looked at the kind of problems that people are running into, there's a short list of things on the left, don't need to, and for those of you snapping pictures, these will all get uploaded, and you can get nice high-res copies instead of blurry things that you look at later on, scratch your head, is that my finger or is that the picture? But hey, and yes, I have a staff meeting now, but thank you for rescuing me for my staff meeting. And anyways, the kinds of problems that are there are just legion, you've got issues of all these different frameworks, and they're all, well, not all, you've got the container orchestration frameworks, and they're pretty similar, but they're not exactly the same, and then you've got some of the other orchestration frameworks, this is an open source conference, so we'll stick to the open ones, like open stack, and it has a very different view of the way that things work, but we both know at the end of the day, what people want to do, particularly in a large-scale environment, is they want to treat their storage in a way that is, you know, orthogonal to the applications. Storage is the family jewels, and storage is different. It's not like network, and it's not like compute. It's got state, and that makes things hard, and what makes it hard is that there isn't really anything that's universal, we get into it in detail, if you haven't spent much time in storage, you don't realize the sort of subtle complexity that you get into the different types of storage, and anyways. So enough of that, the kinds of difficulties that people run into is exactly what you'd expect in that mess, which is the developer walks in and says, this is one more complicated thing I don't want to learn about, can't you just give me some simple quick answer, and can't I go to Stack Overflow and find ten lines of code I can paste in to solve my problem, a little reason why it needs to be complicated for the developers. Using the applications through to the final storage is where the complexity is, and that problem is solved today basically manually in a lot of environments. The old story of the enterprise environment, gee, I need some storage, we'll fill out this piece of paper and submit it to IT, and you know, in a couple of weeks we'll get back to you, and you can have your storage, it works, but there, and that just doesn't work in the cloud era. There's a little story I like to tell, it's a little off point here, so bear with me a second. About two decades ago I spent some time in the networking industry, and if you looked at networking prior to the sort of the big bang of the internet in 95 and 96, some of you may be old enough to remember that, days, others just bear with me. If you looked out at the state of networking, there were a dozen different networking technologies, and if you wanted to figure out who was going to be the winner, because you know, signing on to the wrong networking technology is a really painful experience, okay, because networking connects to everything, if you get it wrong everything has to be changed. So if you were sort of tasked with the problem of saying what's the kind of networking that's going to win out, you do the sort of standard product management, and now you make your chart, what are the categories that matter, let's get the technologies, score them, who's the winner and who's the loser, you've got Token Ring and X25 and FDDI, and just a whole list of networking technologies, and you'd go, you really couldn't actually figure out who was going to win. Everybody had their good points and everybody had their benefits, but actually when you looked at it, you could see there was a clear loser. The worst, the poorest, the crappiest of all the networking technologies is TCPIP over Ethernet, okay, it turns out on that chart, it only wins in one place, everything else it loses on, manageability, reliability, you know, all the abilities, et cetera, but it does win in one category, and that is scalability. It turns out it scales much better than all the networking technologies, and a funny thing about scale is it turns out if you care about scale, then scale is the only thing you care about, okay, and that's ultimately why we ended up with that, as the foundation of pretty much everything, and that actually touches back here because what cloud is about is scale, and the challenges of the cloud era are the challenges of dealing with scale, and it's a different mindset than the old guys like me are used to, you young people are used to that, but changing your thinking from performance to price performance, which is fundamentally what cloud is about, because if I care, if I have a scalable application, a true cloud-based application that really scales, then I can get to kind of any performance level I want, just to question how efficiently I do that, so that conversion from performance to price performance isn't a small thing, it's a complete rethink of the way you do things, and one of the things that that leads to is that in order to scale, you got to get the human out of it, okay, because humans don't scale and the humans that get involved in the middle of something will prevent anything from scaling, except that you don't want to get the human out of there, because the scale programmatically have this nasty tendency to occasionally consume all available resources, okay, now if you're sucking up all the CPU time, it's not so bad, you can just go around and shoot a few things and poof, you've got your CPU kind of the same in networking, that ain't true in storage, if you run yourself out of storage it's usually painful, and the pain lasts a really long time by an even actually worse problem, which is if you use the wrong kind of storage, then you have a bunch of data that's stored in the wrong way in the wrong place, and when things go wrong it goes really wrong, okay, so loud error, what's really going to matter, we connect this sort of legacy of all these storage types to a programmatic environment so that we have the human out of the loop, but we move to sort of a policy-based scheme so that the human is sort of preventing things from going too far off the rails, but not getting into the way of the actual responsiveness in the cloud that we want, so in mind a bunch of people got together and created this thing called OpenSDS, and it's a bunch of industry players and, wow, somebody just keeps advancing this for me, that's really cool, anyways, a bunch of people got together, a bunch of industry players that try to address the issue, because customers that are trying to create a policy-based storage environment for their applications, there really isn't a lot of good solutions, there's nobody really working on that part of the problem, so that's what we're trying to do, so I probably, you could probably guess what the mission and the goals are, the goals is to try to clean up the mess, make it easy for applications to consume storage, make it easy for the administrative environments to regulate the consumption of storage through some policies, so that the application ends up connecting to the right kind of storage in a way that's understood, and in order to do that in a way that's going to get accepted, because this is not the first, this is not the second, it's not the third time people have come in to try to create and clean up this mess, I think what we're trying to do a little differently this time around is to be more community-oriented rather than more vendor-oriented, which is what's happened in the past, okay, so a short timeline, this all got kicked off back in March of 2016, I'm not going to walk you through all the details of this, but it's sort of slowly gaining momentum, there's a number of specs, and I'll get all the URLs and all that stuff toward the end, and a bunch of people have been working on it and contributing code to it, we've had some demos of early functionality back in April, and we're starting to gain some more membership, so the community is growing, it's very early days, not a lot of people in it yet, but growth is good, and more importantly, particularly with some of the later entrants, we're starting to see users coming in, not just vendors, okay, so having that mix of diversity in the members of the community I think is a good thing, and so why do you want that? I probably talked about that already, but one of the key issues is we're all going through this cloud transformation, some people are kind of already there, many people have existing environments that are partly cloudified and are moving toward that, but one of the things that is really driving the cloud era is this continued abstraction of the hardware layer, and more of an anonymization of that, and that's represented here by the no vendor lock-in, so we want people to be able to optimize their consumption, but without getting tied into specific vendors, okay, which sounds strange coming from a vendor, but give the customers what they want, that's actually what people want, usually you do better that in the long run. Some of the key advantages for the developers is that they'll be a standard API for managing storage, and it should be common enough to handle the vast majority of the common use cases, and sophisticated enough to allow orchestration environments, provisioning, life cycle management for the storage, all of those things to be automated in a user's workflow and to operate on the storage complex in a vendor independent fashion, so, you know, and the sort of old Java story of develop once and run everywhere, that would be the goal, you know, the, I think the classic conundrum of don't mistake a clear view for a short distance applies, but then again, if you don't start, you'll never get there, so, you know, I was talking, you know, as a vendor, you have to ask the question why I'm here, you see a number of different vendors. Western Digital, we sell storage to everybody, so, you know, in the razors and razor blades, or if people like the more war-based, we supply ammunition to everyone, so we don't really care who shoots whom, that's not entirely true, but actually what we do care about is frictionless deployment, you know, okay, that actually does impact our business directly. If people are inhibited from getting their environments to run, they're inhibited from hitting production deployments, and that shows up in my bottom line, so that's actually what we care about, and, you know, we see this as one way to get to that sort of frictionless deployment environment for people. So this iChart is all of the thoughts around where this product will go, but basically, if you look at, and I have to apologize, I am color blind, so if I say the wrong colors, please give me a little bit of slack, but, you know, there's kind of the bluish things on the left, and I don't know if those are orange or yellow over the right, and then there's the, I don't even want to guess what the bottom colors are, but the product consists of a set of connectors on the top that connect the specific storage environments into the common controller, and we'll come back to what the controller provides here in a few minutes, but, you know, there's a list of the different environments, there's, today, we actually already have an OpenStack connector and this CSI connector, which covers most of the container environments that are being built, the container, as an aside, the container guys, you know, Kubernetes, Docker, Amazos, under the aegis of the cloud native, have come together to create their own container-based storage specs, so at least in the container world you'll have something that looks regular and similar across, that's called this CSI, which is the container storage interface, I think is the right name for that. Anyways, that's a similar kind of transcoder and connector, and we actually have a back-end implementation for that underway right now, so that'll be usable. OpenStack through both Manila and Cinder is being done, and, you know, we'd like to see, you know, the Azure and the SIFs show up eventually. The controller itself, the expectation is to support block and file and object services. The block services are what people are focusing on now, we're just starting to turn towards the file, the object is in the distance, but underneath of that there is a standard substrate for vendor-specific drivers, so you've got on the far left there, I've got native drivers for basically legacy storage equipment, and then you've got some other connectors that are sort of well-known today. Cinder and Manila in the OpenStack world, Swordfish and LibStorage are also environments that are popular in some spaces, and there's work going on to connect those into the open SDS environment, so once all that work sort of becomes available, you should be able to essentially plug in any of the storage that supports any of those southbound drivers and connect them into environments that are in the top half of this diagram. Ultimately, all the things that are on the right-hand side are things that we think will fit into this environment. There isn't actually any work going on that right now beyond I think some pixels on this slide, because we're all busy making the data path work, but if we're successful with that then we'll be able to turn our guns on that stuff. So I'll spend a couple of minutes on digging into the technology of that and then I think we'll be done. So basically the open SDS breaks down into two pieces. There's what's called the controller, which is actually the sort of middle piece and it's the piece at the deployment environments plug into, it provides the policy enforcement and the orchestration and as you'll see in a couple of minutes when I get into some details, there's actually quite a bit of work that goes into the orchestration piece, particularly because not only is this vendor agnostic cross vendor, so you can actually mix storage from different vendors into common pools that are then made available into the applications and I'll show you some examples of what I mean by that in a minute. You know I took the different kinds of drivers that are available in it, you've got of course vendor specific, but Cinder and Manila, so any existing Cinder or Manila driver will work just fine here and that actually picks up a fairly broad number of pieces of equipment, number of vendors. Most of the vendors I think that are out there support Cinder and Manila as it makes sense for their storage. The flip side of the top half of that is the plug-in in the compute framework and it provides a standard interface for the high-end demands, which you could either talk to natively from your client or through one of the orchestration environments. So in the case of the, I mentioned before, the container people in the cloud native, they're using the CSI, if you were running in one of those environments, you could talk directly to the open SDS controller or you could go through the CSI interface to talk to that. Open stack, again Cinder and Manila, it can plug in as a Cinder or Manila driver itself into your cluster and we're working on support in some of the more enterprise oriented environments. So the overall desire is to try to create not only a universal API, but the ability to merge together disparate types of storage into a common API substrate. And what I mean by that is the reality is everybody's data center has got legacy equipment in it. These things change slowly. Nothing changes more slowly than storage. Storage lasts forever, okay, and at least forever in computer years. And as you roll new equipment in and as your old equipment gets rolled out, you have the basic problem of orchestrating and placing of the various chunks of data that you're going to use. In the middleware, the controller that I talked about, it has the ability to provide storage across multi-vendor environments and to do placement within that. So as you come into provision storage, it will examine what's available in the general pool of storage and select storage from that based on the policy and then connect you into that. And what that means is that if you've got some older storage from one vendor and newer storage from another vendor, you can actually create a common pool and export that common pool as just, in this case blocks, a commonly available pool of blocks and when you provision some block storage, it will pick from one or the other based on the availability of the dynamic performance measurements that it's doing. But a key part of that is the placement policies, those are all centralized, okay. Now they're centrally centralized. There's not a single controller. The technology is being built to run in a clustered environment, no single point of failure. We're storing the rules today in ZooKeeper and there's also a connector going on at CD. The storage interface is sort of abstracted. It'll work in a number of different environments. But the key issue is that the other network service, they have the same reliability and availability that you would expect out of any durable network service. So I think I pretty much so these are my requirements that I'm operating under. I talked about the cluster view of storage. We want to be able to provide a single view of your storage in your cluster independent of where it's coming from, crossing vendor boundaries, etc. You want to be able to do all of the life cycle management for storage programmatically. So you'll get the admin out of that loop entirely. A key issue is that the clients, the connectors from like connector from Docker or even if you want to go native, the key is that at that API it's common. There's no vendor specific code in that. All of the vendor specific code is compressed into the open SDS process itself. And you shouldn't have to configure your clients. They should be able to discover and just have a full set of discovery protocols and a common API for them. So let's see. I think I pretty much talked about everything that's here. Dig into a little bit more. This is a list of the kinds of storage characteristics that are actual. Most of this is associated with block, but a lot of it applies to file on the lesser extent object, but it gives you a small taste of the kind of issues that you run into in dealing with storage in a system environment. And in order to really automate this environment, you have to make all of these options potentially available again under control by a policy to the client applications that are running. Because while the storage media underneath of it, the drives, they're all pretty much the same. The reality is that what we're actually controlling here is access to storage management software. And that software, software defined storage, takes the raw media the raw media and synthesizes that into something that the durability and availability characteristics that we want. So being able to connect the application to the right instance of the storage management software, that's actually what's going on here. So at a very high level, what we have is essentially a catalog of storage profiles, just a system-wide list of what's available. And a profile is a policy, essentially. It's pretty much what you would expect. It could be, I want slow, triple-replicated, rotating store, that could be a policy. Or some maybe erasure-coded flash, that could be a different policy. From an operational perspective, a lot of people just want to think about this as bronze, platinum, and silver. Or if you like, I like long-term storage or short-term storage, pretty much any of those kind of paradigms. That you might choose to apply to your storage environment, this sort of profile and catalog scheme will map onto that fairly well. So your applications will operate under the umbrella of a storage profile. They'll select one. Naturally, there's a security protocol to make sure you have access to the right profile. But once you've got that, then the profile essentially dictates what you can do. Okay, so if you've set up a profile for triple-replicated, rotating store, and you ask for that profile, then that's what you're going to get. If you say, gee, I really want six-way-replicated store or erasure-coded, then you're going to get an error message that says that's not in the profile for this guy. What you generally find is that the applications, they actually don't really want to concern themselves so much with that. They tend to divide the storage conceptually by the way they consume it. It's the mapping of that framework into the physicality that the catalog and the profiles provide. So I mentioned before the ability to cross-vendors. In a clustered environment, we've designed that a controller to have both regions and zones, pretty much the de facto standard for those things in the cluster world. But when you get to pooling of storage, we recognize that there's sort of a physical pooling of storage, which represents the pooling of things with common characteristics. So you're going to want to pool your rotating store into a different pool than your flash store, your dual-replicated store different from your non-replicated, etc. Those are sort of the physical pools. And the virtual pools are the things that you actually export to the users, where you can take multiple physical pools. And typically, a physical pool is going to get mapped to an individual vendor or a slice of a vendor's equipment. And the OpenSDS framework itself, when you make a provisioning request out of a virtual pool, we'll select out of the virtual pool one of the physical pools based on the current load factors, etc., and do the placement and the provisioning within that particular physical pool. An individual request and a profile has a long list of things in it that you can set or you can set that or from the policy or enable or disable it. I don't think anybody really wants me to go through all those in detail. So, I think this is just sort of another way of looking at the product in a very simplified version. You've essentially got your orchestration environment on the top with plug-in talking to the OpenSDS API. And that has the, underneath of it, the controller. And the controller has the policy engine and the placement control engine and its own set of metrics to keep track of that. And then underneath of that, you've got the storage-specific drivers that get the job done. So, when you map this into a cloud-native environment, the integration with the container-based cloud-native people, the Kubernetes, Docker, and Mezos, those people are using the cloud storage interface, the CSI. That's a native interface into OpenSDS. And you see that represented directly in their documentation. But what OpenSDS brings into that is what's missing in that environment, which is policies. The CSI is very much a particular application. It talks about there being pools of storage, but there's really no more description beyond that. You can select from a pool and provision a piece of storage in the CSI, but the ability to manipulate the pools and see what's going on in them and interrogate them, that's actually not something that's gone into. That's one of the things that OpenSDS brings to that environment. And I'm getting the nod here. Similarly, in an OpenStack environment, it's essentially the same situation where you've got a lot of capability that's been exported into those environments, but it's not visible what's going on. OpenSDS allows you to get that visibility. The usual sign-off and good information. Everything here is open source, uploaded to GitHub. And wow, it just jumped again. Sorry. I wonder if my laptop has a bug or a virus. We are looking for new people to get involved. If you have an interest in this, give us a ring. And I'm done. Are there any questions? It's strictly a control plane, yes. Correct. Right, that's the same. I mean, all of these are control plane environments. Once the application's connection to the specific storage is created, then they use the same old protocols and everybody else is out of the way. So that your basic application performance isn't affected. Your reads and your write speeds are totally unaffected by that. It's going to be a function of whatever actual storage gets selected. Well, when we take over the world, yeah, but until then, no. Now, I think the issue of exposing the storage across data centers is a really nasty problem. And I think where the world is sort of landing on that is not something you want your infrastructure doing. So I think that's probably something that isn't going to get tackled here. Well, thank you very much.