 Okay. All right. Good evening, everyone. Welcome to reshaping storage for the next generation data center. I have actually three co-presenters with me. I'm going to cover the SDS section and then transition the other three topics to the three co-presenters. So today, we are going to cover what are the storage challenges that we face. What exactly is the software defined storage vision that we have within Intel? Then I will touch upon what we are doing within the context of OpenStack on the SDS controller, what we really would like to do and what we are doing currently. Then we'll switch over the discussion to Ceph and Virtual Storage Manager, which is the management product. We'll focus on Fujitsu Hyperscale Storage for OpenStack, which is based on the Ceph and Virtual Storage Manager software stack. Then we'll talk a little bit about OpenStack Swift and our contributions to the community, and then wrap it up. So let's talk about the data explosion and the storage pain points. Most of you might have seen IDC report. Storage is doubling every two years, as you know. According to IDC report that got published in 2014, the digital universe will grow by a factor of 10, reaching to 40 Zettabytes in 2020. If you look at the chart on the left-hand side, there is a big gap between the data growth trajectory and the IT budgets. The IT budget includes both CAPEX as well as the OPEX. What this means is from the support and IT management perspective, roughly five times or so, the IT professionals will end up managing the storage resources between now and 2020. So what this means is there isn't really need to fundamentally think about how the storage is being managed in a data center, how it need to be viewed and controlled from a data center viewpoint. If you look at from the storage side, do we see this as a problem? Historically, the storage has been, you have specific applications and these applications are actually deployed on dedicated storage appliances. These appliances are actually optimized to run certain workloads. As a result, what you see is the storage resources are isolated in the data center, which means your capacity cannot be utilized more efficiently, which drives up the cost. That's one factor. The second factor is when you run into specific controller bottlenecks on a specific appliance, you typically tend to buy another appliance from the same vendor to expand the capacity or maybe the performance aspects, which results in the vendor lock-in problem. The third one is the dynamic workloads that are actually shaping up in the data center today requires a lot more fungibility in terms of data sharing. Same data may be used for different purposes. So you're looking at basically the cost aspects, the choice, you need to have a choice, as well as the need to support the dynamic workloads in a data center, is really driving the need for how we can actually manage these things in a software defined environment. So before I jump into the SDS framework and a vision, let's talk about how SDS fits into the overarching software defined infrastructure framework. SDA has three different layers in our view. The first layer is the service assurance layer. This is the layer where your applications are actually making the infrastructure request using SLS. You heard about in the previous talk, some of you are actually were in the previous talk, and there was a discussion about SLS specific mapping to carve out the storage resources in a data center. So that's exactly what we see in the SDI framework as well. The second aspect is the provisioning. Orchestration software, for example, provisions the resources and the goal of orchestration software is to provision and optimally allocate the storage compute and networking resources deployed in a data center to match application requirements. So the matchmaking happens within the provisioning management layer. Then the third one, which is also a critical building block is pooling your resources. As you saw in the previous problem statement, all these discrete resources, you should be able to use them in a more flexible fashion. So we are talking about how do we pool the resources, present the specific attributes, the orchestration software to be able to intelligently schedule resources in the data center. Now if you were to look at SDS, which is one of the key building block components, it has very similar attributes, but it is managing the storage resources deployed in the data center. There are four different elements in the SDS. So normally what you hear in the industry is, a lot of people talk about just one of those elements. What we see is SDS is more of covering four different aspects when it comes to storage resource management. One is the abstracting the software from the hardware which essentially gives you flexibility as well as you can scale independently. That's one aspect. This is where you see a lot of distributed storage software pieces coming together in a data center environment. The second one is aggregating the diverse provider solutions deployed in a data center. Whether it is your traditional SAN, NAS appliances, or it could be scale-out appliances, we should be able to manage them irrespective of whether it is old or new in a uniform fashion. So that's a second aspect. The third one is provisioning the resources much more easier way as opposed to tapping into each one of these storage software solutions out there and trying to provision each one of them on its own. The last one is allocating the storage resources. Based on how to meet the application requirements, you should be able to allocate those storage resources much more effectively using SLA and SLO, Service Level Objectives and Service Level Agreements. So those are the four different layers in the SDS framework. Now, let's take a look at the SDS architecture. So as you can see, there is an SDS controller in the middle. It has visibility into all the storage resources deployed in the data center. So visibility meaning what type of storage system it is, what type of TRC it offers, what are the capabilities, what are the quality of service attributes, and so on. So that's the first aspect. The second aspect is it brokers the interaction between the applications and the storage resources. The way it brokers the interaction is applications describe the requirements using Service Level Objectives. Maybe it is performance, maybe it is capacity, maybe it is latency. Controller has intelligence to figure out what is the best way to map those requests in a data center environment. Then the third one is once the controller allocates the storage resources, it gets out of the data path, application and the storage resource communicate directly in a data path. So the controller job is to purely provide cardboard storage resources and make sure there is enough quality of service after the resource is allocated. Beyond that, this job is done. There is other element that you probably might not have seen is the data services. What we see is we see a rich set of data management functions that are available today in the industry, mainly in the form of appliances. For example, high-speed compression, dedu, encryption, gateways, and so on. We really like to have a way to express and describe those data services deployed in the data center, and have a way to allocate them when an application is making a storage request. So for example, if an application is asking for encryption, and you do not have encryption back-end support, and there is a data service that is deployed, you should be able to take those two things together, stitch it together, and allocate the storage resources accordingly. This is very similar to network function virtualization service chaining. Concept is somewhat similar, but it is very much in the beginning. So we see those two pieces part of SDS architecture to really enable address the data growth challenges as well as the cost aspects that are actually driving the storage industry today. Let's look at the OpenStack. So I talked about the more of a problem statement and generic framework and vision. If you were to look at and map this to, how are we going to do this in OpenStack? We see that the SDS controller doing five different functional aspects. One is provisioning your storage system, ensuring that you are able to expand the storage capacity on demand in an automatic way. The other aspect is the discovery, able to discover the storage systems in the data center in an automatic fashion. Composing them into virtual pools, so it's giving a tool set to the data center admin to be able to compose them into a set of virtual pools that can be consumed by the applications. You could be composing them into gold, silver, bronze, or maybe it is a high performance archive and backup tier. It could be any one of those things, but it's a tool set that we really would like to give it to the data center admins, so that they can actually compose them into virtual pools with certain performance, cost, capacity, characteristics. Then of course, once the resources are allocated, maintaining those resources is also a critical part in the data center. The last one is the data management aspect. Essentially looking at the life cycle of your volumes, shares, or containers and objects, and figuring out what's the best way to manage those resources. For example, this is one of the common complaints we get from end users, which is I have 10,000 virtual machines. I have no visibility into which ones are very active, which ones are not active, which ones are inactive, and I should be able to take the inactive virtual machines and move them to maybe a lower cost tier. We should be able to describe them as part of the policy and have that automated completely behind the scenes to help data center administrators. So those are the five different pillars that we see from the SDS controller perspective. Of course, when it comes to actually carving out the blocks, files, and objects, there are the actual layers that do the work behind the scenes. So ideally, what we'd like to see is, within the context of OpenStack, we'd like to see one control plane where all the control plane operations are being streamlined. Without this, it's hard to build the missing functions that we actually talked about in the previous slide. How do we do automatic provisioning? How do you do quality of service monitoring? How do you enforce the quality of service? Things of that nature, it's extremely difficult to implement when we have four different ways of doing things today in the OpenStack. So one of the things that we are working through the community and developers is how do we enable this? Is it taking Cinder and enhancing the Cinder project, or is it something totally new? Irrespective of which way it is, we really would like to enable this feature set in the OpenStack framework. Now let's take a look at specific features that we are talking about. So if you would look at the end-to-end lifecycle, provisioning, discovery, compose, data type lifecycle management, which is creating the volumes, maintaining your volumes, shares, as well as ongoing maintenance of your storage system as well as the data types. The ones that are highlighted in green are the ones that exist today in some shape or fashion in one of the three or four different projects that exist today in OpenStack. Cinder is the one for block, Manila is for file share, Swift is the end-to-end object, and Glance is the image repository. So those green elements exist today, the ones that do not exist today are the yellow ones, and we are looking at ways of building out these gaps, addressing these gaps via existing projects. We are also looking at is there a way to stitch them together from an end-to-end lifecycle management perspective all the way from provisioning to the maintenance, and what's the best way to enable that. We are currently focusing on two different features as part of the prototype effort. These two things came out as part of our discussions with end customers, and they thought these two are the important ones as part of the SDS controller. So we are currently building a prototype with specific focus on OpenStack Swift at Ceph just as a way to prove how we can provision those storage systems. We are using two different storage system tools. One is the virtual storage manager that Tom is going to talk about. The other one is the Swift Stack controller for Swift provisioning. So the goal here is to understand how do we stitch the existing provisioning tools as part of the SDS controller and create a nice framework around it, so that existing tools can be plugged in, and from a data center admin perspective, they can use one controller to control all the storage resources deployed in the data center. So we'll have a source code available by end of this year. A portion of this may actually show up in Cinder. We submitted a few blueprints, one of them we have a design session on Wednesday. It's basically the discovery part, how do you automatically discover the storage systems and configure them. So we're going to focus existing projects and blueprints and continue to evolve, as well as look at certain elements that do not exist today that are far off from the existing projects and figure out the best way of enabling them as part of the SDS controller. So with that, so touched upon what are the storage pain points? The promises, I mean, why do we need software-defined infrastructure and software-defined storage? What we are doing within the open stack, within the context of SDS controller, specific features that we are really looking for in the SDS controller, and we really want to build that for the course of time. I know it's a lengthy journey, and our goal is to embed all these pieces in open stack in some shape or form. With that, let me transition the next topic to Tom Bones. He is our senior architect. He's the architect for the virtual storage manager. He's going to cover stuff and virtual storage manager. Got that? All right. This is how we're moving forward. Ah, yes. Let's talk a little bit about Ceph first here. So Ceph is a very popular storage solution for open stack because it provides three storage solutions, essentially in one storage system. At its core, Ceph is an object store. It's also a distributed storage system which allows you to distribute data over multiple drives and multiple servers. And as a distributed storage system, it's very resilient against failures of individual drives and individual storage because it uses a uniform pseudorandom distribution scheme to distribute data throughout all of the storage assets. Ceph provides an object interface that's S3 and Swift-compatible, and it's capable of connecting to the Keystone and Swift APIs. Ceph also provides a block device interface that allows you to expose the object storage as storage volumes to VMs, and that interfaces with Glance. There's a Ceph driver that's compatible with Cinder, and it interoperates with Nova so that an operator or a subscriber can essentially provision volumes through Cinder and then attach those volumes to VMs that are being set up by Nova. Below all of this functionality, then, is the Ceph Rados Storage Cluster. Rados, of course, standing for a reliable autonomous distributed object store. And using this system, you then end up with a very reliable storage system that's capable of taking all of the aggregate drives that are attached to it and distributing that storage across hundreds and thousands of VMs. Intel's focus with Ceph has been in optimizing Ceph's block storage for OpenStack deployments, delivering open reference architectures for Ceph, and also the work that Reddy is putting underway to integrate with the SDS controller. In addition, we recognized about a year ago that Ceph was a very powerful storage system, and if you've ever worked with Ceph, it's very powerful, it's very flexible, and therefore it's fairly complex to set up in a standard and consistent way, and it's also fairly difficult to understand all of the ways in which to manage its options, especially if you are not a deep expert in the Ceph system. And so what we set out to develop for Ceph was some software that we call Virtual Storage Manager, and it essentially provides a layer of management software on top of Ceph. Okay, so what we set out to do when we set up VSM, our goals were to achieve consistent configuration to support a segregated storage system so that you could have different types of storage within the Ceph cluster, to allow you to have efficient asset management, and to provide a familiar look and feel to data center administrators. And so in terms of consistent figuration, VSM controls the cluster configuration through predefined manifest files. So rather than having to go in and manage the crush map configuration or having to issue a large number of series of command line commands, you can set up your storage system through VSM using standard, or I would be better to say, using fairly simple manifest files to configure the system. You can set those manifest files up on all of your storage assets, and then add those assets to your storage system, and VSM automatically adds them to your cluster. The interface for this system then has a set of predefined operations that the operator can perform, having to do with the management of the cluster, and that avoids the situation where someone is inadvertently removing more storage from the cluster than the cluster has room to replicate for, or add storage that's then not efficiently utilized by the system. In terms of segregated storage support, VSM is set up specifically to allow you to have different types of storage in your cluster. You can have high performance SSD storage. You may choose to put in 10K or 15K RPM drives, or to have a very low cost tier using 7,200 RPM drives. Under VSM, all of that storage can be held under the same CEP cluster, organized, and pools managed within that system. And VSM once again takes care of all of the details of creating and maintaining the underlying crush map that describes that storage organization. And it also provides you then with the ability to monitor capacity utilization in each of the segments of storage that you've defined for the system. VSM provides you with the ability to manage the system, the hardware assets in the cluster over their lifetime, the ability to add and remove storage servers and to add and remove drives as they fail in the system. And finally, VSM provides a familiar look and feel. It follows the OpenStack UI schema and in fact is built on top of Horizon. It has essentially a flat navigation space, uses a very basic tabular data provisioning system in order to provide you with all of the information about the cluster without a lot of drill down to get access to that data, provides a comprehensive summary on a single page and provides error and status reporting. So what we see then is essentially we've set this up with a VSM controller node. Each of the storage servers are set up with a VSM agent that facilitates communications between the controller and the agent. VSM is then capable of taking pools that are created in the Cef cluster and publishing them to the OpenStack Cinder through the Cinder files. And essentially today that involves going in and actually modifying the files for the multi backend feature of Cinder. In the future, one of the improvements that you discussed is the actual ability for either Cinder to discover them or for a controller like VSM to publish them through an API. So that would greatly improve the flexibility of that system to do that. And then we provide a web-based UI, a RESTful API, the ability to manage capacity, to add and remove these servers through the interface, the ability to monitor the cluster. And the key thing that you get from using VSM is the ability to set up this management framework that results in consistent configuration for your cluster. You use VSM to determine what your cluster configuration should look like. You codify that configuration using a set of manifest files and then you can quickly replicate those system configurations either in an appliance model or using it for a widespread distribution of the system. Next slide. So what VSM then tries to do is manage the storage system throughout its lifetime. Now we looked at three different areas, the commissioning phase, the live operating phase and the decommissioning phase. And we recognized right away that there's lots of great tools for installing this type of software. So VSM is not really a Cep installer. It really, most of its operations are involved with live operation. And so that live operation involves understanding and configuring the cluster according to the physical infrastructure that's being operated with, configuring it as a storage cluster, connecting it to the cloud orchestration system, being able to monitor performance and ensuring that the user gets the storage that they want. And so we're able to add and remove capacity, add and remove servers as physical infrastructure to create the cluster and create storage pools and update the cluster configuration, the ability to maintain a storage catalog of different pools and storage groups that you can connect to and to provision capacity from those pools that you do through Cinder to be able to monitor the system and then to be able to allow the subscriber of the cloud system to create and attach VMs to the storage assets. So this is an example of the VSM. This is the front dashboard. Here it's providing you with summary information for the entire cluster in a single screen, providing you with a summary of storage group information, monitors, all of the OSDs, which are the Obsect Storage Devices that represent each of the disks in the system, a summary of MDS status, PG status and overall cluster health. And then for each of these subsystems, you're able to click in and get detailed information for each of those subsystems. Intel is taking this VSM software and as of this week, as of today, in fact, the VSM software will be available as an open source project. It'll be available at 01.org slash virtual storage manager and you can go there to get access to the source code and to documentation. All right, now I would like to introduce Dieter Casper. He's from Fujitsu. He's the CTO of Data Center Infrastructure and Global Emerging Technologies. Yeah, a couple of years, one year ago, I met with Greg Scott in the Silicon Valley and we talked about CEP and how we can bring CEP to an enterprise level and after some minutes was clear that Intel and Fujitsu are on a similar mindset here. It's not about adding the latest bells and whistles to CEP. It's more about to getting the complete stack into a mature state, to harden it and to optimize it and also adding convenient and easy to use management interface to it, the VSM. So we sit together, we helped and contributed to Intel, to design it, Intel developed it. We also took place in the GA in the quality assurance and in October this year, we were able to launch our new product, Eterno CD10K optimized for elastic storage for the cloud and also for OpenStack. If you look at the challenges for on storage subsystem, IDC defined a term called the third platform which is basically built on four pillars, big data analytics, social business, mobile broadband and cloud services and the needs there are, it's obviously scalability but you also depend on reliable data and from a cost perspective, you also want to keep the operational costs down so manageability is a key aspect and with the help of Intel, we solved this part and on the next page, I will describe how the CD10K principles which are based on the safe technologies handle the data. So the red blocks are describing the wide striping when data is put into the storage subsystem so all the data will be distributed across the storage nodes then internally, automatically there will be additional copies created. So for example, the primary data will sit on the first storage node and the second copy will be put on the node number two, three or four and over the time, the nodes will be filled up with the data and for example, if a disk breaks and the primary copy is lost, then based on the secondary copy, we will be able to create a new primary. Copy of the data. And by the way, when I look at this example as a mistake, so usually there should be the, if you see here the primary copy in red, then the pink copy will be not on the same node but on different nodes, we have to fix it. So sorry for this animation. So over time, all the nodes will fill up and if you want to scale, you will add new nodes to the system and automatically the existing data will be rebalanced to the new nodes. It will be the primary copies as well as the secondary copies and then probably over time after three or four years when you are running out of maintenance for your storage servers, you want to get rid of them, you would like to use the new ones and again the same logic of automatic rebalancing will move the data from the old storage node to the new ones. So here's the question, what is the product eternal CD10K? So it's based on commodity hardware and commodity X86 servers. We have basically two different types of building blocks. We have performance nodes which will use two and a half inch disk drives with 15K and high capacity nodes, three and a half inch drives with a G-board capable to host up to 60 of them in one building block. All these nodes are cross-connected through a powerful back-end interconnect based on infinite bands. And then on top of those nodes, we are running the CEPH storage software which will offer to the front-end interfaces the storage through a 10 gigabit ethernet front-end interface and based on top of the CEPH software we add additional software from Fushitsu central management software as well as enhancements in CEPH for example, a specific Fushitsu erasure coding there and then as Tom described already based on these neutral rados level. There are the three different front-end interfaces in CEPH which is block, object and file access and all these different interfaces can be also accessed through the open stack interface, Cinder, Swift and upcoming Manila. And if you remember the keynote from Jonathan, today he was talking about choice and this is a perfect example about choice. You have a neutral storage subsystem and later on you can make up your mind whether you want to access this storage through block, through object or through file. So you don't have to make up your decision first. It's really highly flexible and it's bound to your demands and additionally you will also put integrated applications into our system which will serve the complete cluster with backup, archiving or sync and share Dropbox style access. That's the summary of the enhancements. Number one, the central deployment of the software. So we will take care if new versions will come up that we will manage the rolling upgrade through all the storage nodes. We will use the setup of the central network, the log files, the cluster management. We also introduce an SNMP agent into our system. We are using the GUI from Intel for an easy deployment and configuration. As I said, there is an Fujitsu, a highly efficient erasure code from Fujitsu which will plug in into the SEPH software. It's on the roadmap but still we will be and keep 100% compatibility on the SEPH code and also besides the pure storage aspect there will be additional plug-ins for backup, archiving and other applications. So here's, we learned already from Tom about the capabilities of the virtual storage manager and you see the Fujitsu branded variant of VSM. And yeah, on the next page I describe some use cases. So for example, if you want to run on high availability scenario or disaster scenario you have two different locations, maybe just in a different room or a different building or maybe even in a metro distance between the two sides. There will be a couple of storage nodes on the left-hand side, on the right-hand side. And these storage nodes will provide different types of storage, SSD, SAS storage and SATA for example. And on the next page, you see some example setup. For example, a gold pool. You want to keep two copies on the left-hand side, a third copy on the right-hand side, on the silver pool, for example, SAS drives. You want to have four copies, two on the left, two on the right-hand side and the SATA drives again with three copies. So basically, you can combine in a matrix the storage class with the replication level and you can freely design your own storage pools, give them names and all through the VSM interface you can also use the erasure coding and then you are completely free, therefore this will be offered to the upper-layer hosts or clients which are consuming the storage whether it's file, block or object. So basically, probably you will ask where's the benefit of this appliance solution? If you, for example, go back 20 years in the past when Linux come up, some people took the source code, took the kernel from kernel.org, create their own distributions, but over the time today, when you look in the data center, I would say more than 95, maybe 98% are not running Linux from the source. They are using enterprise distributions because they rely on this layer and they do not want to waste time to deal with this lower-level infrastructure layer. They want to concentrate on their own values which they are putting on top of Linux and here it's the same story with the software defined storage. Of course, you can do it yourself, but you really have to invest lots of time to get the right hardware, to optimize it, get the right interconnect, the right management software and here from Fujitsu, you will get everything out of the box, you don't have to care about this layer and you really can concentrate on your values which you can put on top of this product. So let me introduce the next speaker, Tom. He's a senior architect in the software development of Intel. Great, thanks, Peter. It's actually Paul, but that's pretty close. We're all jet lagged, so. So you want to switch gears a little bit. I'm one of the core developers in the Swift community and at Intel we've been working closely with the rest of the community on really two big, big major things over the last year that I wanted to touch on. So if anybody was in Atlanta for the last summit, hopefully you got a chance to hear a little bit about storage policies. John Dickinson and I gave a talk there that's still online if anybody wants to go see it. Anybody get a chance to catch that? Because we had a really cool catchy title. The biggest thing to happen to Swift since it was open sourced. So this really was a big deal. This was almost nine months into making and this was absolutely not Intel writing a bunch of code and dumping it in the community. This was a combination of Swift stack and Red Hat and Intel and a bunch others all working together, multiple face-to-face meetings, lots of collaboration, lots of development code. And essentially, I certainly don't have time to go through what is Swift, so I apologize for no Swift 101 here, but storage policies is much easier to understand if you have a handle on Swift. The easiest way to think about it is prior to storage policies, if you wanted to do multiple things in your Swift cluster like have some of it be triple replicated, some of it be double replicated, some of it be erasure-coded, you couldn't do stuff like that. It was kind of a one-size-fits-all for Swift. Storage policies was this enormous effort to add really an extensibility framework so that we could do other things with Swift, optimizing around performance or durability or even tagging by geography, all sorts of cool stuff, just a gigantic feature. And really the building block for erasure code support, which I gave a talk on a couple hours ago with Kevin Greenin from Box. Anybody get a chance to see that? All right, so, hey, fantastic. So I don't have to talk a lot about that for those of you that saw it, but it's really the follow-on to storage policies. It's kind of really why we did storage policies. There's a lot of benefit to policies, but we started talking about the erasure code work well over a year ago and identified what we now call the storage policies as sort of a prerequisite framework for doing erasure code correctly. So it's a very generic framework that we put in place for policies and erasure code is implemented as a storage policy. And again, a really big community effort, the talk earlier today, if you didn't get a chance to see it, we tried to spend a lot of time talking about erasure codes themselves. And we didn't actually build EC into Swift. We built an external library that's a dependency for Swift. So that external library allows us flexibility of choice in actually which algorithms that we use for Swift might also give us the ability to actually put the math in C instead of Python, which has some obviously performance implications. But we use the Intel optimized EC library as one option. Kevin Greenin who gave the talk with me is one of the co-authors of J erasure. So some of you guys may have heard of that. That's one of our options. We've all got an option from NTNT. So a lot of good work going on there. I encourage you to go check out that talk in those slides if you haven't seen it. Next slide real quick ready. I'm gonna try and get this wrapped up. Hopefully some of you have heard of Cosbench. We just wanted to throw this in to get a full picture of sort of Intel's activities in OpenStack. This is our distributed object storage benchmarking system that's not specific to any particular system, but allows us to really measure an object storage interface regardless of whether back end of Swift or SAF or whatever. And this is open source as well. Very popular in the community. Lots of contributors. It's doing great. So moving on to the wrap up here. This is a build. You can go ahead and build this out ready. So you may be wondering you've seen a couple of sort of disparate topics that we've gone through here with as those of us that have come up here and talked about our areas of expertise. This kind of really does all wrap up and hopefully you can see it under this umbrella of reshaping storage. No one of these things that we've talked about is going to reshape storage. It's just not possible. It's a long activity and everything that we've talked about here from what Dieter talked about to the SDS prototype that Reddy talked about to the VSM work, to the Swift work. They all support this notion of moving things forward in the data center with storage and trying to get to that more software defined infrastructure and control plane. Okay, I think we're out of time. I think we're out of time a minute ago. So thanks. If anyone would like to know more about the VSM open source project, I have some cards up here that will guide you to the site.