 Welcome to our talk our session on Copperhead Before I forget make sure that you have your raffle ticket because we're giving away one of these guys at the end and They're pretty awesome. They're pretty good. They're big big robots So the name of our session as you know orchestrate all the storage things An open stack they showing the open stack data availability using Copperhead today we have an and Palagudu Paladugo and Ready and they're gonna Lead us through it Ready is from Intel Corporation and Anand is from EMC Corporation Copperhead as you know, it's an open source Organization and a product and we're taking people from everywhere and with that I'll give you ready All right, can you hear me? Is it on? cool Okay, so I'm going to spend first Probably ten minutes going over Giving you a giving you a little bit of a background on what exactly is the Copperhead architecture and We will drill deeper into a specific use case that Anand is going to go over And then we'll talk about the community engagement So we are really looking forward to you guys actually getting a glimpse of what exactly is Copperhead How can you play a role in influencing the community? technical direction as well as the contribution aspect Okay, so before I jump into the Copperhead architecture So let's go through the software-defined storage architecture. What do we mean by software-defined storage? In a nutshell software-defined storage, you know brings in the cloud Architecture benefits for the storage side of the world What do we mean by that? You know, how do you are completely automate the provisioning aspects of the storage resources sitting in a data center? How do you provide self-servicability? and a single pane of management The key element of software-defined storage architecture is the SDS controller so if you were to look at the The green box that SDS controller it has two major aspects one is SDS controller has visibility into the storage resources deployed in a data center So what do I mean by storage resources? These are the resources from traditional appliances We call them as scale-up things like sand, nas all-flash arrays Or it can be a scale-out architecture Based on the open-source software stack deployed on standard high-volume servers or it can be a proprietary software like scale I vor Deployed on standard high-volume servers Irrespective of the type of the storage backend that is deployed in a data center controller needs to have a visibility into those storage resources And provide the services to applications via some sort of an orchestration that it is running under Typically what you look for is the applications make storage requests not by specific array, but rather requirements The requirements could be we call them as the service level agreement Agreements as well as service level objectives things like how much capacity I need what kind of performance I'm looking for What is the latency? How should I manage these volumes or the course of its lifetime? Maybe I need to back up periodically and so on so applications provide those requirements controller having the visibility of storage resources We'll figure out the best way to optimally allocate the storage Request coming in from applications once that is done It completely gets out of the data the data path Between the application and the storage resource so your data path typically happens to be you have a volume You may use ice cozy fiber channel or ethernet NFS Or some sort of application specific proprietary protocols Irrespective of that that is a data path controller does not involve in the data path only the control plane operations So why do we need? open source SDS controller as you guys know storage management is a complex is a Pretty big challenge. It's a challenge that the whole industry is facing with What do I mean by storage? Automation challenges things like completely automating the provisioning of the storage resources How do you place the storage resources optimally? How do you protect the volumes across multiple data centers? Migrating your volumes the whole rolling upgrade HA all those pieces need to be included in the SDS controller So the question is how do we do this in open way? And we want to solve them but we want to make we want to solve them in an open way and We need to be able to support multiple orchestration frameworks as well things like the traditional Orchestration stacks like Microsoft and VMware Cloud stacks like open stack and then cloud native computing stacks like Docker Kubernetes Mesos and so on The controller needs to be able to work across wide variety of orchestration stacks While managing both scale up and scale out and addressing the storage management challenges so The what is copperhead? Copperhead is the open source SDS controller it provides a Foundation capability to manage storage resources deployed in a data center a Discovery capability Logically pooling those resources based on certain requirements that the data center admin desires Maybe you have a capacity pool. Maybe you have a performance pool. You need to be able to group those resources into certain pools based on what services that you want to stand up and then completely automating automating the management aspects of Storage resources deployed in a data center. So let's look at from the bottoms up so Key foundation element in copperhead SDS controller is how do you discover? Wide variety of storage systems deployed in a data center Traditional as well as the scale out Storage backends we need to be able to discover them. That's a fundamental aspect of discovery You also need to be able to discover fiber channel Networks the whole configuration aspects and the topology and so on the connectivity aspects and then the host configuration So where exactly you are going to consume the storage resources? So typically the discovery includes all those three elements your storage resource discovery Your networking aspects and your compute where the storage resources are getting consumed Once you discover the storage systems, you want to be able to classify them classification can be fairly advanced in within the context of copperhead you're going to see the You know use case where the classification can be based on Certain types of policies you want thin provisioning or thick provisioning. Do you want? Continuous volumes and so on so all kinds of filters you can use to be able to figure out This is what I want to stand up for a given pool. Here are the types of storage backends. I'm looking for So it can actually filter it it can give you the storage resources that meet your SLO Policy criteria and you can actually cherry pick those storage backends and pull them together And then the third element is the end-to-end storage automation The foundation element is how do you? intelligently select resources deployed in a data center and how do you place and Service the resource requests coming in from applications in an optimal way You need to be able to support local and remote productions somewhat advanced capabilities Sand zoning host attachment host configuration and maybe load balancing on the host side sand fiber channel ports as well Migration aspect how do you migrate the data from one storage back into the other storage back end? How do you migrate one release to the next release in the copperhead SDS controller? How do you do the rolling upgrades? How do you retire a specific storage back end? How do you add a new storage back end all those things need to be completely automated? Copperhead provides that kind of a end-to-end storage management The these features need to be exposed via open API's rest API's There are two ways to consume The storage management services one is you can use the rest API's command line interface or you can actually use the service catalogs as a way to deploy shrink-wrapped packages, so things like you want to deploy let's say SAP you know Application with certain back-end configuration completely in an a complete automated way you have a service catalog for that maybe you have a service catalog for Oracle database application that deploys 20 different VMs maybe with remote replication intact all those things so you can actually do package the basic foundational features into a set of catalogs and you can expose them to end customers last but not least irrespective of you know The storage management plumbing it needs to work across all orchestration stacks not just one of them, right? So that's a key thing. We are marching towards things like it has to work with VMware vCenter Microsoft open stack and then the You know emerging cloud computing frameworks as well things like Docker Kubernetes and so on so that's kind of what we think The open source SDS controller should be addressing managed wide variety of back-end work across different orchestration stacks intelligently manage the storage resources Make sure you can actually do the data migration the life cycle management of your storage resources as well as the SDS controller need to be You know comprehended as well in the design So these are all the foundation features that exist today And you know, but we are also looking at what's the best way to integrate into the orchestration frameworks We have open stack integration. We have VMware integration. We are looking at what's the best way to do that with Docker The other emerging cloud-nating computing frameworks Okay, so with that hopefully you got a glimpse of what do we mean by what do we mean by SDS controller? What exactly does copperhead provide today? We'll go through one of the use cases that Anand is going to cover After that, we will look at the community road map what we are currently working on just to give you a flavor of what is Happening right now so that you can actually participate and help us. Okay with that Anand Thanks Hi, everyone. So as Hura has introduced me, my name is Anand. I'm a product manager at EMC My part of the presentation is actually to show you how Copperhead orchestrates use cases so we can provide highly available storage in an open stack environment, right? We'll do this in four parts. I think we'll first define the problem scenarios One for an app admin and one for a storage admin review the EMC platform technology that actually provides a chess storage And the third part is you know understand how actually Viper as copperhead orchestrates use cases along with sender To provide highly available storage as well as data migration Another fourth part is actually go through the demo so you can clearly understand, you know, how this is all done I'm in the right slide Okay, so in defining the problem scenarios, right, you know application It means always want to define SLAs for their applications Particularly for mission critical applications They want the data loss to be zero right in the event of a disaster and they want the downtime to be zero as well Right, so it's actually defining RPO and RTO as zero It's easy to achieve a zero RPO if you have a synchronous replication between two sites But it's not that easy to achieve zero RTO right downtime should be zero. So that's the requirement From a storage admin persona perspective the problem scenario is they do want to migrate workloads to a different data center So to balance infrastructure resources that happens every now and then right so then they want to do it in a non-disruptive way, which is a key challenge So those are the two problem scenarios that we were talking about today and then Vplex is a Storage platform from EMC that provides the HA technology that will actually let us orchestrate use cases to provide a HA in an open-stack environment So as you see Vplex provides high availability and resiliency by allowing users to Create mirrors for their storage volumes the mirrors can be created in a single site or across sites And after mirrors volumes are created Vplex will allow you to access data from any site You know these sites are at a synchronous distance So site A and site B you could access data from site A or site B, right? That's what the mirror does Vplex also allows you to migrate data from one array to the other a whether it's in the local data center or in the remote data center And then finally with the recover point It also has the ability to give you a local protection or a remote DR protection So when it's actually protected in the remote site Site C we call that solution Vplex Metro point So there are three ways to deploy Vplex a single site HA If you deploy it across two sites, that's Vplex Metro cluster And then if you want a DR, that's the Vplex Metro point solution, right? So that's Vplex So copperhead working with Cinder in an open-stack environment orchestrates three use cases for Vplex Provisions highly available storage for EMC and non-EMC arrays It actually can orchestrate data migration use cases for Vplex We'll actually see how it all done Also, it has the ability to do change the normal storage to a highly available storage, right? You may have provision storage for an application and then later on it became critical and you wanted to provide HA option for the application Copperhead can actually can convert that to highly available storage So I'll give you a high level overview how things flow in this slide, but we actually have a demo to show everything so once you have registered the third-party array driver Cinder driver and the copperhead Cinder driver the workflow is essentially like this, right? The end user goes to horizon UI. He requests a volume to be created Open-stack Cinder gets the request and it's passed on to SDS controller copperhead and Copperhead using the Cinder-Southbound interface creates provisioning on the non-EMC arrays and That storage that it provisioned actually is now mapped to Vplex right through sand so the proper storage that Copperhead provision here is now available to Vplex as a backend storage, right? And after that copperhead uses the native integration that it has with Vplex to create a virtual volume which actually represents Two volume mirrors, right? And then this volume is given back to Open-stack Cinder attaches that volume to the instance that you have created. So that is how the whole workflow goes So I'll actually get into the demo next So I have to get off the deck So this is actually a live demo, but we recorded it for convenience sake so I may have to Pause it sometimes to talk about the stuff actually this a little faster than Okay, so this is the copperhead Dashboard when you log into copperhead, this is a web-based UI You see all the Infrastructure that is available to you in copperhead the physical assets the logical assets that you have created the state of the Wipe copperhead instance whether it's stable and then what capacity you have for free right at the total capacity And then how much capacity have used etc So before you orchestrate use case and any storage array the storage array has to be added to copperhead right So if you go into storage systems You'll see that we have a bunch of non-EMC arrays here the three-part array the Fujitsu Eternus array and then the EMC VNX array. So all the three arrays have been added to copperhead And as Reddy was talking about the fundamental value of copperhead is that it abstracts hardware and it creates logical pools of storage So if you were to go into virtual arrays You would see that we have created a virtual array called open stack This virtual array represents the Vplex array and all the back-end storage arrays and all the fiber channel pores that are required to provision storage Right, so that's the abstraction here User is not aware of what is below this layer. I think my mouse has moved a little bit. Sorry And after you have created the virtual array we can go ahead and create a virtual pool Virtual pool the notion of virtual pool in copperhead is that you know, you can actually define Storage on a policy basis, right? For example, in this case, we're creating a virtual pool which is for highly available storage So we can define what hardware we want here It's a third-party block and highly available Storage we actually pick the Vplex local option and you can also define other policies like you know, how many Snapshots you want etc in terms of data production So this is where we select the storage pool, right? We actually picked up Eternus storage pool for this Pool So the screen just shows you that the pool is created and actually here what's happening is now We have gone switched back to horizon UI will create a tiny instance called copperhead demo So the instance should be ready And now we want to provision a volume for this instance, right? So since copperhead sender driver has already been registered The volume that we're going to create here is coming from copperhead. So let's call this volume data volume And the pool you're going to choose is the pool that we just created, right? size 10 GB Create the volume So the volume is getting created So if you switch back to copperhead Events you'll see that I don't know if it's actually clear You will see that the create volume operation is underway, right? Copperhead has received the request and it's creating the volume on the 3 power array so it creates the volume it adds the volume to the masking view so that the Vplex can see it and And it creates the Vplex which will volume as well So all of this in the back end is being orchestrated by copperhead So we're going to attach that volume to the instance that we just created Tiny so the volume is attached And the volume is actually dev PDB So we can actually now go to the instance console And the create a file system on the volume that we just attached there is no way I can speed up so So creating the file system And there's a reason why we are doing this right actually we wanted to show how the failover are you know disaster recovery condition works So it becomes seamless to you then so create a directory Mount the file system and then create a file Okay, so that is the initial provisioning right what we have done so far is We have an open stack environment with copperhead and then third-party erase EMC and third-party erase User has requested a volume to be created from open stack horizon Copperhead has actually provisioned the volume and that's the Vplex which will volume right the two other use cases that will Actually go through is you know one is where you just created the volume on three part Right you want that volume to be migrated to the fugitive fugitive return as array How does that work right? How does our copperhead do that? That's one use case The other use case is you know the volume that we created will actually make it a highly available volume Meaning create mirror on the other data center Take off one array and then see what impact does it have on the application right so that's the highly available part So we'll go to the first one. Oh, sorry. I didn't click So the initial use case of migration the workflow starts in the horizon UI you pick the volume and then say change volume type So that actually triggers a data migration workflow in copperhead Copperhead communicates with Vplex and seamlessly transfers data from one array to the other so as you see here retyping is happening And then if you were to go back to copperhead console You would see that copperhead actually creates a volume on the other array where you wanted the data to be migrated to and Then the orchestrate the migration workflow through Vplex And then all this is happening All of this is happening when application is still alive, right customer doesn't even notice anything So everything is non disruptive. So there's not enough data. So the migration actually happened quickly, right? So now as you see the volume is a turn us and if you go to the host and then say my file My file is still there, right? So you simply move data from one array to the other The last use case is you know Convert the storage to highly available storage take one leg off and then see what impact does it have on the application? So that's what actually gives you zero RTO even if one array is down. You wouldn't even notice a thing So the way you create highly available storage is you know go to copperhead UI in the service catalog There is a feature called create continuous copy So this create continuous copy function actually creates a mirror For the production volumes on the remote data center. So we pick the same volume data volume that we created We call it data volume mirror and then order for mirror So mirror is getting created So again, I think you know Vplex has to make sure that the Volumes are created on the other end and then the synchronization happens between the two copies All of that is done. And then if you click on the data volume And then if you actually see the continuous copies, they are created Right data volume error zero is created So the way we can actually map these two back ends is to go to the Vplex UI So let's note down the device label for this device Let's go to the Vplex UI And if you look at the inventory for that particular device label You would see that it has two legs, right? The virtual volume has two legs One is on VNX and one is on the Eternus shown by two different paths Now what we'll do is we'll go to the mapping and then take a learn out of the Consistency group for VNX. So that means we're actually breaking one leg down and then still see if customers can access data from the other leg So volume is being taken off from the consistency group. So when you do that what happens is the Mirror is no longer in a healthy state, right? The UI actually will show you that So as you see the mirror is in this trusted state but if this has no impact on customer data because Data is being no access from the other mirror, right? So one leg is down The other leg is still active and if you go back to the application and then Find the file file is still there That's what gives you zero downtime one leg is completely down one array is completely down, but you still are able to continue the business I think that ends the demo All right, thank you. Hello again So I'm here to talk to you a little bit more about the community side of the project and If you forget everything I say everything we've seen today But you want to remember There's a community site right there Copperhead HD at github.i.github.io And if you don't want to remember that either you can just go copperheadhd.org Org and that will take you basically to that page from there. You can then jump over to our wiki and There are links to Stash which is where we have the source code. We also have an image of copperhead in GitHub so if you want to if you prefer to use github and just download everything from there, that's fine We always keep our stash repository in sync with the github But we are using stash for like if you want to do a submission of code you would do that on the on the stash server For issue tracking we're using JIRA for continuous integration. We're using Jenkins. So pretty standard In everything in everything in terms of the tools For our communications Instead of IRC we're using hipchat it's it's nice it works and There were issues with running IRC at the time between all the different companies etc. So We decided to go with hipchat and then for Forums and that's sort of conversation. That's a longer term sort of conversation questions or discussions, then we're using a Google group Which has served us fine so far. So, you know, we're using that So But again, if you forget all of these pages copperhead.org On the outside, I've got some stickers and some cards which can remind you that So feel free to take as many as you want Just leave some for the guy next to you So things that are going on right now. Let me just get down here So we're talking to different companies storage vendors open stack integrators northbound orchestrators and You know, we're trying to figure out who who wants to collaborate who wants to play Right now EMC and Intel are kind of already in all in and and doing development and We're in we're at the stage where we're like I said looking for other contributors Which is why we want you to join us Things that we're working on so today As most of you know, there's a product from EMC called EMC Viper Copperhead is the same as EMC Viper, but it's the open source version So EMC Viper comes out in Susie. It's built on top of Susie Copperhead is built on top of open Susie and we are in the process of creating new distributions Platforms so that it works on Ubuntu that we've got some people working on Docker There's a guy who actually already built a vagrant file which works really well So that's one of the things we're working on one of the projects. We're looking for support and more people We're working on you always improving the copperhead on per stack integration We're always looking for more southbound interfaces more Being able to support more arrays from more vendors Okay, this is stuck okay, we have This right here the copperhead command is One of the little projects that was built on top of it, which basically what it does is it gives you Sort of a CLI interface. That's more of more like a like just using a regular shell tomorrow, we're gonna be we're gonna have a Both session verse of a feather session at 9 a.m. In Sakura S1 and S2 and then in January We are going to be doing a the second developer meetup It's gonna be held In the Oregon State University campus in Corvallis, Oregon. You are obviously all invited It would be awesome if somebody shows up there from this thing come check look look me up Send me a message on the on the copperhead forums And with that, you know, we thank you very much There is a raffle to do next so before I forget as any does anybody Not have a ticket Raise your hand Before that, yeah before we raffle the thing. Are there any questions only the good questions will get that Oh, yes, good questions get you an extra ticket. I Would hope so I don't know See I can say these things and then she's gonna be you know if there's any questions We have yep right here. We have a microphone coming to you So as you mentioned about we planks with Corporateity and you said that business continuity will go Right. So what about the right operations because as far as the operation is concerned you shown the demo But what if the right opportunities are going on Actually, as I said We flex mirror can be accessed from any site site a or site B So in this case, you know, I probably can assume that your host is inside a But the two legs are inside a inside B, right? You're actually doing reads and writes from site a So reads are served from sorry site a but writes out to be acknowledged by both areas It's a synchronous distance. Obviously it has to be less than round trip has to be less than five milliseconds Yeah any other questions and And then using copperhead does not add or in any way affect Those numbers because we are not in the data path at all. Well, this was either the most awesome presentation ever or Just not really good at all because there are no questions. Hope it's the first one. Thank you very much And so for the for the raffle She has the bag. It's a secret bag. Who is the very high technology right here? It's an open source bag There is a number on this ticket That says Two seven three five three three zero We should have a winner back there. I see a hand come on over I am very jealous, but I was not allowed to be entered Thank you, everybody. So we have another session starting at 440 with Randy bias a meat tank Lachlan Everson and Josh Bernstein. So it's a panel about open stack futures and Web scaling for versus containers. So if you'd like to stay, please feel free We are not doing a giveaway for that session though