 All right, hello everyone and congratulations for making it to the end of the universe Took me a really long time to get here. So I appreciate you guys doing that as well We're gonna talk about we're gonna talk about Swift storage today, and I have I have my name is Varun Chabrah I'm a product marketer at EMC and I work on an object storage platform that that EMC has Also helping me today is John children John say hi John's gonna be helping me with some of the demos I know it's just before lunch, and I'm sure you guys are all hungry So I'm gonna try my best not to bore you with too many slides. We're gonna be have three demos for you guys It's gonna be pretty quick, and we want to make it interactive. So please stop me if you have any questions all right, so What are we what you know? What are we hearing from customers as it pertains to storage? I think that the biggest thing that I hear when I'm talking to customers is that The the notion of you know whether it's platform three apps cloud native apps buy model it pick your you know pick your buzzword What it's fundamentally doing to infrastructure? Is that the requirements? We have of you know storage infrastructure and other kinds of infrastructure is very different These applications have different requirements than what traditional platform to applications have and one of the things that That's happening a lot with storage Especially with applications like cloud native applications, you know mobile applications data coming in from sensors You know large amounts of data coming from social media or from from your line of businesses a lot of this data is unstructured And you you've all heard that unstructured data is growing much faster than then structured data By the way, does everybody here know what unstructured data is? Okay, good So, you know, I've seen numbers like unstructured data is growing to be 80% of the total amount of data All of the growth and overall amount of data is happening through unstructured data and what this is doing is it's really stressing out Resources that that you know you have in your in your data center Right, so one is capacity. The other thing is how these applications actually work, right? They have very very different requirements You know, they depend more on on app resiliency. There isn't as much of a focus on things like infrastructure resiliency like You know high availability and disaster recovery those notions are really changing in this new world So what what all this means is we need different kind of storage platform to address the growth of unstructured data to To really help our application You know a lot of the application developers that among this audience that are looking to develop these these cloud native applications You really need a different infrastructure and you know the question then becomes what would this you know What what really are the tenants of such a storage platform, right? So the the The few things that we find that we hear from customers again and again It has to be hardware agnostic, right? So it has to be software defined I have to be able to run on my existing hardware if I if that's what I want to do And we certainly see a lot of customers that are asking for that it has to be able to scale at a really large large large exabyte level right even more and We're finding that there's just keeping up with the storage growth and doing it at an economical cost is is hard Right, so you have to be able to scale and it doesn't have to break your bank It has to be multilingual, you know what that means is it has to speak English French and Spanish Oh wait, no never mind. Sorry wrong session. It has to be able to speak different protocols. So API is like You know everybody supports Swift is especially in this conference, right? But you need to be able to support S3 You know just all the rest API's that are out there that customers are looking for the developers are looking for because you know If you're developing an application for Amazon and you're using S3 We often find that over time storage costs for you know, Amazon is great It's very very easy to get you started you swipe your credit card and start putting your data in there awesome You know you become a hero for you know your division Got things going really fast What happens over time is you know more and more data goes to Amazon more and more data goes to Amazon and Every time somebody in your you know, whether it's your end customer or whether it's your internal business units They access that data. There's you know egress costs, right? And then if you're running mission critical workloads, which which all of you are in the cloud You start having to pay for network leasing and you have to worry about bandwidth and all those things and the cost really add up So a lot of customers they come back to us and they say well, I've architected this application for S3 Can I just drop something in for? you know for From on-premises the same way you may be using a service provider that that's using open stacks So you're writing the Swift so you have to be able to drop in those things so the the the platform has to be multi-lingual It has to be able to handle all these different APIs So that you can you know you kind of decouple yourself from the hardware and you can focus on as an as a developer You can focus on writing an application that drives your business forward 3x replication so a lot of vendors that you talked to will say well you need 3x replication You know the data has to be secure. I have to replicate it across all these things Which is great, you know replication is important But there's a lot of storage and efficiency that comes in with replicating three copies of your data, right? Especially as we start talking about you know petabytes of data that your business is generating on it on you know A monthly or an yearly business your storage and efficiency starts adding up pretty fast so we believe there's a better way to actually manage this right and we'll talk a little bit about this but You know there's we do a lot of erasure coding Which is basically a fancy way of saying XOR you know you're XORing different bits together And you're using that and then and we believe that that gives us a much higher storage efficiency While giving you the same level of resiliency that that's something like replication would do Finally or at least on this side of the column. It has to be multi-tenant right everybody knows what multi-tenancy is I'm not going to waste time on that one It has to be VAN efficient what that means is you know You're going to be pulling data off the internet all the time I was talking to somebody at the at the EMC booth yesterday, and they were talking about You know they have so much data that that they are basically ingesting And they're basically sending out that that their network throughput and the cost are becoming an issue even in their own data center So it has to be able to do something that's you know makes it easy. It's VAN efficient You don't have to you know chunk things together and send everything over Active active architectures are you know a big deal and this goes hand-in-hand with sort of the my point about replication The the old world of you know having a primary and a backup You know it's just not efficient right what what customers are looking for more developers are looking for it should just work You know I'm going to write to you know if you have a if you have a mobile app And you have customers all over the world or in different geographies and you have a data center Let's say you have a data center in Japan, and you have a data center in the US You know the the the mobile app user in Japan has to be able to write to You know the the Japanese data center for the best experience, right? And there shouldn't be this notion of what's my primary? What's my backup? How do I fail over etc etc so so all those things are very important as well Related to that it has to be geo accessible right so Take Facebook, you know I like a photo on Facebook today, and then I have friends in India who you know Updater photo on or I put a post to comment my friends in India should be able to see that comment right away, right? But what's actually happening behind is that there's a you know Facebook has a data center here They have a data center ostensibly somewhere in Asia maybe in India and There's a lot of work that needs to be done to get the data to be synchronized right so so Having that that being able to not only being able to access this data across geographies But also having it be strongly consistent right so there's this notion of consistency A lot of vendors that you will hear from will talk about eventual consistency. I'll update I'll update something in this data center, and it will eventually you know 10 15 minutes later it'll get updated maybe even longer across you know the other side and That's we don't think that's good enough we think that we you know as you As if you if someone if one of our users writes data in one area It has to be instantly you know laws of physics being obeyed it has to be updated Right away on on the other side of the world because that's really what your customers are asking for or your end users are there and then the last thing is this notion of Large and and small files, you know back when EMC started sort of the object market with centera Did does anybody remember centera right? You know it's it's a 12 year old platform at this point at this point It's still going strong You know a lot of customers are still using it a lot But the notion of what object storage was and what the needs of object storage were at that point of time are very different from what they Are today back then it was about large files, right? You have really really large files That that the system needs to be able to handle today That's not the the majority of the data you're getting the majority of the data that you're getting is unstructured It's very very small right? It's like 10 to 40 kilobytes. Maybe even smaller and it's it's large Not in size, but in quantity right so you're getting a lot of that So the way you write the small files pose a very unique challenge for a lot of object vendors because a lot of them are architected back when large files were they were optimized for large files when you start doing things like erasure coding and you break things up there's overhead added and To be able to do that that efficiently for small files is difficult it imposes You know a lot of vendors will basically when they're writing small files They're doing erasure coding or checksums and things like that and the right operation ends up taking a lot of time Right, so so the platform has to be able to handle both large and small files, right? So efficiently it should provide the same experience and all of those things okay, so You know in a nutshell What we are seeing is a platform like this would be you know right on the top It's able to support a bunch of different workloads whether it's more traditional workloads like archiving to the cloud right basic basic archiving Backing backing up tiering or whatever Being able to use you know file workloads right files a lot of applications are written using NFS or SIFs And you know the the traditional NAS platforms of yesterday. They aren't able to scale to handle You know geo distributed Requirements or being able to do eventual consistency global namespace all of these things are hard for For traditional NAS systems to do so being able to handle file Platforms is very important as our file APIs is very important as well You have mobile apps you have packaged apps coming in and then analytics right another buzzword that gets thrown around a lot And and certainly people you know in my profession and marketing are guilty of that But the reality is that a lot of customers are looking to Mine all of this unstructured data It's it's you know it there's some there's some usefulness in Retaining this data for compliance purposes of course, but the real value the real competitive advantage is in mining this data Right, so how do you do that? How do you do that when all of this data is across the world? It's coming in really fast. It's unstructured. So so you have to be able to handle that as well so You know all of that means in the middle layer that you see here is that different APIs These are all different APIs, you know, it has to be able to handle HDFS natively You know protocols like NFS without the need of a gateway, right? Because a lot of vendors will will have object storage and then they'll ask you to slap an NFS gateway in front of it Right then increases your cost You have to be able to handle Swift right and you have to be able to handle S3. So so for us That's really what's been driving What we're doing with our object storage platform. It's called ECS. It's elastic cloud storage And you know what I'm going to do now is I'm going to stop talking Stop talking as much I guess we're going to talk about demos, right? So we want to show you three demos that really go through some of these capabilities, right? They're very straightforward demos, but we want to we want to make sure that you're not just seeing marketing slides So let's start, you know first is open stack integration, right? Everybody here is talking about Swift integration. I think that's great, but Swift integration is not enough You know to be a truly drop-in replacement for Swift The platform has to be integrated with Keystone as well, right? Because a lot of the applications that you're writing on open stack They they have reliance on Keystone right for identity models and and for all those things So you have to be able to pull that in and and and a lot of those applications If if the underlying system doesn't support Keystone, it's really difficult to migrate your apps to that platform So we not only do we do, you know compliance with Swift, of course We're also we're all we've also baked in Keystone integration. So let's show you what that looks like John do you want to walk us through? What we're going to see here? Okay. Yes, this is the integration between Keystone and ECS so here we're going to log into the ECS UI and We'll show you what steps you need to take in order to make Keystone and ECS work together So here we've defined a new authentication provider for Keystone and so in there we have to configure some of the data like the Keystone server URL to the endpoint that we'll use to validate tokens that are received by ECS and those tokens that they're generated by Keystone and another thing that ECS needs to know about is the Swift account or the Keystone project So we'll go into the horizon UI and in here we'll navigate to the project section and we'll look for the demo project and to do the mapping to ECS we'll use the project ID for the demo project itself and with that ID we'll go back into ECS and We'll create a corresponding namespace with that same value So this is what allows the mapping from a Swift account to an ECS So ECS understands it in terms of a namespace. So here we have it the demo project is created inside of ECS just to validate that and Now we can go back. I guess we're doing our validation. It's there. It's the ID and Go back into our horizon UI And we're just seeing that the user that's logged in is the admin user You can see in the upper upper right corner and they're in the demo project So actions from the horizon UI that will access the object store later We'll go through this but in order for horizon to know where the object store is it needs to know where the Object store endpoint is so you can register the ECS node as The object store and so that any queries to these service catalog looking for the object store No to go to the ECS node here and we're just going to go show you that the node ending in 208 is in fact the the same ECS node that we saw earlier, but we configured the The auth provider in the namespaces So now that all all object store service request will be going to ECS node So now within the horizon UI you can navigate to the little UI they have for the object store Where you can create containers and objects? So here we see that we don't have any containers for the demo namespace. I'm sorry to demo project And now excuse me in the corresponding namespace in ECS. We just validate. We don't have anything there as well So now we'll go back into the horizon and we'll create a container creatively named container one and We'll create that and then we'll see that that container is actually has been created on ECS so there we'll should see it here and There's a container sitting on ECS Which is the same container that you're seeing through horizon and you could also upload content through the horizon UI So we'll pick a file There may be a pretty picture of hydrangeas and upload that to through horizon and that's actually being Stored on ECS through using the Swift APIs and so now we'll go to ECS and verify that to do this We'll use cyberduck, which is another tool that can access Swift APIs and we have a Bookmarked configuration. We'll just open that up just to verify that we're still talking to the ECS and You can see that's this is the 208 again if you recognize that from the earlier UI And this is the demo project ID. So all requests here will be issued to that thing so there's the container one and we should see in here the hydrangeas JPEG and You can perform some other operations in the horizon UI such as we can remove the object Pardon me And then we should see that object also has been removed off of ECS as well So we'll do a refresh for cyberduck and So there you see sort of the seamless integration between Keystone and ECS Thanks John. So next up So as I said, it's you know Swift integration Keystone integration is important A lot of you are probably writing apps for S3 as well, right? So you want to be able to provide compatibility with the S3 API. So what we're going to show you here is you know on one site Making edits using using Swift and then seeing them through through S3 and basically making sure that You know the data is the same and the reason why this is important is some of the vendors that that are in this market the way that they handle different APIs is They actually create silos of data Right. So, oh, this is all my Swift data. This is all my HDFS data And then you know when you're trying to get the the data to work together You actually have to work across different silos and I think That really defeats the purpose of having a flat hierarchy of an object platform Right, you really need to need to treat all the data the same and that's really what we're doing here And that's quickly. Let's show you what what that looks like All right, so here we have two different tools the access to two different APIs We have cyber duck and S3 browser S3 browser of course access is the AW the S3 APIs and cyber duck is for the script APIs and they're both going to connect to ECS so cyber duck will connect to To a ECS and then we'll go you know show you that in S3 browser that there is also nothing there And so we go back to cyber duck using the Swift APIs We will create a container and then that same containers will be you know accessible through the S3 APIs and by using the S3 browser. So there's a container and now back in S3 browser There's the same container now just want to emphasize. They're actually the same container not somehow copies of each other Yep, so in S3 here, we're gonna put content into that container. This is showed as sort of bi-directional So we're gonna put in another picture a lighthouse and so I'm doing one based although so and then we'll See that that same lighthouse is now in Swift Accessible through the script API, which is the same the same JPEG and so here through cyber duck We're gonna do a little edit To the lighthouse and then we'll put the edit back in and then you'll see that change Reflected back in on the S3 side. So here we're gonna do a little adjustment to that It's gonna desaturate it make it black and white finding the right slider All right, there it is. All right, so now we just save it as a more dramatic black and white photo and then We go back into S3 and then we can see that the change is Reflected back in the on the S3 API's and we'll see that the lighthouse is now black and white and that that just shows you the Crosshead capabilities between S3 and Swift and then we could also delete it and you'll see that that same file is now removed and it's Reboot, you know, you can see that's removed from both API's All right, great. So John what you're saying is if you have an app that's writing To the Swift API and then you have another app that's writing data to your object store using S3 Either app can access the day. It doesn't matter which it what what right same data You're just using two different API's to access it or manipulate it. Okay, okay? and finally Two parts two other things that we want to talk about is one is NFS, right? A lot of applique a lot of customers that we're talking today They have traditional file-based applications and they're looking to modernize these applications, right? They want to be able to use those applications To really want to be able to extend their file data across a global namespace and and for those of you that have worked with NAS systems You know how hard slash impossible that is right in a in a global environment with data centers that are all over the world so Being able to do that not only handle NFS data But also be able to treat it just like an object store where you're you're applying a global namespace to it is Is something that that a lot of customers and developers will find value in so what we're going to show you here is to separate? To separate sites right and ECS is sitting on both sites and we're going to write to one site using Swift And then we're going to pull that data from another site using NFS And so John do you want to show us what what's happening? All right? So here we're just going to show you in the ECS UI that we have two sites that are defined and Just try to keep track of you know some of the IPs associated with them just to refer back to you later So you're going to just navigate into where our sites are defined So you have sites one and sites two and just keep note that site one ends in 208 and that's site Say one isn't 208 and site two ends in 107 so we'll just show you those in different Tools so here in cyberduck again We have connections defined for each of those different sites and this one and cyberduck is a swift cyberduck is easy Yeah, right accessing the Swift APIs right and so this is site one 208 and site two here is defined for connection to 107 so First thing we're going to do is now we're going to connect it to site one And we're going to see that we already have a container defined now thing about this container It's been configured now to be accessible as a you know as an NFS file system So now this is an Ubuntu we can mount that same container We just saw called crosshead and just note here that we're on 208 which is equivalent of site one now and there's the The container name crosshead We'll mount it And so now we're mounting that container so the data in the container and the data on the mount are the same So from Ubuntu, we're just going to navigate into that Using the file explorer Go to the mount directory And we'll show you that there's nothing in there And so what we're going to do now is going to add add some content in there So we go to our text editor and add something pithy like hello world And then we'll save that off and Then we'll give it a navigate to our directory and then we're going to save it We'll give it a name first Hello dot text right now after we say that we can go it'll be going into cyber duck And you'll see that that that change is reflected on the Swift APIs as well So it's the same file then it'll be accessible via the Swift API So here we are back in cyber duck refresh There it is and so we'll just open up to verify that the content that we wrote On Ubuntu is the same content that we have here through the Swift API and it is All right So now that we're in a multi-site environment We're going to disconnect from site one and I'm going to reconnect the site to and just show you that the That the data is actively Propagated to site two and that the hello text there is the same Hello text that we created in Ubuntu and yes, there it is and so well while we're here We're just going to do a little edit Just to make it maybe a little more contemporary Hello from the other site Thought I was going to say side right All right, so we'll save that from site two and then you know the highly consistent aspect of ECS will Mean that site one, which is the Ubuntu mount will be able to see that reflected in the file itself So we'll open up the hello text and we'll see the changes that were made through cyber duck from site two And there they are hello from the other site And we can do one more thing. We'll just delete it and then we'll show that that delete is also reflected Immediately immediately over to the site to which cyber duck is already connected to so if we do a refresh we'll see that it is gone and That's clues that All right. Thank you, John. No Let's let's just take a step back and recap what what we showed you so first Not just swift integration, but also keystone integration so you can drop in your apps that are written for open stack seamlessly After that it's not just about swift also s3 So if you have other apps that are writing to s3 and you have apps that are writing to swift They're going to be able to access the same data It makes no difference to how the platform doesn't treat them any differently, right? And then finally you're looking to modernize your your nfs apps or file apps You can point them to the object store it speaks nfs and not only that it can actually you know Those two sites they could have been a world apart right and the data would be updated any changes made on one side would be updated you know as long as Speed of light laws are obeyed Almost instantaneously so that those are those are some of the things that we think really differentiate what our platform does right and and You know all of you are writing potentially writing applications to swift and what we want what we think we're doing here is taking Swift to the next level so you have strong consistency you saw that already reflecting changes across sites very quickly High efficiency, you know, we didn't we didn't talk about that more than the the initial part, but really being able to Go away from the modality of sort of 3x replication or 2x replication or whatever it is that that we're doing from a replication perspective to something that's so far lower overhead right so with with ECS you can get you can get overheads for you know in in a Up to 1.2 or 1.3 x right so as low as that right so you can really reduce the the amount of storage you need To back up your data and and have it stored in resilient format You need to be able to scale right we talked about it We have we have many customers that are deploying, you know multi-parabyte environments in ECS And and really there's there's no limitation to scale. That's a scale out It's a scale out platform, you know, you keep adding more nodes whether it's our turnkey appliance or you know Your own server you can keep just scaling up and then Developer productivity is a really really big deal in this new environment, right? It's not just about What what what we do for the folks on the operation side? It's also making sure that life becomes easier for for a lot of the developers in the audience So having that multi-protocol support makes it easy for you to decide whatever works best for your application You know, you can be assured that the data will be written You know all the same and you will be able to read data that's been come That's coming in from other systems as well So whether if you're running an analytics app that's ingesting all the data that's coming in from a line of businesses And you want to run, you know Hadoop on it you can you can run You know you want to run MapReduce you can you can do all that stuff regardless of how that data came in Where it came from you can be assured that the platform is going to take care of all of that for you So as an application developer, you don't have to worry about those things so that's You know ECS in a nutshell I don't think we have Yeah, one last thing like I said to you before ECS is available as we think that software defined storage is a really really important part of Providing choice to our customers. So ECS can be downloaded as a as a software version That you can run on you know supported hardware and then there's also there's also a turnkey appliance If you want to get started right away and essentially You know, we told you a lot of stuff. You don't need to take our word for it. You can actually go and download The the ECS free and frictionless download. It's an unlimited use Download not for production workloads, but basically, you know, it's you can deploy Container using a container you can basically deploy ECS in your environment. You can you can test it out and You know see if it meets your needs There's also an ECS community that you'll see once you go to this link where you know You can ask ask questions and and people on our team will answer them as well If you if you don't want to download something you can actually go to our sandbox environment So we have a test drive up environment as well that somebody can go in and and play around with the with with the platform so I'll stop there Questions from the audience please yes, so So maybe then you will have to help me out any how do we handle permissions across the different APIs? I mean, it's basically all configured using the portal right the ECS portal. That's okay. I can yeah, you want to take John's mic Yeah, thanks to me any other questions? You talked about strong consistency and high availability. Can you talk about? in terms of if you had network failures or a network partition between site one and site two would you just block Rights from site one if our users trying to write to site one to how does that work? So so what you're saying is if for some reason there's a network failure or one of the sites goes down How do you handle the the the rights across the site go down? But you just have a network where you can't site one cannot communicate to site two Would you block the right to site one because site two wouldn't be able to see it. How would that work? Can you hear me? Yeah, so The first the first part is the consistency so the way we do it is on read So basically you write on one site this site is considered as the owner you read from that site You don't care you are the owner you read from the other sites Even if you have a local copy we check with the first site if we have the last copy or last version so this is for consistency and Then you have to give the choice to the customer. Do I prefer consistency or availability if you prefer? Consistency then we don't give you read at all. So you lose you have network partition. You cannot contact the owner I cannot check the consistency. I don't give you access that can make sense for some use cases and The other thing is no I prefer availability and in this case we we just like let's say disabled the consistency check in that case and we Do something that is really I think Unique is that we give you read write access. We don't just give you read access. We give you read write access and Basically, we transferred your ownership to do the site so that means that imagine you have an application that's right on site one and You have a network partition. Even if we transferred your ownership, there is no impact because there is nothing on site, too Now imagine you lose your site Then you want to move your application to site, too And you have transferred your ownership and now you're fine because you can read and write and then we just reflect on the other side So we and then he is an application developer. You don't have to worry about that right like once the system is set up It happens seamlessly. Yeah, so what we've seen in the past because I was working on at most as well Which was our previous platform is that we were doing something very similar to all the other platform in the market Which means that you cannot check you are read only and it was a big a big problem for adoption of object storage because many people are coming from file usage and they're used to have like a Failover fail back, you know, it's very rare to have active active application You have many application in different sites, but most of them are active in one site at least for one part of the data So what they want is they don't really care about this active active access They want to have a failover fail back and that's what we give them. Thanks. Any other questions folks? Yeah, please Encryption so question about what we do for encryption So we have two different So we we don't do that with self encrypted drive because many customers don't want to pay for that So we have done that with in the software. I've seen one session this morning I think that was you were under you are in the room. I recognize you and that's very very similar to the implementation It's done in by the swift community currently so we are using like a key that different level name space container object and so on and What we have done that you could do also in the swift community is that on the SPI side we give also the flexibility that you can use the encryption You can provide the inclusion key through the API for each object so that the application can manage itself the key So you have two different options either you don't care You want to be done you want that to be done by ECS itself and then it will be just for that specific container I want encryption and it's managed by ECS itself or you can say I disable it on ECS But I send the key whenever I need that or I just say for that object I want to activate encryption and for that we just follow the SPI like Amazon is doing it Perfect. Thank you. We'll be here if you guys have more questions, but I want to do the the raffle which I'm sure you guys are all waiting for Let's see, okay, who's gonna get the Amazon echo funny it says my name on here. All right. Thanks guys Take it number last three digits. Let's let's do first three. We want to build a suspense. So 970 who won? Okay, nobody. All right, so I'm gonna take it 970384 All right, great We're here if you have more questions and I Hit me up on Twitter if you guys want to talk more, but thank you for attending