 Well, welcome to the set. Can I get a quick show of hands on this one? How many people are in the room right now because the word sender is in the topic? Can I get a couple? All right. How many are here because containers is in the topic? I've got a lot more. And how many because persistent storage is in the topic? Excellent. Good. That seems like the most, which is good because that's what I'm actually excited about and that's what most of the talk is going to be about. How many are here under some sort of social obligation because your friends are coworkers of mine? All right. How many are here just killing time until lunch? All right. Good enough. So what we're going to talk about today is persistent storage for containers using sender. And so how do we get started with this? All right. Let's start with me. So I'm Drew. I've been a Linux nerd since about 1995. My first kernel was 1.2.13, downloaded on 28, 3.5-inch floppy disks. I've really not stopped since then. A couple of years ago, I moved to San Francisco to join a startup called Cloud Scaling, which some of my co-workers are here in the audience. Cloud Scaling, I don't know if you remember, was a distribution of OpenStack. But it was actually more of a product company. We didn't have a distribution that you could download. We had more of a, it was a product that you could purchase. We sold a hardware design along with the OpenStack software to create sort of a complete OpenStack solution, arguably the most advanced one that was available at the time, which is why we kind of got the attention of EMC, who acquired us in late 2014. If you know much about EMC, EMC is a very sort of old money East Coast, very corporate style corporation, and it doesn't... So we went from a spunky little startup of about 32 people to 67,900 over the span of a little while. And incidentally we got picked up by Dell, while the acquisition is still there, the merger is still happening, but I think that puts us up around 200,000 employees or something like that now. So it's a bit crazy. Anyway, when I joined EMC, I was drafted into a group called Code. And Code is a group, EMC Code, what we do is we do a lot of community outreach. We're open source engineers. We focus a lot on open source infrastructure, and we focus on infrastructure as code. Everything we do at Code is public. So everything we write, we publish to GitHub, and we sort of preach the gospel of how to properly use GitHub within larger enterprises. So you see a lot of enterprises that when they try to release open source for the first time, they just take a product and they dump the complete product on GitHub and never touch it again. We actually act more like we develop the software from day one on GitHub. We solicit comments and pull requests. Everything that we do is published there. All of our meeting structure, all of our notes for our interactions, our visits to conferences, everything else is on GitHub. We also have a Slack channel or a Slack server. It's codecommunity.slack.com, and that's open to anybody that wants to join. It's not just internal to EMC. So if you're one of these people that sits on Slack at work and you want another place to hang out with people who aren't necessarily your coworkers but who are also in the same industry, codecommunity.slack.com is a good place to go. And just like my atmux Twitter, we have an atemccode Twitter and a blog atemccode.com, so that's worth your time. So introductions done. Let's talk about persistence. How many here know who this guy is? I kind of like two. Well, you'd think with an open source audience, there'd be a lot more sort of juvenile. This is a character from a cartoon called Rick and Morty. And Rick and Morty, it's not a cartoon for kids, it's definitely more of a late night thing. The reason he's on the screen right now is because this is a character called Mr. Meeseeks, who comes from an alternate dimension. And he comes into play when you press a button on a box. You press a button, Mr. Meeseeks bursts into existence. And then you give him an instruction like wash the dishes or clean the floors, and he does that with glee. And upon completion of his task, he vanishes and that's the end of his life cycle. Mr. Meeseeks is a perfect example of a third platform application. He's basically a microservice in cartoon form. The reason that's important is because that's what we've been focusing on so much lately, that's what we all sort of consider the future of writing applications for the web and for web scale. Is to write applications that are small, they do a single task. They come into existence, perform that task, and then they disappear. And that's really what containers and microservices are all about. Pardon me, I brought some water with me and I hid it on myself. There we go. So the problem with microservices that people don't really talk about very often is the fact that you can't really take every application as it stands and turn it into a microservice. You can't just take all of your legacy stuff and put it in a container and throw it up on the web and it's just going to work. What you need underneath is a way to maintain persistence within those applications. Every, well not every, but a huge majority of current applications require persistence. So one of my co-workers has an example. He says, even the simplest of applications in Windows, if you use, if you're Windows people, if even the most basic application that comes to Windows, he uses notepad as this example, even notepad requires persistence. If you use notepad and you go up into the menu and you click one of the view buttons or something like that, one of the menu items is word wrap. And all that does is just let your, as you're typing, whether it's going to wrap the word down to the next line or not. But if you click on word wrap and then you close the application and then you open the application again days later, that little checkbox is still there. So that bit of information has been stored somewhere. It's persisted between that application closing and not. So even in the most simple of microservices applications, even in the, say, something that just serves up a web page or, and my background is almost exclusively in web applications. So I might kind of tend towards that for examples. But even in the most simple of web applications where it's just PHP kind of chewing on something and spinning out a web page, if the user is logged in or if there's a cookie set or if there's a bit of information that has to be used somewhere that's going to persist as the microservice delivers its content. So that data, how we do that in modern web applications is we store that in either a database or a caching layer or a memcached D or a redis or some sort of key value pair. And that has to persist somehow. That has to be written to disk because that's valuable information. But when you're deploying all these applications, I mean, you're going to deploy your PHP churer and your image server and all of these things as microservices. But you want to deploy these caching layers and these databases side by each with it, probably on the same servers. Probably in your same container servers. So how do we put an application that persists data into a container and then have that container able to persist, basically? So the way we do that is, well actually we're going to talk about that a little bit later, OpenStack does somewhat provide persistence. I mean, OpenStack isn't really the substrate for microservices to begin with. OpenStack gives us the ability to run virtual machines. Those virtual machines have storage that storage persists. Once a virtual machine goes away, the storage doesn't necessarily go away. The storage is attached, but when that machine goes away, the storage can stick around. That's pretty normal. So we throw Docker on top of that to provide this layer of easy deployability, layer of rapid deployment, rapid instantiation. But does it provide persistence? And the answer is, well, sort of. I mean, with Docker, you can have a Docker volume. And a Docker volume is provided from the command line, and it writes at the disk. And if that container goes away, that Docker volume doesn't necessarily have to go away. So where's the problem? Well, let's follow up from the start. If you start with a container, you deploy it to a container server. That container server has storage attached to it. The container goes through its life cycle, dies, no problem. So what happens when you start to grow? Your company gets bigger. You start to have multiple container servers, multiple containers. You start to scale past what's available to you in your hardware substrate. What do we do? You need a scheduler. You need something that's going to sit in front of that, and tell those containers which container server they're going to sit on. You need something that's going to be able to auto scale those containers up and down, so you're making the most of your hardware. But what do you do about storage? So what we did to solve this is we built a little tiny application. And it's really, it's pretty basic. But it solves one thing and it solves it perfectly. And this is a piece of software that we called RexRay. And RexRay is, well, we build it as a storage introspection tool. But what it really is is an abstraction layer for the storage for containers. What it is, it's a driver for Docker that sits inside of Docker and allows Docker to talk to more than just local storage. So when we started this, we had a mission and that was to create a storage management component. We wanted to be able to handle as much different storage as possible. And one of the first ones that we built up was access to Cinder. Which is great because Cinder's APIs are very well documented in the very public. So we also jumped into EBS on AWS. We have native support for Google Compute Engine. And I mean, as is probably very obvious, we support all of the EMC style storage. So we got Isilon and Scale.io and Vmax and Extreme.io. And one of the most important ones is that we also support a driver for VirtualBox, and I don't know if you use VirtualBox, but it's a place that you can build virtual machines on your desktop. And why that's really cool is that now using this driver, you can build Docker containers on your desktop and then have them attached to storage on your desktop. And then deploy them out into any of these other substrates, pardon me. And the commands will be the same. So the driver works exactly the same across all of these different layers. How does it actually solve the problem? So basically, as I just said, you start Docker and on your command line, you specify it as a volume driver. The container attaches the storage to that container. And that's it, it works. It's simple, it's elegant, and it's simplicity, but it's pretty basic. So here's the big question, why is that exciting? Well, basically it makes running containers across many different substrate layers simple and easy. Again, if you like to develop on your desktop, you can deploy those same containers directly into GCE without making any changes, except to a single tiny configuration file that sits on your container server. So wherever that Docker engine is running, all it has to do is reach out. All you have to do is deploy it out, and that container works the first time. Pardon me, sorry, much love to the facilities crew here, but I'm pretty sure that caffeine or that coffee was not decaf. Anyway, here's what the config actually looks like. It's tiny, it's YAML, it's super easy. As you can see, all it's really doing at the top is just declaring that open stack is gonna be the storage driver. We have preemptibility, which is a really interesting feature. So why that's, why preemptibility is awesome is because if you have a container that has storage attached to it and something goes wrong within that application, say it's a database or something, and you want to replace that container with a new container runtime, you can have that new container come up and seize the storage, write it from under the previous container and write to it immediately. So if that container has had a problem, this enables basically all kinds of high availability and yeah. So basically, you can steal storage from the other containers around you, but only if you have preempt to true. The rest of the config is fairly easy, I mean it's the exact same thing you'd expect, with open stack, you have the URL for auth and the rest of your creds, if it was an AWS configuration file, it would be AWS and then the auth URL and your access key and secret key. And yeah, so this is very simple. So rather than continue to ramble, I'm going to show you guys something really simple here. If I hit over here, I should have access to, you may have to have me log in. So side note, this is actually open stack running on top of AWS, which pleases me kind of because, well, when I was a kid, we had a game that we used to play that we called stacking emulators. And in stacking emulators, you'd try to see how deeply you can nest things. And another co-worker of mine was telling me today that IBM was just showing this morning, what was it? Kubernetes and Swarm running on Mesos running on open stack. So this doesn't appear to be loading though. Well, good. Well, that's not great. Let's see if we have SSH access. Well, that's extra not great. Is there a 2G one? We'll try. What was it again? Cal Newton. Fingers crossed. There we go. That's better. Thanks for the tip. So this is open stack running on AWS. So what we're actually going to show is containers running in Docker, running in a virtual machine, running on open stack, running on a virtual machine, running on AWS, which pleases me unnecessarily, but I really like that kind of stuff. For what it's worth, drawing up my friend who won the ultimate nested virtualization track, he was an Amiga guy and he had Amiga 3000 tower running Amiga OS 3.1 in 3.0 emulation mode with hardware 386 bridge board running IBM OS 2 running Windows 3.1, running a Windows 386 emulation, running DOS 6, running DOS 4 mode, running a Commodore 128 emulation in Commodore 64 mode, running a VIC-20 emulation, running a Radar Rat Race. Okay. So here we are. Team likes attached. So what I have here is two machines, Docker 1 and Docker 2. And on Docker 1, I'm just going to do Docker PS. Then we have nothing running. So I'm going to create a volume through the Docker binary. And what this is going to do is it's going to use the X-ray driver and it's going to create a volume in Cinder on the OpenStack instance. So if I do Docker, volume create minus, minus driver equals X-ray. My name equals Postgres data and minus size equals 5. So this is going to create a 5 gigabyte Docker volume. And if we do Docker volume LS, we should see that as the third one in the list. If we go back to here and go into the project and compute volumes, you can see Postgres data was just created. It's available. It's not attached. All right. So I've got a long command here somewhere. That'll do. And on this Docker, I'm going to run Docker. I'm going to give it the option Postgres password. I'm going to use the Docker volume driver equals X-ray. And I'm going to mount that volume Postgres data into our lib data. And I'm going to run the Postgres container. So as this comes up, you'll see that it's going to initialize that drive. Since we just created it, it doesn't have any Postgres database data in it right now. So what's going to happen is you're going to see a bunch of messages here sooner or later saying that it's initializing that as a Postgres database. Once that's done, it's going to shut itself down and then restart and then go into listening mode. So if we go over here to a Docker PS, this is a completely different machine. So you can see the container isn't running on this machine at all. But if we do Docker volume LS, we can see that because Docker has access to X-ray, it can see that X-ray Postgres data volume that's sitting on Cinder one layer up. Competing, syncing. All right, it's ready for startup. So that database is now up and running on Docker and on Docker server one. And over here, Docker server two, we're going to run the exact same command. And what's going to happen here is it's going to start up, but it's going to use that same volume. So it's going to use volume preemption. It's going to steal that volume from the other Docker container. Again, it's going to take a minute. It's a little bit steady. This is the point where I guess I can tell you guys a joke. I'll tell you a UDP joke, but I can't guarantee you get it. Okay, and that's up and running. You can see that at the top of the log, it says the database system was interrupted. It was not properly shut down. Recovery in process. It recovered that record. And it says a redo was not required. So this database is now up and running on a second server. So where are we at here? Right here. So just to recap what just happened. In that black box, we created a Cinder volume using the Docker command line. We ran a container with a database using that volume as the database storage. And then on a completely separate Docker server in the cluster, we ran another copy of that database and we reconnected that storage. So we preempted the storage, stole it from the first Docker container, and started using it without notifying the first one. So why that's important and why that's great is that it enables an awful lot of crazy high availability stuff. One thing that I really like about this tool in general is that there's no central cluster management that had to be used here. These are just a small binary that's running on each of these Docker servers. It's a native Go binary and it's talking directly to Docker as a Docker driver. So that's pretty much everything for Rexray. The real question is what's coming up next. And so what we have coming up next, actually it's not coming until next week, but we're dropping a new piece of software that we're calling Polly. And Polly is a polymorphic storage scheduler. What that means is rather than being a driver directly into Docker, this is a driver that goes directly into Mesos and Kubernetes and Swarm. So rather than just allow Docker to ask for storage, we can now have Mesos and Kubernetes and Swarm and even Cloud Foundry request storage, regardless of what the storage backend is. And so that's pretty important stuff. The official launch will be next Wednesday, but the code isn't already up on GitHub on our emccode.com, or sorry, github.com slash emccode repository. And if you're into that sort of thing, you should really check it out. And I think that might be just about everything. So questions? That's a good question and that's an application level thing. So we do it at a very, very low level. So basically what's happening is it's literally stealing the volume and dishing it over. And that's kind of up to the application what happens to that data. So the ability is there. But whether Postgres is written to recover cleanly from it, Postgres is one that actually is because it maintains a transaction log and that transaction log is written before the data is actually written to disk. But this is something that would have to be written into your own application. Like you don't provide a playback log or something that's waiting. Oh, sorry, actually. So just for note, the question was what happens to the data when we preempt? And if there's a second question coming up, would you mind using the microphone? Oh, there we are. I had a couple of questions. So in the YAML file, you're using the admin port for Keystone. Is that required? No, no. Keystone, it all needs to be as any user that has access to... Okay, so we don't need the admin port. Correct. The second question was if Cinder at some point supports Active-Active to multiple instances, then you guys will plan to add... Instead of just stealing, maybe sharing that volume between containers? That's a really interesting question. I am not sure and I would have to escalate that one actually to the architects of the software. But I'll absolutely add that as a note. Okay. Thanks. Cheers. Is there any advantage in using native Docker plugin versus the open Cinder driver plugin? Well, I would say the main advantage would be the fact that it's a tiny go application that works directly with Docker. If you were going to just use Cinder and Cinder alone, then that makes sense. But if you ever had any interest in moving beyond Cinder, in moving into maybe using Ceph as a back end, or maybe using some of your workloads over into AWS or GCE or anything like that, then it would make more sense to have something a little lower level. Preempt would not work though. Hi. May I know how much your is a luxury driver? It's an active production in quite a few different places now. We've had, we have some marketing people here, but I'm not exactly sure how many of the names I'm allowed to throw around. But if you go into our GitHub and look at the contact information for people who have contributed to Rex Ray, that might give you an idea of who's actively using it. But I know it's in production. Okay. Thank you. So when the second Docker container there attached to the volume, it triggered a detach from the first container, right? No, it did not detach the first. It has the first, doesn't know anything about it. It's just had the storage has disappeared. Right. So how would that appear in the first is actually my question. Oh, yes. What does it see when the Cinder volume detaches from the first host, move to the second host? All it sees is that the storage went away. So the usefulness of this is more or less, it's not to put this into your work stream as this is something that you can do to just like to move a workload around. It's more like if that container has failed or if there's a problem with the substrate or the server that that container is sitting on, you can continue that workload on another container server. Right. So from its perspective, the directory is just suddenly unmounted and gone blank. And you need to do that in application logic. Exactly. If there was a possibility that the first one might actually be live, but you're not detecting that. Sorry, can you say the last part again? If the first container is dead, you don't have a problem. But if the first container is unresponsive but not dead, such that you've triggered the failover, then you're obviously going to need to just cleanly shut that down in application logic and handle the fact that something is... Either your application logic or perhaps your scheduler could determine that. If your scheduler is determined that one of your containers is unresponsive, then probably it's going to kill it off anyway. Right. Back up at the daemon level then, how does that detachment and sender volume appear to the host? A volume has disappeared from that Docker daemon and it's appeared on the other one? Or is it that there is a volume still registered there but it doesn't actually have any data attached to it? So all of the Docker servers are seeing storage presented to it as a single layer. All that's changed within Docker is the name of the container that it's attached to. Oh yes, that's right. It's at the volume driver level. Correct. Okay, thank you. Cheers. What is the storage back in that directly support right now except sender? Say again? What is the storage back in that... Oh, well, we've got sender. We've got all of the EMC backends, of course, because we're EMC, which is Scale.io, Extreme.io, Isilon, and... What's the fourth one? Vmax. Vmax, correct. Yeah, say again? Backend. Oh, this was sender. This was sender running on OpenStack. The whole thing was running on AWS, so we were talking to the sender demon for the back end. We also support... There's a plug-in for VirtualBox as well, so for your desktop. Anything else? Say one more? Any plans to support EBS? Absolutely, we support EBS natively right now. I guess not. All right, I guess that's fairly early, but thank you very much. Cheers.