 So, this talk is about file systems and specifically NFS used as Cloud Foundry Services. You will have heard reference to this topic from both Sam Ranji and Chip Childers yesterday. In this talk, we're going to dig a little bit deeper into the details of what you need to do if you want to use this in your own apps, why you would want it, how it works, all that stuff. So like I said, we'll talk a little bit about what it is, why we built it, go into the architecture real quickly. We've presented on this topic a few times before, so I won't go very deep into the architecture, but there's existing presentations that get into the details if you want to link back to those. And then we'll talk about what you need to do if you want to use it, which is the real focus of this talk, because we really want you to try it out in your app, give us feedback. It's just now in the latest releases of Cloud Foundry officially supported and available in PCF as well. And then what you need, what we built isn't exactly what you need, I'll talk a little bit more about what you can do to develop your own solutions based on the framework that's in place, and then updates on what's coming, and I hope we'll have time for some questions. So what is it, and why did we do it? Essentially volume services in Cloud Foundry is a set of modifications to existing systems in Cloud Foundry that enable you to use a service broker and a volume driver to mount volumes into your container, so essentially mount file systems that are available to your application. And on top of that, we've built a few implementations of volume drivers and volume service brokers that you can use for particular kinds of file systems. Now why does anybody want this? It's Cloud Foundry, so you're supposed to be building 12-factor apps, which should be stateless, they should only use network-attached persistence. So what do we need volume services for? And even if you do need them, you can just start a privileged container. And do your own volume mount, so who wants this thing? Well, privileged containers, you might be aware, or I don't know if you've ever used them, but it's a very bad idea, because essentially you're turning off all of the container security around your container. So if someone in the container can get root privileges, then they can go out into the VM and start doing terrible things. It's a bad idea in your data center, it's an even worse idea in a multi-tenant environment. And thinking a little bit more about the 12-factor thing, really network services are just some protocol that you use to write data, read data off of some network-attached persistence, which isn't really that different from a file system except for the network-attached part. So by casting file systems as network-attached storage that you interact with first by using a Cloud Foundry service broker, we're essentially making file systems into a network service. Not much different from a database or Amazon S3 or Swift, Blobstores, you name it. And by doing this, we make it possible for your apps that weren't exactly 12-factor before to be 12-factor apps. Now we don't protect you against all the other stupid things you might have done, right? So if you're keeping session state and caching things and doing other stuff or other engineering abominations, we won't fix those for you. But if your problem is that you rely on a file system, we can address that problem. So now we'll just take a quick look at the architecture. And like I said before, we've presented on this architecture a couple times in the Santa Clara Spring Conference in 2016, and back when it was a proposal in 2015, and it's spring one. So in the end, I'll give links to those presentations if you want more details. But at a super high level, this is how Cloud Foundry looks with Diego today, right? You'll have your Cloud Controller. It sits there taking requests from the CLI or whatever. And it talks to the BBS, which is a Diego brain, over a little piece called the CC bridge, also known as NSYNC. That sends messages to BBS to tell it, oh, I have an app or a task or something. I want to place it somewhere on a cell. BBS then sets up an auction, talks to the various reps on the Diego cells, places your app makes a container off you go. Now to introduce volume services into this picture, we have a few pieces, right? We have the actual volume, whatever it is, could be NFS or SIFS or some other network at that storage. We have a service broker, which just like any other Cloud Foundry service broker, this is responsible for creating or identifying the resource that you want in whatever language that resource speaks. And then translating that into information that's passed to the Cloud Controller. The final piece is the volume driver. This is deployed into the Diego cell, so if you have 12 cells, you'll have 12 copies of your volume driver. And that piece is responsible for taking whatever information came from the Cloud Controller and using it to mount the volume into the Diego cell. So what does a volume mount look like? It's kind of like this. When you bind your volume service to your application, that results in a call into the service broker. The service broker then does whatever it needs to do. Normally it's just a little bit of bookkeeping, and it sends a response back to the Cloud Controller with a volume mount data payload in the bind response, and that gives the rest of the system whatever information is necessary to go attach to that volume. Now it might just be an IP address, maybe it's some kind of credentials. Really depends on what the driver needs in order to do the mount. That gets passed through the regular channel, gets stored in the Cloud Controller's database. And then when you start your app, it goes to place it, and at that point it goes through the bridge into the rep. The rep parses that out and finds the correct volume driver for whatever kind of volume it is, asks the volume driver to do the mount. The volume driver then does the mount onto the cell and responds back to Garden, which does the regular bind mount from the cell into your container. This is using the facilities that Garden already had for the volumes that it already mounts. If you use any kind of disk in your app today, those are bind mounts that are done by Garden off of the cells file system into the container. So that's more or less how it works. Let me talk a little bit about how you use it. If you actually want to use it in your app, then you need to set this up somehow. This is a very verbose slide. We'll share the slides so you don't need to take down all of this information, but I wanted to kind of record it so that when you get the shared slides you can look at the particulars. We built the first versions of this into Cloud Foundry a couple of few months ago, starting from Cloud Foundry version 238 thereabouts. That version was using an experimental service broker API with the first cut of it. So that's the version that will be, for example, in PCF-18. So if you're using PCF-18, you'll have that experimental API. We subsequently honed that API a little bit because it was experimental and nobody was using it yet. We didn't invest in backward compatibility. So there is a slight incompatibility between that version and the version in the shipping that we announced just recently. So you have to be a little bit careful about what version you're using to make sure that the shape of the volume mount that's returned by your service broker matches what Cloud Foundry is expecting. The good news is if you're using any of the service brokers that we developed, we've built version detection into those service brokers so that you don't really have to worry. The latest version of our service broker, it will run with whatever Cloud Foundry version you have provided that it's recent enough to have any kind of volume services supported. And we'll talk a little bit about how you would do the same trick if it's your own service broker later on. So that's kind of the Cloud Foundry and Diego side of things. So we'll talk a little bit about the service brokers and volume drivers that we developed. The first one is this thing we call local driver and local broker. And these are packaged together into a local Bosch release. This is really intended kind of as a test sample broker. If you're using Bosch Lite, then this is a good option because it just enables you to write an app that uses a file system that's mounted through volume services. And then you can later on take that app and use it with a different, you know, EFS or Cephaphas or whatever else you want to use in a real Cloud Foundry deployment, but you don't have to make any changes to your app because after all, they're all just file systems. So the real limitation of this implementation is that it doesn't actually do any mounting of anything into the cell, right? When the local driver gets a request to mount something, it just makes a sim link from one directory on the cell to another and pretends it's a volume mount. So this works fine for single cell deployments because we're just sharing a directory between apps on the same cell. If you have a multi-cell deployment, it's not going to work at all because it's not actually shared, right? It's just a directory sitting on the cell's file system. So two copies of the app that are placed on two different cells are going to get two different file systems. Like I said, the real goals of this are not to pretend that a local directory is a shared directory. The goals are to enable you to develop apps in a slightly easier environment than a full scale AWS deployment. And secondarily, to provide a simple seed project if you want to build your own driver and broker because most of the code in the local broker and the local driver are all about the plumbing that's needed in order to build a broker and a driver. The actual work that it's doing is pretty much nothing, right? It's creating a directory on Create and it's simlinking that directory on Mount. So you take those bits out, you put in real stuff off you go. Next implementation we did is Cephaphis. This is quite a bit realer. This is based on the Ceph file system. So this gives you real shared file services anywhere you can run Cloud Foundry, more or less. It doesn't work in Bosch Light today. I suspect that it could, but we had some issue with the mounting happening because it's using Fuse mounts and I guess something about the fact that Bosch Light is running containers and not VMs caused it to not work. We didn't spend a lot of time investigating it because it didn't seem to be a priority. But if anybody here has a deep burning interest in seeing staff work on Bosch Light, let us know because we're definitely taking our priorities from user input. As it stands today, the only manifest generation we have for this is for AWS. But there's no particular reason that it couldn't work in your VMware environment or GCP or anywhere else, right? I mean, there's nothing special about Amazon apart from that. It's just what we were working against. And then we have Amazon EFS. This is a nice easy option if you're deploying into Amazon and you might know this is a fairly recent offering from Amazon prior to a few months ago that it didn't offer file systems at all. But this driver and broker pair will create a file system in Amazon for you and mount it into your app. So basically, you just need to give it your Amazon keys and get it deployed into your Cloud Foundry environment and provided you're running in EC2 should all just work. A couple things to note here. Right now, as of today, the read me for this release is lacking quite a bit. It says something like, in progress. So it's not very helpful. So I tried to put a little extra information into this slide about what you need to do if you actually wanted to deploy this in Amazon. Basically you're creating a Bosch manifest and Bosch deploying it and then you're doing some tricks to get the driver part code deployed into the DAO cell when you do your DAO deployment. We're going to try to very quickly patch up the read me so that it gives you meaningful instructions when you go there. So I hope that by the time you come to it, there's something useful for you there. If you find that that hasn't happened, then just reach out to us. There's contact information all over these slides, so we're happy to hear from you in any way you choose to reach out. I think the only other thing to say about this particular broker driver pair is that if you find yourself building your own service broker or volume driver, this is a very good example of an asynchronous broker. So you may or may not be aware Cloud Foundry has this pattern for its service brokers that enables the service broker to do slow operations asynchronously. So if your service broker isn't going to complete within 60 seconds, it has to come back to Cloud Foundry and say essentially we're working on it and then Cloud Foundry will continue to pull your service broker to find out, you know, how things are going. Because all of Amazon EFS APIs are also asynchronous, we pretty much had no choice but to implement our service broker as asynchronous, so I think there's kind of good patterns in this code if you find yourself building an asynchronous service broker. Okay. And the last implementation we have is the EMC Rexray driver. This is a volume driver that was built by the EMC code group. And maybe I should have mentioned this earlier. Our volume drivers are built against the API that Docker uses for its volume plug-ins. So Docker supports volume mounts through plug-ins and you can build your own Docker plug-in that will, you know, so long as it implements their plug-in API, you just feed Docker your plug-in and then you can use that driver in order to mount volumes into Docker. In the absence of any standard for how these things should look, we picked up the Docker API as the next best thing. So Rexray is a Docker plug-in that allows you to mount all different kinds of EMC storage into Docker but because we use the same plug-in API, it also works in Cloud Foundry now. And the team in Cambridge, Massachusetts has paired that Rexray, the existing Rexray driver with a service broker that will manage the service broker side of things in order to, you know, create volumes for Icelon and other EMC storage. So if you're using Icelon or you're thinking about using Icelon, this could be a very attractive option. If you want more information on this, make sure to catch the talk at 10 of 2 this afternoon. Dr. Fung, back to the room, will be presenting on this and it has a very nice slick cool demo. So if you leave today this morning disappointed because I'm not going to do a cool demo at all, I'm just giving you boring slides like this, then you should go to Victor's talk, which will be more entertaining. So definitely catch that, he'll be giving a live demo of Icelon running on a bare metal CPI, doing cool stuff, I want to spoil it, and I've got contact information for Victor if you want further details on that. Definitely if you're thinking about this and it's a good idea to engage with that team, they would be delighted to work with you on whatever your particular requirements are. What we've found so far is that with Icelon deployments, there tends to be a lot of variation from one to the next in terms of the use cases. So it's a good idea to get in touch and make sure that our software works for you. And if you want to see an online demo, we have one up on the EMC Dojo blog, sorry, still training myself, the Dell EMC Dojo blog, which is worth watching. So we have a few options for how to mount volumes into your Cloud Foundry and what if you're looking and you're thinking, none of these is quite right because I need to, for example, mount my existing storage or have SIFs or who knows what. If you just want something slightly different from what you find out there, then we'd be delighted if you put in a pull request. Happy to hear your feedback. If you have any suggestions on the work we did so far, you dig into the code. Be awesome to hear from you. If you want something completely different, then really you're looking at building your own driver broker pair, or maybe you found a driver from Docker or wherever else and you just need a service broker, or you're going to adapt one of our drivers, but the broker needs to be completely different because you don't want to create new network storage when you use existing ones or whatever. I think a lot of variation will happen on the service broker side of this pairing. The driver in the end, especially for NFS, is just doing an NFS mount and there isn't much that's going to change from one mount to another, apart from all the options we pass it. But on the broker side, it could be anything. You're binding your app to some network thing. Who knows where it came from? So in order to do this, if you need to build a volume driver, then your best starting point is going to be the Docker volume plug-ins back, link here. If you need to build a broker, then you have to build it against the Cloud Foundry Service Broker API, which has some minor extensions for volume mounts, again there's a link. If you're in Golang, then we have some good base libraries for both of these things, that we use ourselves in order to manage the HTTP parts of the API, because they're both just REST APIs basically. So for both of the broker and the driver, it's a good idea to start with those implementations because they'll do all of the, you know, HTTPM points, basically, you just have to provide the implementations. And I mentioned this earlier, but we have this nasty version discrepancy between Service Broker API 2.9 and Service Broker API 2.10. If you use our base broker API library, then the code that sniffs what version Cloud Foundry is and converts your Service Broker 2.10 response into a Service Broker 2.9 response automatically is built into that library. So if you use that library, then you'll automatically get backward compatibility to older versions of Cloud Foundry, so maybe we're thinking about. And then, you know, just to repeat, if you're building a fairly simple driver Service Broker pairing, we built our local driver broker pair as a seed project for people wanting to build their own brokers and drivers. So that's a very good place to start. You know, if you're in Java, you're a little bit more on your own, although I think Cloud Foundry has good resources for Java-based Service Brokers. So what's coming next? What are we doing? Where's the future of volume mounts in Cloud Foundry? Future disclaimer. This is software, and we haven't delivered it yet, so if you believe everything that I'm going to tell you in the next few slides, then I have a very nice bridge. Maybe you want to take a look. So driver deployment by Bosch Add-on. I kind of talked briefly about this before, but as of today, in order to use volume services, you somehow have to get your volume driver deployed onto the Diego cell. The way we do this today is to use Diego's manifest generation in order to kind of inject our driver, the standard scripts for generating a Diego manifest, have a little bit of, like, kind of a side door where you can add a driver stub file and then it'll wire your driver into your Diego manifest, but it's a spiff merge. It's kind of a, it's heavy, deep into the kind of yucky, yaml parts of Bosch deployment. So this is definitely not optimal, and it's also not very flexible. Like it's, you don't get to actually pass any properties to your driver using this mechanism. You can also just go hand-edit your Diego manifest, but that's also terrible because hand-editing yaml is no fun, and, you know, if you need to take a new version of Diego, you'll have to do it all over again. Bosch has a mechanism called Bosch add-ons. They're available in Bosch today, and they allow you to set what they call runtime configuration, which lets you run jobs on every VM in your deployment. So if you have some common thing you want to run on every VM everywhere in your, Bosch deployment, you can use an add-on. The problem is, as of today with the current version of Bosch, you don't have any option except to run your job on every VM in your deployment. So nobody wants this, right? Having a volume driver on your Diego brain and your cloud controller everywhere else is terrible. So it's not really practical right now, but coming in the next version of Bosch is the ability to introduce filters to say, I only want to put this on this VM, or I only want to co-locate it with this job. So that will give us the ability to just target Diego cells and deploy drivers as Bosch add-ons. The nice thing is that then if, for example, you used ops manager to deploy your Cloud Foundry, you can still set an add-on and deploy drivers into an existing Cloud Foundry without having to kind of interact with whatever the Diego manifest looks like. So that should be very exciting, much better. It's coming very soon, we don't know when. We keep asking the Bosch team, and I keep saying, not yet, but why aren't you using our dev builds? Validation and error logging, right now, unfortunately, if there's anything wrong with your service broker and you give it like a bad payload, it doesn't tell you. And then you don't actually find out until you try and start your app. And then when you try and start your app, it tells you that the app crashed. And the CF logs don't tell you anything. So this is a bad situation, I guess. We have a couple pull requests in a couple different areas to try to get better logging out of the system more immediately when you're binding services to apps. So if you've got a binding, like if there's something wrong with your service broker and it's giving a funny response, we'll have better validation to tell you right up front, like you've got a nasty bad volume mount that we can't parse because x, y, z. And likewise, if Diego has trouble placing your app, because maybe like your driver's not actually running, your broker says, oh, I need X volume driver, and there is no cell that has one of those volume drivers, we should have coming soon better information about why your app couldn't be placed. Right now, basically, you're fishing around in the Bosch logs. You're ssh-ing into machines and looking in logs, or you're doing a Bosch log and unpacking the logs, and that's kind of a big hunt. I think we're running short on time, so. The other thing we have in the works are potentially a bunch of different deployments. We're going to try and build this into PWS, Pivotal Cloud Foundry running on Virtue Stream, PCF Dev, and the native hybrid cloud stuff that we have coming out of Dell EMC. This is all very much speculation at this point. If any of these things seems appealing to you, then definitely let us know, because probably if we don't hear from anybody, we're not going to build this stuff. So your vote counts in all of this. We try and do our development based on data. So if we don't get a signal, probably won't take an action. And finally, we've got this essentially, I mentioned earlier that we're using the Docker plugin API in the absence of a standard, but we are talking to folks in the OCI and hoping that either under the auspices of the OCI or CNCF, we can come up with a new standard for what volume plugins should look like to give us a little bit more granularity and control on how volumes get mounted. So if that ever happens, then we'll definitely adopt it. It continues to support Docker plugins, but we'll also support this new standard. And finally, some resources. We definitely want you to try this, like they say. The proof of the pudding is in the eating. So we really want you to try it and let us know what you think of it. There's contact information on the bottom and once again, we're going to share these slides. So here's a pile of resources you can go to if you want to get more information on any of this stuff. And I don't know if we actually have time for questions because I see people kind of swarming in the back of the room. Yeah, all right. So yeah, definitely come. I'll be over there. Come talk to me if you need anything.