 My name is Julian Yorzoe. I work for the Dell EMC Cloud Platform team. The West Coast bit of our team is core Cloud Foundry contributors. We constitute the Diego Persistence Team, which is building volume services into Cloud Foundry. And that's what we're here to talk to you about today. This is Paul, my colleague, and for starters, I want to say thank you for staying. This is the tail end of the conference, and I know a lot of people have flights. So a little bit of what are we going to talk about today? Diego Persistence, volume services, file systems as a service, what is it? How does it work? Mainly, we'll be focusing on what's new since the last time we gave a talk, which was CF Summit in Frankfurt some six months ago. And then we'll talk a little bit about the stuff that was in there before. I'll kind of brush through that, because once again, we have quite a bit of content. And then we'll highlight spring content, which Paul is the brilliant inventor of, go into a demo, and then we'll give you some clues about what's coming next. And if you want to do your own thing, how you would go about building your own volume service, maybe you're interested in that. Or maybe if you were interested in that, in the other volume services talk, which is inconveniently scheduled at exactly the same slot. But, and then just a big old page full of links, which you should ignore until this deck gets published on the various websites. And you can pick them off of there. And if we have time, we'll have some questions at the end. So what is volume services? What do we talk about when we mean volume services in Cloud Foundry? Basically, it means that we've added plumbing to Diego and the Cloud Controller that allows you to identify file system resources out there somehow. Maybe provision them, maybe just identify them, bind them to applications, and run them in Cloud Foundry, which will eventually, for your app running in a container, get delivered to you as a directory. It also means that we've got an initial set of implementations for things like NFS to allow you to identify an NFS share, bind it into your application, and use it like a directory in your Cloud Foundry application. And we've got various implementations and we'll get into that in more detail later. So how does this work? The architecture at a very high level, once again, we have a fair bit of content, might be sort of familiar with this very high level view of Cloud Foundry. You have a Cloud Controller, which you're interacting with using your CFCLI or App Manager or whatever you used to interact with Cloud Foundry. That talks through the BBS message board to Diego, which runs in a bunch of cell VMs. And that Diego has a rep, which is there representing the cell to the auction process. So when we place a workload, the rep will say, yes, I want that workload. It then coordinates with garden to create a container, put your bits in it, start it up. Adding volume services means we take to this architecture, we add some network volume somewhere, a service broker, which knows how to communicate with that, maybe to provision it, maybe merely to identify it. And then co-located on the cell, you have a volume driver. And that's a piece of software that runs in a process and knows how to mount things into the cell. And these things all sort of conspire to make a volume available to your application by first essentially acting as a service instance through the Cloud Foundry Service Broker API, which can provision things. So in the case of, say, our EFS broker or our ECS broker, when you ask it for a new service instance, it goes and provision some storage. And now you have a thing. You've got some new storage thing that you created. And then when you bind it, the service broker passes back a blob of JSON that tells the Cloud Controller, well, here's this information that the volume driver is going to need later. Cloud Controller squarles it away. And then when it starts up your workload, it takes that same bit of JSON, feeds it to the volume driver. The volume driver's responsibility is to mount the volume onto the cell VM. So we do some kind of mount, whatever that is. It might be a kernel NFS mount. It might be Cephuse. In the case of NFS, we actually use a Fuse Mouthing device to mount NFS volumes into the cell. And then the last step is that Garden gets fed this directory and says, here's one of the things I want you to put in the container. And Garden then bind mounts it into the container. And at that point, it's not really any different from any of the other resources that Garden is already putting in the container. Because your application bits, for example, need to get bound into the container so they can run. So we talk quite a bit about this. In Frankfurt, it's been around, kicking around in Cloud Foundry for the better part of a year. I think we started this project about 2 and 1 half years ago. So what's new since Frankfurt? We've made a bunch of enhancements. And we'll go through kind of a no particular order because I procrastinated on the slide deck and did it in the last minute. We now allow you to co-locate your driver onto the Diego cell, which is one of the key deployment things, especially if you're in open source Cloud Foundry that you'll have to contend with as a Bosch add-on, which, depending how familiar you are with Bosch, you may or may not know this means Bosch Runtime Config, which is essentially an additional bit of YAML that allows you to say, if you see a VM that has this thing running on it, also deploy this other thing. And it's a way of essentially identifying additional resources that should get put in places in your deployment without changing your main manifest. So this is, for us, tremendously convenient because we have these volume drivers that have to get put on Diego cells. And the way that we used to do this was to go fiddle with the Diego manifest and redeploy Diego. Now you can set a very short, very reasonable bit of YAML for your Runtime Config that identifies your drivers and then redeploy Diego and then automatically get deployed. We have much better validation and error handling than we did six months ago. If there was anything wrong with your mount, for some reason you maybe mistyped something or whatever. And the resource that you were trying to identify didn't actually exist, or who knows what. Cloud Foundry, when you started your app, would just say, your app crashed and give you no additional details. And your only recourse was to dig around in the Bosch logs, which, depending on your cloud, you may or may not have had any access to. So now we have much better, we give some sort of reasonable error message to let the user know that the app crashed because volumes couldn't be mounted. And CF logs have better and more information in them as well. This is a big one for us. PCF Dev, which you might know is kind of the easiest way to get started with Cloud Foundry if you're just mucking around on your laptop, now includes volume services and local volume release. Local volume release is our kind of toy volume services release. It doesn't really actually provision storage. It goes and makes a directory. But it looks a lot to your application as though it's a shared volume. And the context of PCF Dev acts just like a shared volume. So if you're developing an app that needs a file system, as part of its thing, you can develop that in the context of PCF Dev, interact with that shared volume through PCF Dev, even though it happens to be just a directory on your laptop. And when you push it to Cloud Foundry with real volume services, it'll act exactly the same way. You won't have to rewrite your app in any way. So that's an excellent step for us, for app developers who are wanting to develop offline or locally or whatever to create volume-enabled applications. NFS support for existing shares is basically means that any NFS server that supports NFS v3 can be connected to from Cloud Foundry. So again, I've mentioned a couple of times that the service broker may or may not provision storage. And this is sort of what I was talking about. This is the may not case. The service broker essentially takes the address of a share during the creation of a service and memoizes it basically into a service instance that represents that share. You bind that to your app with some additional parameters we'll talk about later. And then when you start your app up, it'll mount that share, add it to your app. And you can use it. We've seen people using it with Iceland, ECS, NetApp, the NFS kernel server, if you name it. So we have a bunch of people using this. And we also have a bunch of people writing their own service brokers, which again is exactly what the other talk at the same time slot is about, what a bummer. So POSIX UID mapping, this is as soon as we started thinking about NFS, and we started thinking about NFS v3 in particular in the context of Cloud Foundry in particular, we run into this problem. And that's that essentially, security in the context of NFS v3 basically means the NFS server keeps us a list of IP addresses or side arranges for servers that it feels like it trusts. And once it decided to trust that server, it takes RPC traffic from that server. And that RPC traffic has a UID. And using the NFS kernel client, that UID is whatever the UID was of the running process that was connecting. And in Cloud Foundry, especially if you're running build pack apps, that UID is always 2,000, which means that you've got an NFS server that trusts, because we don't know where your workload is going to be placed, all of the Diego cells in Cloud Foundry. And any app running on those Diego cells gets to say connects. And it's always coming across as 2,000. So you have no way of having heterogeneous data access control. Basically, if your NFS server trusts Cloud Foundry, it trusts all of Cloud Foundry. So we needed some way of cracking this problem and making sure that it was possible for people to say, well, this app is supposed to run as this user. And that's how it works. When I'm running it over here on my VM, I run as this service account. And I use LDAP to synchronize with my NFS server. And it's cool because I could see on my NFS server that all of my files are created by this user that I know. To make that work in Cloud Foundry, we needed some way to map that Cloud Foundry always 2,000 build pack user to some other specific user that maybe the app developer gets to specify to say, yeah, I don't want to be that guy. I want to be this guy, which is the same as my user, my old service account that I was running in my legacy app on my VM over here. We did that by digging up this fuse NFS thing, making some enhancements to it. So essentially, we're intercepting the traffic between the incoming file system request and the outgoing NFS RPC call. We make a mapping between that running user 2,000 and the user that's specified in service binding. So now when you bind a service, the volume service to your application, you can bind it with a configuration that says, hey, I want to be UID 2001. We'll map that going both directions so that by the time it gets the NFS server, it'll look just like you're running as a specific user. And then from there, you can use the same kinds of POSIX permissions and whatnot you had before. Now, that's great, but you still have the problem that an application developer can arbitrarily decide to be any UID they want, which kind of stinks, right? If I know that Paul is doing payroll and he's running as UID 4,000 and he's storing data about everybody's salary, I can just go pretend to be UID 4,000 and off I go, right? Super, I can go download everybody's salary from my little Cloud Foundry app. So maybe you could hope that I couldn't guess that it was supposed to be 4,000, but that's a pretty terrible security story. So we've added something like a poor man's version of security, we enabled LDAP connections from the volume driver to an LDAP server. And this gets us a little bit closer to the security story you had in your legacy app connecting from your VM, where essentially it was gonna be that trusted remote VM image that was gonna decide that it was okay for this application to be using this UID. We now enforce that in the volume driver. So essentially, rather than accepting a UID, we in this configuration require user credentials. So we'll make you put in the username of password for that service account. That gets tucked away in Cloud Foundry in the same place as everything else. Then when it comes time to mount the volume and stick it in the container, we go over to the LDAP server and we say, hey, are these valid credentials? If so, what's the UID associated with those things? And that finds it back to some UID, which the application developer may not even know. So is it perfect security? No, because NFS V3 is never gonna be perfect security, but it does get us something a step closer to security for NFS running under Cloud Foundry. So file provisioning support in ECS. This is basically ECS's Dell EMC Elastic Cloud Storage. Some folks using Cloud Foundry will already know this because they've been using it as an S3 object store or a Swift object store, whatever you like. A lot of people use it as their Bosch Blob store in various configurations, but it's also capable of being an NFS V3 file server. So we've picked up the service broker for ECS and added volume services enhancements to it. So it's now possible to provision a bucket in ECS that simultaneously an S3 bucket and also a file share. And if you use that option, then we will expose that ECS bucket as a file share to your Cloud Foundry application, which has some kind of interesting implications because it means that you can interact from one piece of code, maybe your legacy code, with a file system and then access the same data from a different Cloud Foundry application as an S3 store. So that's coming soon, I guess, would be the right thing to say. It's available already on GitHub, but we're busy polishing the rough edges off of it. So it should be kind of fully baked within the next couple of months. So we're also seeing volume services available in more Clouds. This is what we were kind of nowhere on this six months ago, but as I mentioned earlier, PCF Dev has had this for quite a while, shortly after Frankfurt just got added to PCF Dev. PCF itself has this, I've got the version numbers wrong here too. I'm an open source PM, so I'm not actually supposed to care about the version numbers of PCF, but this was beta in PCF 19 and GA in 110. We included the NFS volume release that I mentioned earlier. So if you have an NFS server and you're using PCF, as of 110, these things are already turned on, all you have to do is expose the service access or enable service access to the NFS broker and you'll be able to get up and running quickly. We're also seeing support for volume services emerging in other Clouds. I know Azure is working actively on this. If you don't see it in your CF marketplace, then bug your Cloud provider. So a few things that aren't new, but still maybe worth mentioning in case you weren't paying attention six months ago or didn't get to that show. Local volume release I mentioned earlier, this is essentially a toy thing that exposes a directory on the Diego cell as though it was something that had been mounted externally. It doesn't work in multi-node Diego environments because we're not actually using shared network storage, we're just using some directory on Diego. But if you're in PCF Dev or some other single node environment or you just want some code to look at as an example, as a fairly straightforward example of what you would need to do to make your own volume driver or volume service broker, this is a great starting point because it lacks the complexity of some of our other service brokers. CephFS Bosch release is Ceph. This was the very first volume service driver broker pair we created, but still works. Still maybe an interesting option if you're just looking for something lightweight that actually provision storage and you're Bosch deploying things yourself. And finally, EFS volume release is for AWS EC2 environments and this has a service broker that will actually go out and provision an Amazon elastic file system, make it available to your Cloud Foundry apps. Now I'm gonna hand the microphone off to Paul who's gonna tell you all about spring content and then take us on a little demo. Hi folks, yeah, my name is Paul Warren. I work alongside Julian on persistence, which is our day job. And over the course of this, you can see what's happening here. We've got the initial volume services in and we've kind of been going horizontally out so we've been stamping out new service brokers stamping out new Bosch releases that provide a service broker for this next volume service and possibly even providing the volume service itself like in the case of Ceph and our Dell EMC ECS. We also decided we'd go up the stack. So we've started creating spring libraries, spring content being the first one that make it ridiculously easy to use the volume services in your apps. So you can create, you could really easily create applications and services that handle rich content. So that's documents and files because some estimates vary, but 70 to 90% of all usable business information is actually locked away in documents in unstructured data, not in structured data. So if you're familiar with the spring ecosystem and spring data, this plugs into spring data and allows you to create content management applications. It's got an abstraction for storing content and it uses the same kind of interface first approach of spring data, which means that you can literally write one interface and associate content with a spring data entity. So all you need to do is write one Java interface and annotate it and you can store and retrieve the content and have a set of REST endpoints that allow you to access that from your front end mobile app or your Angular app or React or whatever. Because it's got an abstraction in there, we support different types of store. So S3, which would allow you to access Dell EMC ECS or AWS for example. We have a file system abstraction implementation which obviously allows you to interact with files on the file system. And if you push an app, as we will do in the demo, into Cloud Foundry and you bind an NFS volume in there and that's where you write your data and then you can achieve, you can achieve, you can have apps that will write to the file system and that will persist when you're running in a container system. We support Mongo, so we'll write data to Mongo's grid FS and obviously we support relational database through Blobs as well. It's sports spring boot. It also supports Apache Solar and elastic file systems, elastic searches to come. So when you start storing your content, we'll index that content for you and then we'll allow you to search in it. And again, all we have to do is implement one interface and you get the implementation for free. And we also have an extensible rendition service. So if you've got content in a Word document or an Excel spreadsheet or a slide deck and you need it as a PDF, we'll allow you to build applications that will do that really easily. Okay, so let's have a demo. So, whoop, how do I play this? Do I just click go? Yeah, there's gotta be like a little play button somewhere, isn't there? Yeah. No? No. How, good grief. There's a play card. There you go. Yeah, there we are. So what you'll see here is a Jay Hipster generated application. So it's a spring boot application that uses spring data and spring content and we called it Springflex. Think Netflix. So we're storing a video here and this is just running on the local file system. You can see we're selecting a video and we're adding it and then we can play the video. And this, like I said, this is literally just created by implementing one interface. That's it. And so what we wanna do is push that to Cloud Foundry and eventually we're gonna end up binding this app to a Dell EMC ECS volume and therefore allowing us to persist that data over restarts and allowing us to scale the application and all that good stuff that you would expect to do when deploying 12 factor apps to Cloud Foundry. So we're pushing it here at the moment, right? It's not bound to Dell EMC ECS volume. So you'll see that it's gonna break when we try and restart the app. So we're just going in for the first time here and we'll create the same video just like we did on the, when we were running it locally on our laptop just now. We create the video. My typing isn't very quick. We'll select the same videos we did before and save it and then you'll see we can go in and access it, all good stuff and spring content by the way supports byte range. So you can do seeking through the video and so on and so forth. But what you'll see next is that we're going to restart that app. And because this was just writing the video file to the containers file system, that's ephemeral, right? So when we restart the application, the metadata about the video is still gonna be there because we put that in a database, but the video file itself we actually wrote to the containers file system and so that's gonna be now gone in this new instance of the application. And so the video doesn't work, right? So how do we fix this? Well, because we've got volume services now and we've got the ECS Bosch release, we're going to create an instance of a ECS bucket and then we're gonna bind, we're gonna expose that as an NFS volume. So you can see we're just having a look at the, what brokers we've got here. And in the marketplace here, you can see we've got the ECS buckets available to us. So we're going to create an instance of that and we're gonna ask that it supports NFS. So you'll see we're creating an instance of the ECS bucket and we're gonna choose the five gigabytes NFS plan and we're going to call it Springflex volume because we know we're gonna bind it to our Springflex app and then we're going to then bind that to the actual Springflex application. So you'll see if CFBS Springflex, so bind Springflex to my Springflex volume. Now when we restart the app, we're gonna create another video. You'll see the old video there because again the metadata is stored in the database. Still gonna be broken because we wrote that video out to the ephemeral file system which is long gone. But now when we create new videos, when we store them they're going to be being stored in the ECS bucket which is bound into the app container. And you'll see when we restart, everything's gonna work and we can scale the app and the videos are gonna be there and life will be good. So effectively what we've done is we've extended through volume services, we've extended the 12 factors to include file services. So I think we're just restarting here after having bound our volume and the environment variable we're setting there is when we bound our volume, the volume will appear in the container at some part of our VCAP data in this case. And so we just have to tell Spring content that when you store content we want you to put it in that VAR VCAP data location. So we're just creating our new video here and we expect it to work. You kinda get a hint that it's going somewhere else so you see the progress bar along the top. It's taking a little longer than it did before and that's because obviously we're stringing the content all the way to a remote ECS server and the video place. So we're now gonna restart and if you remember before, this then failed when we went back in. You can see we have a really quick instance here. There we go, we restarted already. So we go back in, go to that working one, nine was the broken one. You can see the video's still there. So thus proving that we actually wrote that video content out to an external system. Once we restarted the app and it got restaged into a new application container, we rebound that volume to that new application container and then the application could find the video. And then lastly, that also means that we can then scale that app to a greater number of instances, four in this case, and it doesn't matter which instance your users hit, it's gonna work, okay? So that's a nice demo. Encourage you guys to actively go look at Spring content, obviously volume services and Dell EMC ECS which is I'd say available in a Bosch release right now. We're just ironing out the kinks a little bit. But please give it a go because a community addition is included in that. So you have everything you need to interact with volume services in one Bosch release. Thanks. I got a little bit more content for you. So just a quick note on what's coming soon for us or soonish in the case of depending on priority and definitely if you have opinions about what should come sooner rather than later, I would tell you to like stop by the booth or come to our office hours, but you know, it's the end of the conference. So find us on Slack. We've been working with Kubernetes Docker and Mesosphere on something called the container storage interface. This is a common plugin interface that should allow storage providers to create plugins that will work with any container orchestration system to mount storage into containers. In the past, we used the Docker plugin API in the absence of any spec. This is the spec we were missing two years ago when we started. So as this emerges, we'll be working diligently to try to implement the container orchestrator side of it into Cloud Foundry so that we can consume these plugins and also, you know, working with the other container orchestrators to build out new plugins. So we're quite excited about this. Various possible support enhancements for NFS, the most glaring one that we don't support today is NFS4. And again, that's to do with the UID mapping and identity mapping things that we do in Fuse NFS, which itself doesn't support NFS4. So locking support, another big one. So most legacy code that uses NFS3 isn't gonna probably be using file locks, but it's important to note that it's missing from our Fuse NFS driver. So file locks will seem to lock files and lock them in the context of a single Diego cell. But fail to lock them across the NFS RPC. So that's something that we'll be looking to address in coming months. So the only takeaway is if you're relying on file locks and think about maybe an alternate implementation in the near term. And service brokers support for provisioning. We've already seen this for some, you know, people writing their own service brokers and we have service brokers that, for example, in ECS provision storage, but, you know, look to the future for additional service brokers for other kinds of storage. Because I think a lot of, you know, in legacy use cases, it definitely makes sense to connect to existing shares and access existing data. But for a lot of cloud foundry environments, people wanna be able to use a service broker to go spin up a new fresh volume that they can use for their application. Plug-in support for lots of other shared volume protocols. Again, if you have a burning desire to see any of these implemented, please let us know. That'll help us to kind of prioritize what makes sense to do first. Single attach storage is something that we've had a fairly heated debate over for quite a while. This, in many cases, comes down to a debate about whether or not it makes sense for things like databases to be run as workloads inside containers in an environment like cloud foundry. The prevailing opinion had been, no, this doesn't make any sense at all. That seems to be softening. So as we look to maybe implement that, we'll be looking for your use cases about what you would like to see and what kinds of workloads you would like to run in cloud foundry that you can't today because we don't support things like block storage attached to a container. Windows support, another big one we've been hearing buzz about. We're having garden windows and windows workloads support. We do not yet support volume services in that context. That is somewhat dependent on the garden windows team to pick up the latest version of the windows operating system that they run in that context. And the cell or Windows Server 2016, I think is what they're running. Once that happens, it'll become possible for us to implement the volume services bits that would then allow us to write volume plugins run in a Windows context and do mounts to Windows. Just a quick set of resources. This is all maybe almost too small for you to read. But there's a bag of links here to most of our documentation, most of our Bosch releases, spring content that Paul mentioned earlier. The spring flicks demo that we just showed you in case you kind of want to see how the code works. And so this once again will be available after the conference in the slide deck. And if you want to build your own thing, there's a few more links to the various specs that we follow for the volume plugin piece and the service broker piece. And some nice starting points. I'm going to breeze right through this too and get on to questions and we all have questions. Maybe you have some questions about not these things but cloud foundry type stuff. So, questions? Yeah. Existing NFS with three. A single volume or multiple volumes combined? You can mount as many volumes into a container as you like. Essentially these all get delivered to your container as they have to be mounted into the container in different directories but they'll just end up being mounts. So cloud foundry also provides this VCAP services environment variable which is the same thing that you would look in to get your database credentials or whatever. That should contain volume mounts section for every service instance that's bound in. So in the case of mounting multiple shares into the same container or binding them all to the same app, you just get multiple services, each one having a little section for its volume mount. Sure. Have you ever had a capacity in the first few box situations when a container is scaled up or scaled down? You've got some strange issues coming up. So, meaning that you have concurrent access to the file. Not access basically, you have an application running with the first one and then try to consume the same value to the second app. Oh yeah, so we did have some bugs to be sure around the preservation of our mounts on the Diego cell in the case of multiple locks. So yes, we've addressed those. So if you're- You can see that kind of locks in the traditional NFS because you have a big parameter like new space for NFS, like version three and then zero zero write C and R or whatever, like read write many times. Option is available in Kubernetes but option is not available in Cloud Foundry. So I think if you're running your own Cloud Foundry, all of those options can be- But why do you stop the NFS? Oh, no, we actually, if you get the latest bits, we take a hash of all of the various options that come in when you bind something to an application. If anything about the bind options is different from one app to another, we'll do a separate mount. So it's kind of depending how fresh your bits are because we definitely had some wrinkles in the couple months ago version of all of this stuff. So yeah, the standard developer caveat, right? Like, are you using the latest thing? No, well, we must have fixed that already. Anyone else? NFS two, no, no, we don't. But maybe we should consider it. I'm sure there's some apps out there since 1984. All right, well, if that's everything then, thanks very much.