 The time is 4.50, the microphone is on. I think it's time to get started. So hello, everybody. Great to have you here. My name is Andreas Müller from SAP. And together is my colleague Silvestre Savalla. We are just going to present how we implemented a volume service at SAP and finally got it up and running, deployed and up and running in our SAP Cloud Platform. So as you all know, it's one of the last sessions on the last day of this year's Cloud Foundry Summit. As expected, the room is not fully crowded, but that's OK. As you might also know, one reason would also maybe be that just right now in parallel, there's also talk going on from the inventors of the volume services. So I hope I didn't tell any secret now. Don't leave the room. It's a good decision to be here. Hope you agree at the end of the talk. So the volume services have been around for quite some while now. So to be more precise, they were just basically introduced and announced last year on this Cloud Foundry Summit, just by the team that I just mentioned, so the Dell EMC Group, the persistency team. And just during that one year from last year to now, we at SAP noticed that there was internally quite some huge interest and demand for having an implementation of such a file system. For the reasons, maybe we'll talk about the reasons for the slides. But basically, let's get started with what you can expect in the next 30 hours from us to see. So in the first step, we will just describe what our volume services and especially why would you, as an application, want or need to use them at all. And then we step down and drill down a little bit more into the architecture, how they're really implemented on different levels of detail. And then to finally conclude with sharing the lessons learned while implementing such kind of a service. So basically, when we noticed that demand was in SAP, that we would need to provide such an implementation of a volume service. We just went ahead and kicked off a project and started implementing it. And while doing so, we thought it would have been quite handy if there was some detailed documentation or kind of a developer guide of doing something like this. And basically, unfortunately, there is none. So of course, you have a quite detailed and comprehensive documentation about the ProCy API. That's really great, but there's just no details about the volume services yet, at least. So the idea of this whole talk here is now to get anybody who also wants to jump into that experiment and really bring up such a volume service implementation just gets anybody kind of jump-started to really know what to do. And even though if you are not interested in implementing such a service on your own, it's of course still interesting what's really going on behind the scenes in case of running volume services. So as I said before, let's first have a look at what these volume services are all about in general. So since we're on a cloud conference and everybody's talking about cloud-native apps and so on, of course, I couldn't expect that everybody here knows what the trial factor app manifesto is. And let's just quickly recap what it's telling us about the app processes. So trial factor processes are stateless and share nothing. Any data that needs to persist must be stored in a state-food backing service, typically a database. So at that point in time when this was written, there was a typically a database. As you all know, they're all a kind of backing services around right now that you can use. Doesn't have to be really a database. You can also have a blob stores and so on. And with the introduction of the volume services, you can now just finally also just have the file system mounted into your Diego Cell container that you can also use to persist stuff. So now you might have the question, so why the heck I would need that? I mean, I have a local scratch disk anyways attached to any VM, so I can use that too. But there are of course differences and they're really important. So the most important difference being that with volume services, you really have shared instance of some file system that you can use for multiple instances of your application for the instance scaling that you obviously don't have with your scratch space. And of course you could also have all kind of specific requirements that you just don't get from the just base basic disk that you have on your VM. I think we should give some examples for that. So if your VM just gets restarted, depending on you usually get your disk back that you had before just talking about the scratch space that's attached to your VM. But if the error is that bad that your VM is even completely recreated maybe, then it depends on the infrastructure layer that you're working with. If you get your disk back or get a complete new disk even, so then your stuff that you stored there locally is just gone. So what we see, the only thing that using just the plain scratch space could be good for is just doing some cache rates, not relevant if the data is lost. So since we now know what in general volume service is, next step is to ask why would you want to use that as an application? And we see two main scenarios for that. So being SAP we're a huge company obviously and for any other huge company, definitely we are sure you have just plenty of legacy applications around in the installed base. And even so at SAP every application is asked to move into the cloud as quickly as possible that's obvious. It wouldn't be really too much to think that can just happen overnight. I mean there are a lot of talks here on the conference too about how you transition from an old school application to release these new cloud-first native applications. So it just takes time and resources and so on. And also these legacy applications quite often have some frameworks like PHP, WordPress, whatever in place which just basically expects to have a POSIX file system in place. If it's not there it won't work. So you will really need to re-architect and rebuild the application completely. And now this volume service come in quite handy just to solve exactly this problem because you just need the time to really rebuild everything even if you plan to do that. And for the transition phase you can just use the volume service, attach a disk and have the application run in the cloud. Even following the trifecta regarding processes as we just mentioned before because now you have a backing service which is just a file system. And regarding the last point here that was just the area of having another use case of using file system service is you have all kind of limitations with your physical scratch disk attached to your VM regarding size, throughput and you name it and you even would like maybe to attach very specialized hardware which has higher throughput or whatever. And we can also give two quick examples. So for example the run times in cloud foundry could have the demand in case of out of memory crashes to write a memory dump just to the disk. And what it would do today is just to write on the scratch disk as I described but then it's not safe that you can pick it up because it could be gone. And also right now if you have really big machines with huge memory, these memory dumps can be big of course and maybe you don't want to store not only the last one but plenty of them. And with that you can easily get into the size limit of your local disk. Another use case, so if we're talking about really new newly built really cloud native applications you could imagine you want to build an image processing like satellite image processing application. And obviously you have then a huge amount of images that are all of big size and yeah in that case it's a perfect fit really to use such a file service implementation even with maybe specialized hardware which has high throughput and so on. Okay, so on this slide we will look just into how really the volume service exposed to the end user kind of. So we have two end user perspectives. One is the developer or then a deployer kind of or operator of the cloud foundry system who wants to create these services by applications. The services just created to the applications and so on or also query the marketplace. And in that regard the volume services don't behave any different to a just the standard as the usual services because it's also just following the service provider interface. So the main difference that you can see is so when the application is started that has such a service bound in the standard case of services you would just get the credentials that you use to connect to your application in the credential section of the VCAP services properties that you can add into your application as an environment variable. So this stays the same but for the volume services you have a special volume mount section in the JSON and the most important information in there being the container deal. So that's really then what you as the application can expect to be filled and it just points to some local directory where you can write to. And you can just expect the remote shared directory of whatever technology to be available in your local space. And what we have in there on the volume mount section in addition is some kind of write mode. So it could be read, write or just read. So as an application you should make sure to really check that and if the mounted shares read only just don't try to write because it will give you an error anyways. So it's just a hint kind of. And the other important and interesting information is the device type that's just shared and shared is the only option that you have around right now. We also asked the colleagues from the consistency team and there are at least currently no plans to also allow for a single attached volumes to single instances of your application. So that there could be use cases like for if you wanted to attach block devices because for block devices it just doesn't make sense if multiple instances just try to write at the same time. Okay, so now we will as I said in the beginning just deep dive a little bit more into the architecture how this all looks like. So we start on the upper left side with the deployer actor who basically as we said before interacts with the volume service proger as it would interact with any service proger at all. So it just would query the marketplace to see what plans for example are available for this concrete instance of the service proger and then decide for which plan for example it wants to create a service instance. So of course it just you use the CLI interface to connect your direct your call against the cloud controller and the cloud controller just transforms your request into the proger API call which is the provision call. And now the question is if this provisioning call hits our volume service proger implementation so what does it need to do? And the answer is it depends. I mean it's all a quite generic concept and we are displaying it now how we implemented it for NFS share provisioning but the question is still how is the NFS server is coming into the landscape? So there are different approaches. So it could be that you have an NFS server and your landscape around anyways. It doesn't need to be even part of cloud foundry. It could be just outside you had it before in your company and you just want to connect to existing shares would be one option. But we did it differently. So what we do we really also manage the NFS server as part of our installation and we bring it with us and take care about backup and all that stuff. So it's really managed. And in that case if you want to provision a new instance we just create a new directory on the NFS server right away. And in addition that's just important for the quota handling I'll explain or the rest of us going to explain that in even more detail we need a user attached also together with that share and both correlated or assigned to the concrete service instance squid. So and this information is what we store as the volume service broker within our own persistency. And then we just return back kind of an okay to the cloud controller. At that point of time the cloud controller doesn't need to know anything specific about that created instance except for that it was successful. Then as you know if you've used any service before this next step you just want to bind these newly created service instances to any application. And we set bind call now. Our service program implementation just basically returns the VCAP service properties that we just saw on the slide before back to the cloud controller with all the information about the share where it's located. And now the cloud controller just persists that information. So in the standard case of services at that point of time the cloud controller could pass on this environment of the VCAP services directly to the application but the application only gets to know that if it's restarted because then it's the environment variables are injected into the application. But now we get into the specifics of the volume services really because it wouldn't make sense to just start up the application because there's no share yet. I mean you get the location mentioned in your VCAP services but it's of no use if it's not really available in my container. And for that to just get really the volumes mounted we rely on the so-called volume manager component which is part of Cloud Foundry. So it comes with Diego if you switch it on. I mean you have the whole feature of volume services has to be switched on for the whole landscape. And with that you have the volume manager component within the Diego VM cell. And this volume manager is just able by means of the Docker volume plugin API to talk with several instances of different even multiple different volume drivers. And they have the job of doing the actual mounting. And the volume drivers make themselves known to the volume manager by basically just putting some descriptor file with their own name and so on in a certain directory which is constantly pulled by the volume manager. So the volume manager just knows which drivers are available on which cell. Could be even different drivers for as I said before different specific hardware and so on. They could be distributed on different cells but usually you would have I guess all the drivers and all the cells available. And basically Diego can also make use of that information for its placement decision. So if you then later on you were just in that step you said you wanted to bind a specific service to an application it can decide on which concrete Diego VM this application should be running because only maybe on a few of them the concrete driver that you need would be available. So now it's a really important step is that the driver itself really does the mounting and based on the recap service information that's passed on from the cloud controller to Diego. And in the second step these shares also bind mounted into the individual cell containers. So this is really important now also from a security perspective because these multiple cells could be related to different applications and of course they are not allowed to see the different shares of the different other applications. And this is just insured because the only component which is able to see the shares is really or even can talk to the NFS server is the Diego cell VM itself and not the individual cells. They really just bind mounted from outside into the container. And there's the second aspect of security that's really important and the last really go into the details too in a few minutes, which is if the space scope ProCas are enabled also switch in the landscape anybody could just build a malicious ProCa also using file systems, the volume service API kind of. And if you get somehow whatever reason you get hold to the volume mount section of another instance, you could just inject it and bind an existing share to a different application. And we have some means to mitigate that situation and I said the rest will show how that works later on. So only component I didn't talk about yet is the Fuse NFS ProC here. Before on the left hand side when we said we provision the concrete directory on the NFS server I also said we also create a user with that. So the only reason to do so is because we want to make use of of using the user ID to define the quotas on the file system was in the NFS server. And typically, basically all the cells and the applications that are running within are run as a specific user which is the container vcap user which has a fixed ID and they all have the same user ID. And with that you wouldn't just be able to really define different quotas for the different service instances. And for that we use this file system in user space to make the kernel do the switch for you to concrete users that we created on the NFS server. But the server also go into the details about that step. So just before we move on during development and other cases if you really get the application up and running you need to most likely migrate data into your newly created volume services instances. And what you obviously always can do is just use the CF SSH. So we are in your cell and you can go to the directory and look at it. But it's a bit inconvenient if you really want to interact with that share and bring data on it or copy something down. And just be aware that you can first of all use any SSH client as you like to just connect to the cells, that's obvious. And you can even use SFTP to connect that share of the cell to your local system, at least for your development use cases. And with that you can just interact with the share really conveniently. Actually what came into our mind is also that it would be a good idea maybe to even include such functionality into the CLI of CF as a plugin. Because once it's your share and you connect to the application just as you can fuse the environment of the application why shouldn't you be able to just look into and browse into the files of any connected volume service instances and maybe even copy files down from their uploads. And would be a nice thing we think. So with that I hand over now to the investor to continue with even more details. Yes. Okay, basically when we started at SAP to look into the file system service into volume services, we looked at what's available as open source. And if you look for example at the NFS volume release that is in the GitHub Cloud Foundry repositories you will see that there are quite some brokers and drivers available not only for NFS but also for CFFS, for EFS and whatever and they are all very nice. And for example the NFS volume release has a very different use case in mind that might be your use case and which means that you can use this that release basically unchanged. And their use case is basically the idea that you install your Cloud Foundry system or completely on a inside your organization inside your IT and that it runs on machines that are connected to the rest of your systems and that are for example connected to an already existing filer that you want to access. And if that's the case, then the NFS broker that's already in the open source version is perfectly fine. You can use it and you can configure it and say just say okay access that NFS server and mount that share. And then if that's okay, that's a good use case. But we as SAP Cloud Platform, we are offering a public platform as a service. And of course we are not running our Cloud Foundry in our internal systems right besides our filers and our filers probably don't contain your data and shouldn't contain your data. So what we did is provision an NFS server onto the infrastructure as a service that we use completely independent of if you're using our public offering which is running on AWS or you're using for example our Minesphere that the Siemens Minesphere offering that is running on OpenStack or our beta versions that are running on Azure or our upcoming version is running on Google Cloud Platform. So that's completely generic. We deploy an NFS server onto whatever landscape is we are running on. And we provision and manage shares on that NFS server. So our implementation has to consist of a custom broker and a slightly modified NFS v3 driver from upstream Cloud Foundry. If you're building a broker just quickly aside, there are many libraries around for basically every popular programming language. We built our broker using the Go language and pivoted as a nice library in their repository the broker API library. So what has our broker, what does it have to do? That's not a completely normal service broker that you see all the time. It basically has in the catalog and the service broker catalog to require the volume amount permission. That's two things. One, it tells Cloud Foundry when Cloud Foundry sees it and the volume service support is not switched on globally for Cloud Foundry, then Cloud Foundry will just tell you and know this won't work. And on the other hand, if you try to create a volume amount binding without requiring that permission in your catalog, then it won't work either and Cloud Foundry will tell you, no, this is not allowed. You should have asked for permission before. More from a functionality perspective, when the provision call comes down, basically creating a service instance. I'm using a little bit API names that are in this broker API. When the provision call comes, then you have to create this share. When a bind call comes, you have to return that volume amount structure that Andreas already mentioned. And to go a little bit into detail, basically if you're a user of the file system service and you say create service, then the Cloud Controller will basically call provision on our broker. We create a share, we create a user which owns that share. We set a quota on that user and get basically for free every whole quota handling. When we do this, we have got two pieces of information that we have to persist. One is the password, the quota where the share has been created and the user that was created with it. And when the bind service, when you bind the service to your application, a broker is again asked and has to return this volume amount structure. Now, this volume amount structure is quite central to the functioning of volume services because it's the main way the broker emits information to three different recipients basically. One is Diego or more specifically the Volman structure, the Volman part of Diego, which driver to use? It's just a name in the name which the driver itself identified itself to Diego. It communicates to the driver how the mounting should actually be performed and from where to mount with what options to mount and whatever you can imagine. And to the application, of course, to tell it where the container will end up and that's the volume amount structure as it looks inside this library that I mentioned. Of course, it's equivalent to a JSON structure and to just go quickly through it. The first is the driver is just a string, a name that identified container there we already know about. Of course, read only read write or we also know device type is again this funny thing that is a string that's only allowed to be shared. In the future, there is interest in doing some kind of not shared instance-specific kind of volume mount, which would allow, for example, if you, for stateful containers, this concept that already exists in Kubernetes, for example, would then be also available for Docker containers running on Diego, which you can do right now, but you can't do really stateful containers. You can somehow fake it now with volume brokers in the sense that you can carefully handle the content of this volume mount, but instance-specific amounts would then allow for real stateful containers, but that's just a sort and there are no plans to implement it right now. But if we have a shared device type, then we have to also provide it with an ID, as you see on the bottom, and with the mount config, which is just a map and is completely dependent on the driver, which what has to go into that. As I said, we used the CloudToundry NFS v3 driver that we selected just by setting that name. It gets the volume mount structure, the most important part being a source parameter, which is basically just a URL that tells the fuse NFS. If you look at the fuse NFS as the description, you will see that it gets a URL for the mount. It's basically just encoded to server, which should be contacted to share that should be mounted and which use ID it should use to talk to the server. And let's have a quick look how the driver actually works. If you start an application, which can of course be of multiple reasons, you actually started it. The application has been restarted, you scaled it up, whatever. And Diego will create a mount, we'll call it create endpoint on the NFS v3 driver. This is the Docker volume API basically that's reused here. The persistence team is looking into defining a new kind of the API for the drivers right now. It's a reuse, maybe it's a little bit idiosyncratic, how it works. It then calls mount on it. And when this mount is done, it basically starts just the fuse NFS process, which actually mounts this NFS share. And then it starts the app, Diego starts the app. The app is of course running under one user ID of the Compactainer vCAP user. And that's the same for every application running on this cell. And when it writes to the file system, the fuse NFS driver then performs this translation into the user that's actually on the NFS server. And that's why we use the fuse NFS driver because it can provide for this translation. Right, so security, as everything just goes through this volume on structure, you have to think about if you need to authenticate this volume on structure. If you have enabled a space called brokers, every user or a space developer can actually create a broker. This broker can then emit a volume on structure and just select a driver. Now it depends a little bit, but if your users know that you're using the upstream driver or one upstream driver, then they can just look up how to configure that driver. And then maybe they can even guess something and maybe they see the original mount that you're using and then they can guess or try to lob off sub directories and try if they can get higher up in the shared directory and look at other data basically. So what we did is we actually added a callback from the driver to the broker where we asked, is this a valid mount request? Should I actually mount it? And we actually added a little bit more of an optimization where we actually validate a signature to make it a little bit faster. But that's an important thing if you implement it for your own public offering. Lessons learned, just to be quick because we're a little bit over time and I don't want to keep you from going home. Built it and they will come. We started actually with just one application request that they say we want this and okay we implement it and then immediately other people started going coming to us and saying yeah, finally the Cloud Foundry is usable, finally we can do a lift and shift, finally we can turn basically every legacy application into a 12 factors app. And that's something that you will see if you implement it on your Cloud Foundry that it is an enabler for such a strategy. So there's much demand, especially in companies where you already have applications that are not easily ported and where such a volume service can add a little step so that the whole migration goes faster. But on the other hand, your volume service has to be rock stable because if your normal service binding is not available, your application may start and it might try to contact that service and if your application is well written, it might even expect the service not to be there and handle it gracefully. A normal application, especially a legacy application will not expect your file system not to be there. And even worse, if you are on starting trying to mount the share can't mount the share and tell Diego that no, this mounting didn't succeed in the driver, then Diego will say, okay, then I can't start this application. So to not introduce a source of instability, your server has of course to be very stable. So it doesn't go down. So you need some kind of solution to actually to actually be stable. Fortunately, there are solutions and you have to find the right one for you. Last slide, you may be seeing it often. As you know, the vision of SAP is making the world run better and improving lives. One way to improve your life is maybe to start working as a SAP. Thanks. Are there any questions? Then? Yeah. If I got that wrong, with the volume services, you have one volume which is shared across all the instances of your application. Is that correct? No, I mean, if you create a service instance, you create a new share. On this new share is then mounted into all applications that are bound to that service instance. As you can create multiple service instances and they are all separate, completely separate. Right. I mean, underlying, usually you mean. I mean, every instance of your application. Right, it's the same. Connecting to the same share. Exactly, that's the idea. Yeah. Okay. Have a nice day. Thank you. Bye.