 So, this is going to be OpenShift API data protection and using SAS CSI combined with OpenShift Container Storage. So, I'm an accluate, I'm in the storage business unit and been working with OpenShift and OpenShift Container Storage as well as ROOK and SAS CSI for the last three or four years. So, I know Dylan is on, Dylan, you want to introduce yourself quickly? Yeah, sure. Thanks for that. My name is Dylan Murray. I actually come from the OpenShift Migration Engineering team and we learned a lot doing the migration work using Valero and we wanted to take that knowledge and apply it to what we're calling the OADP and essentially our backup restore initiative at Red Hat. Thanks. We also have another folk, Ragu, imagine out that I wanted to call out because he also comes from the OpenShift storage side and has been doing a lot of work helping us make sure that the OADP project aligns nicely with the CSI initiative and all the other work coming from the CS. So, I'm talking a little bit about the OpenShift backup restore plan. So, the key goals here is that we want to provide application backups that are granular and can cover not just the metadata of the resources in the cluster, but also the application's persistent data. So, that would be backing up persistent volumes on the cluster as well. And we also wanted to enable a wide range of backup solutions with different backup ISV partnerships. So, our strategy here was to provide backup APIs that are built on open standards and can be extended and used by additional backup applications. So, the solution really contains OpenShift, Valero, RookSef, and CSI option projects. OpenShift container storage incorporates RookSef with CSI and we have a community operator OADP which enables the Valero APIs. So, it allows you to install Valero easily, configured with OpenShift, and then exposes the Valero backup. So, what does a traditional backup restore workflow look like? Before we get too deep into the solution, let's talk about what's the problem here. So, when you want to manage backups and orchestrate them, you kind of have three sections here all using some type of backup target to save the state of the previous backups that you can restore from in a future point. So, starting on the left here, you always have an application definition. And some people simplify this to just be the pods and the persistent volumes, but that's not totally true. You probably have a large set of Kubernetes resources that make up your full application, so we need you to define that. Then we want to extract the resources that make up the app. So, in this case, it's generally just some YAML files that represent Kubernetes resources in the cluster. It could be some internal images if you're using OpenShift, and you've enabled the internal registry. But then it could be persistent volume data if you have a stateful app. We need to take all of that up and then package it up into the backup target. So, this is someplace off cluster, preferably, that contains the state of the cluster at a given point. You can think of that if you use Valero in the past. This is the S3 bucket or some backup storage location where all of this data is going to get put into. Then let's say you have some disaster scenario and you need to restore from that backup target. We reference the backup, right, execute a restore, and that's going to go ahead and apply all the Kubernetes resources. So, backup time, you can almost think of this as like an OC export. Then at creation time, essentially an OC apply, obviously taking into account that we can strip some of the data from it. You're going to want to restore the internal images, right? So, let's say they're just container images, it's essentially like you're pushing an image to a backup target and then pulling an image from a backup target, get it back under the cluster. Then persistent volume data, right? So, restoring all the PVs in the state that the backup was taken. So, let's take that just a little bit more as we look at the solution. So, OADP itself, right, we kind of want to think, well, with that backup restore workflow, what do we need to provide to enable OpenShift users and partners to effectively take backups of the cluster? Well, OADP needs to provide cluster consistent backups, right? So, providing the ability to take all the resources in the cluster and the PV data and providing a stable backup while the application was running. We need to be portable, right? So, have a platform that's able to use plugins and extend the core backup platform to make sure that when we take an application running on one cluster and moving it somewhere else, even if it's in a different cloud storage provider, that the application works. Then finally, obviously providing a Red Hat ecosystem support, making sure that the backup API is actually speeding up the partner solution. And because the current state of OADP, we are a community operator, so we're available in an operator hub, easy to install, and obviously easy to provide new future version updates. So, with OpenShift backup in store and what the OADP project is solving, right, there's kind of three points here that OADP is going to help out in the effort. Defining the application itself, the backup API needs to be easy for you as the user or application developer to be able to label your app or supply a backup object that can effectively express what makes up the application of the cluster. Then there's the orchestration endpoint, right? So, the ability to actually enable an easy to use API for users to take backups of the clusters, then a set of controllers, right, that are watching for when we need to create a backup or when we need to create a restore. Maybe you're scheduling backups and we need to be able to orchestrate that effectively for you and quickly. And then finally, we can integrate the execution point as well, right? So, we want to be able to provide plugins that make this backup API OpenShift native, so providing backup of any app that's running on OpenShift and using OpenShift custom components, then obviously extending OpenShift container storage solution to have plugins that make it easy and painless for folks like backups of their application data. And then obviously, all that data is going to have to go into some backup targets. We want to make sure that we can effectively provide an easy way to integrate with any type of backup target. So, getting into, you know, how does the OADP API satisfy all these requirements? Well, we're using Valero, which is an excellent backup restore utility that is native to Kubernetes, who are also supplying a set of OpenShift backup and restore plugins, right? So, we're extending the Valero code using their native plugin system to make sure that if you take backups of OpenShift resources, we're doing all the custom logic required to make it to be able to restore on any cluster, right? So, really supplying the application portability stamp. So, we do this, right, from the cluster resource standpoint by the OpenShift plugins and a plugin to manage internal images as well as a set of vendor plugins that you can paddle with any backup under. Then, from the storage resource component, right, we're going to be using controllers to orchestrate these backup restore actions, and we want to make sure that we're integrating with native OpenShift. Right, my backup is the application. So, moving on a little bit, talking about the architecture, right, so since we depend heavily on Valero, I want to talk a little bit about the Valero API, so between this and the next slide. But I also wanted to briefly mention this dating mover concept. So, if you look at this from the top, you have some backup ISP project or product, right, that is providing policy, scheduling, governance, compliance, reporting, and they want to be able to integrate with some set of API on the cluster to provide the backup interface, right? So, Valero API is what we're doing to do this. It exposes at a high level really four major custom resources, but then another big one that we'll get into in a second. But they are a backup storage location, right? So, this is where is all the backup data going to be put, and then a volume snapshot location. So, if you hold the... Sorry, hearing echo, not sure this one. A volume snapshot location is the ability to supply Valero with an API that takes snapshots of your underlying. This as cloud storage volumes, so EBS, Azure GCP, and you want to use the native snapshot API that's on their platform. Valero allows a way to extend that. We also use that API to provide really good support for CSI volumes, as we'll get into it a little. And then the plugin interface itself, right? So, there's kind of obviously three ways to extend this. The ownership plugins, we've kind of talked about a little bit, but a good example of this is, say, a route. Say you have... You have a route that's supplying the application on cluster one, and the domain changes between the two clusters. You want to let OpenShift regenerate the route name. This is a very simple transformation, but we supply a plugin that does that automatically for you, allows the cluster to dynamically restore on the second side. Volume snapshot plugins, we just talked about, right? You have the CSI extension point and pretty much any cloud provider. Then you've got a backup storage location plugin, so the ability to support custom backup targets. We generally use S3 native solutions today, but Valero has support for Azure Blob, Google Cloud, vSphere, and people are writing their own custom plugins to support other providers, right? Even on a festival, you can write a plugin for, which would basically be file system targets. A lot of extendability with the Valero plugin interface. But then there's this other topic that is still in the design phase for ADP, but you'll notice that a lot of other backup partners and vendors have this concept with a data mover. Let's say that I do have an application with persistent volumes that have CSI drivers enabled, and if I take a CSI snapshot of a, for example, a self-cluster, my backup of that volume is still going to exist wherever the storage provider is. So if that storage provider is on cluster, there is a desire from your standpoint to actually move that snapshot and be consumed elsewhere. So we want to be able to provide a controller that can allow backup ISVs to plug in with the ability to move these CSI volumes wherever they like and restore them for many worlds. So I just want to briefly bring up the data mover. It's still an early design phase, so it's not really usable by anyone today, but it's definitely a core component of a full backup solution here if you're taking on cluster backups. So briefly I want to get into the Valero API, just kind of everything that exposes at a high level. We've got kind of five big ones here. I've already covered four of them. You've got the backup, right? So some way to describe a backup of an application in the cluster or a set of applications. You can see it allows for pretty granular specifics, right? So let's say you just wanted to define everything yourself. You can use a label selector and label every resource you want to include in a backup. And when you create this backup CR, it's going to go and grab everything with that label. Resources, and it's a persistent volume or an image stream. We're going to automatically backup the PV data and the internal image itself. You can maybe scope it to just a set of namespaces, right? And then we'll go on intelligently grab every namespace scope resource in those namespaces and then try to grab any cluster scope resources that may be referenced by those names. Then obviously storage location, whether or not you want cluster resources and all that stuff. But the one thing that I really want to call out here that we didn't want to talk too deeply in today, but I wanted to make sure you guys were aware is that the Valero API extends the backup ability to take hooks as well. So what that means is, because I kind of butcher that sentence, is at backup time, if you have an application that, say, is stateful and it needs to supply some custom action in order to have a crash-consistent backup of the app. So let's take a database, for example, like MySQL. You may want to run a command to sort the MySQL database safely and halt transactions while the backup is running. And then when the backup is complete, you want to bring the app into a functional state. So Valero does have an extensibility, what they call hooks, to be able to control that behavior of backup. A then restore is obviously the ability to restore from a backup and perhaps even take a granular set of resources from the backup. You got a backup storage location, which is where we're going to store the Kubernetes resource data, then optionally potentially some PV data, but probably just PV metadata, like CSI snapshot information, or cloud provider snapshot information. You got a volume snapshot location, so the ability to define where to take snapshots for PVs. Then you've got this fifth one, which I find a lot of people ask for as well, is a schedule resource. So the ability to schedule backups on a timely basis, Valero extends that API as well. That was kind of from the user perspective. We kind of talked already about the individual backup CR itself, but make sure you know you can set the hooks and you can also set the retention policy as well. The restore, kind of the other interesting thing I wanted to call out about the restore here, is generally restore is assuming you're going to the same cluster. We have the OpenShift ability with our OpenShift plugin to make sure that we restore pretty effectively into new clusters, but you have to be aware if you're restoring to new clusters, you have to make sure that plugins are handling that transformation for you. And you can also, you know, provide a namespace mapping. So let's say you had, you want to test a backup restore workflow, right? You don't want to have to restore to the same namespace. You could restore to a testing space, make sure everything's working as expected, and then you know your backups are safe. The backup stores location, I want to call out kind of what support exists today. So we have name support from existing supportive Valero plugins that we contribute to as well. You've got the S3 plugin. So if you want to use Amazon S3, Nuba, Minio, et cetera, the Zero Blob storage, Google Cloud storage. And then as I said, there are some community plugins that you can use for like file system backup targets, but they're not officially supported just yet. But I wanted to also call out that Valero does have the ability for you to use this solution called Rustic. And OADP does enable you to install Rustic as well with Valero. And what Rustic does is essentially for all of your PVs that don't have snapshot capabilities. So it's not a cloud provider PV. It's not a CSI-enabled persistent volume. Then you can always fall back to the solution that's called Rustic, which essentially is going to take a file system copy of the PV. And if you do choose to use Rustic, since it needs somewhere to put the data, we will store the Rustic PV data in the backup storage location. This would be your S3 bucket, wherever your object storage is configured. You will get Rustic PV data stored there as well. But for most other use cases, especially if you're using more upstream work, you're going to want to use CSI snapshots. And the object storage in this case would only contain the metadata, referencing the snapshot information. Okay, just quickly talking about the VSLs, so the volume snapshot location, this is kind of the interface that will allow us to be extended to support CSI snapshots. But you also know that you can pretty much write a plugin for any type of PV provider here. Out of the box, you've got Amazon EBS, Azure File System, and Google Cloud Storage. And obviously there is also the supported CSI driver that we're going to be demoing for this. So finally, the schedule, right? This is actually pretty straightforward. Most backup ideas have this, but this is the ability to essentially have a cron job-like syntax for specifying when you want backups to run. So you run this every three hours, every 24 hours, that kind of thing. If you want advanced scheduling, obviously there's a lot of backup vendors that have more fine-grained scheduling capabilities. You have to look there for a solution. If you want to deeper dive into this stuff, I've provided a few links, so if you want to look at the presentation, look it up. If you're looking to extend any of these with plugins, kind of have to dig in to figure out what exactly you need to support, but here's some links to different portions of the API if you want to click the motor. So with that, I will turn it over to Annette where she's going to talk about how CSI works in conjunction with OD. Actually, Dylan, we're going to turn it over to Ragu to join. Oh, thanks. Ragu is going to have you advance the slides for him. Ragu, would you briefly, quickly introduce yourself? Yeah, sure. Hi. My name is Raghavendra. I am part of OCS engineering team. I have been working on OCS for the last few years, and most recently, I have been working with Dylan and others on OADP and storage integration. Yeah, that is a quick introduction of me. And regarding OADP with OCS or storage, specifically what we are doing is using the Steph CSI in the storage area. So specifically, that's the one. So, generically speaking, if we have to understand CSI, so it is a plugin in the generic container storage interface in the Kubernetes. So CSI or container storage interface provides a proper interface for the application to consume storage in Kubernetes without worrying about what the underlying storage layer is or how the storage is implemented. So if you look at this diagram, we have Kubernetes services like API server, control manager, et cetera. Whereas if you look at the nodes, that is where we have things specific to CSI running, things like CSI provisioner that would ensure that like the CSI plugin parts are running. Means like in this context, there will be a plugin from Cep that would be used for integrating Cep with the CSI layer. And that particular plugin would be started by things like CSI provisioner. And like you can see there is a CSI plugin as well that would be the Cep CSI plugin specifically for our integration. In general, there can be different plugins provided by different storage vendors for CSI. Whereas in this particular case, the storage would be provided by Cep, hence the plugins would be Cep plugins. And like moving on to next, this is how generic Valero to CSI snapshot communication look like. So there is an entity in Valero as well called Valero's plugin for CSI. So that would be the one which will ensure that like it will be executing the CSI snapshots and the CSI snapshot would be there in the storage provider or the CSI provider which in this particular instant is Cep. And Valero will look like what it will do is like whenever a CSI snapshot has to be executed for a particular persistent volume, then it will look at the configured volume and it will also look at the snapshot plugins that have been configured. And for the particular persistent volume whose snapshot has to be taken, it will just identify the provisioner there and compare it with the provisioner or a driver that is there in the snapshotter and based on that it will take the appropriate CSI snapshots. So and hence like whenever Valero has to take a backup of an application that has persistent volumes, so the moment it has to take the snapshot of the PV or the persistent volume, it will identify the storage provider there based on which it will be able to identify the appropriate snapshot or plugin for that and execute that. So that like the CSI snapshot would be taken and will be stored in the storage cluster. And yeah, next if we go, this is how a typical backup request would look like in this particular setup where the application that makes a backup request would first make it through Valero's APS and the call would come down to the Valero CSI plugin which would identify the backup request and understand it like okay, this is the particular persistent volume whose snapshot has to be taken and like I mentioned before, it would identify the appropriate provisioner for that particular volume which in this case would be safe and based on that it would identify the CSI plugin and would issue a CSI snapshot to that and it results in and the resulting snapshot would be saved in the Chef cluster or the storage cluster. So that would be the generic flow with respect to creating a backup. However, like Dylan mentioned, there is an evolving component called Data Mover. So the Data Mover would be actually the one which would ensure that it copies out the snapshots to a different target location so that the data is actually or the snapshots are actually backed up in a different location. This is an evolving component which is still being worked on but this is the overall idea of a Data Mover would look like in the overall things of like how an application would take the backup. Yeah, now I'm handing it over to Annette. Hi, thank you. Yeah, thanks Dylan. So I'm going to do a few slides here and then we're going to go into a demo. So first is the OEDP operator and Dylan if you want to, yeah. So given in the demo I'm not going to install I just want to make it clear that if you deploy the community operator catalog OEDP is already in the community operators and that's actually what I'm going to use in the demo. So quite easy to deploy it. And next slide please. So you just basically install it and if you're in OpenShift 4.5 it will actually create the namespace for you and once you hit subscribe you basically have it installed. So I'm going to go ahead now since we're going to do the demo and I'm going to share my screen. So everybody see this? So now that I'm sharing my screen again I've already installed the OEDP operator and OpenShift container storage. So we're going to look at the application that we're going to use for this and it has two pods WordPress and MySQL and in addition to that this particular application has persistent storage supplied by OpenShift Container Storage and SUF and the type or the storage class that we use is from SUF RBD so these will be two block volumes that support this application. So basically the goal here is to see if we wanted to be able to back up and then restore this application. So the front end of the application is WordPress. I've already put in an article and just to test when we actually do the backup and restore I'll start a comment here and this is great and then we'll go back to looking into OEDP and how do we set this up. So as Dylan mentioned OEDP has a lot of APIs and the very first API that we need to create is the Valero API. So I'm going to create an instance of the Valero API and to do this I'm going to go to the YAML view and I have a YAML file here that's set up in the way that I want for my particular bucket and CSI setup. So I'll go through this, let's first copy it out and then we'll take a look at what's in it. Let's copy it in here. So a couple of things to point out here enable CSI plugin is true also down under Object Store so in this case the volume the location for storage is going to be an S3 bucket on AWS and I need to say what region I need to give it my secret which I create which has my creds for my bucket and the other thing is if you look down there the very bottom enable rustic is false but I do have my default plugins including CSI. So once I create this resource we can see that it's starting to run there so let's go back to the CLI take a look at the resources that are coming in to the OADP-Operator project so if we do just a watch on as it's being created here we see that there's a Valero pod coming in, there's also an OADP registry for images so we can see that actually it's already running so Valero is now running and we can also see if we take a look at it that we now are able to go to back to the CLI and let's take a look now at our WordPress project so once we have the Valero resource running we're ready to do a backup and restore as Dylan said that can be at the level of a namespace or resources within the namespace so in this case we're going to do it at the level of a namespace right now we have no volume snapshots they haven't been initiated yet and down in the bottom terminal you see there are some CSI volumes so what this is is I've done a I'm in the CF CLI and CF was installed via OpenShift Container Storage in this case and I have CSI volumes two of those volumes map to the persistent storage or the persistent volumes that I have in the WordPress project so as of now we haven't done any snapshots so we'll go back to OADP and we're going to create a backup resource so again I'll use the AML in this case I'm just going to add a few lines into the AML so I have the ability as Dylan showed to have included, so I'm going to include namespace and in this case my included namespace is just going to be WordPress you could envision if you had a lot of namespaces in your backup you would put the namespaces at that point to include more than just one so uniquely named the backup so if I want to look at the backup as a custom resource and because of that I can describe it in the OADP operator I can also see on the bottom in the CF that I now have a new snapshot so that's already happened as Ragu said the snapshot is initiated out of Laro, but the actual snapshot capability is the CF CSI snapshot so CF CSI supports snapshot capability and clone capability in OpenShift Container Storage 4.6 which is the next release as well as Rook Upstream version 1.4 is already supported backup and clone excuse me snapshot and clone so looking at my backup it's currently in progress and if I look now in my for the snapshot class so this is really the glue between being able to initiate a snapshot and get a snapshot into a particular storage so based on the provisioner of the PVCs and the PVCs if the snapshot class with the same provisioner then I'm able to snapshot into that storage type which in this case is CF so we now see down in CF we've now completed two snapshots and if we go back and describe our backup we should be able to see if it's completed yet and it has completed so at this point in terms of what we've done we know that we have the persistent data in CF and if we go now into the S3 bucket that I specified in the Valero resource we can look to see here under backup 1 here's the metadata and again this is not the actual persistent data but it is you'll notice there's a volume snapshots backup 1 volume snapshot that is the metadata for the PV and the PVC so it can be restored and then the actual contents of the volume are restored from the snapshots in CF CSI and just quick if we had data mover available for a particular vendor like Trilio or some other backup vendor then they could move those snapshots off platform and that would further protect your data so now that we have a backup the thing we want to do is test restore so we're going to go ahead and delete the WordPress project and it's deleting now if we go now back to our application given that we've deleted it the application is failing which would be expected if you delete the namespace so given that we now have deleted it we want to just make sure that it's fully gone so that when we do the restore we're doing it from basically you know with the namespace not existing and if we go back here we can see all the resources are gone but just to make sure the project has terminated let's go ahead and get the project it's not around and before we do the restore let's put a watch on all the resources that will be in this case we're using the same name namespace WordPress and what we'll see here as of now we have none because we just deleted them we'll also put a watch on the PVCs so to restore is similar to backup from OADP go back to OADP and choose the restore API and similar to backup we're going to do it from the YAML view and we're just going to add a few lines here the first is the name of the backup so this is why it's important that the backup have a unique name so you could go back to a particular point in time to restore from that point in time and then the other thing we want to further do is say what namespace do we want to include in our backup many namespaces that we had backed up we can now just say restore just this name space so once we create here then we can watch it it's in progress and actually it's already completed so we'll go back to our CLI and almost even before we can get back we can see all the resources are actually in a running state so we've used the snapshots for the persistent data to put back into the PVPVC resources the pods are all running so basically the application appears to be restored so now we want to test it and refresh on that and on my comments I actually had not finished my comments so I'll post my comment and you can see that it did work so we're up and running on our application so what we've done here today shown how OADP combined with SF and CSI via OpenShift Container Storage or XF upstream you can use the Valero APIs to initiate the CSI snapshots and using the backup and restore capability we can completely delete a namespace and then restore that namespace with the persistent data that was there before it was deleted so I think I'm done so I'll go ahead and stop sharing and you can bring up the slides again Dylan I guess that was the only slide it's always good to say thank you that's always a nice one I'm sorry I was on mute but real briefly I just want to bring up super quick we need to have a link here to the ODP operator github I just want to pull it up if you're interested in using this we do have instructions on installing it from operator hub outside of OLM we have the instructions for that as well so the read me here is a really good place to start if you want to try this out yourself and it's also an operator hub right Dylan you want to put that link into the chat sure thing then I don't think we have any more slides I'll have you pop over to one other place I'll go to Valero IO just so people can more background on Valero you ran through a whole lot of acronyms and other things as well here trying to get all the resources here you also mentioned RESTIC earlier on too we didn't want to talk too much about RESTIC because we wanted to focus heavily on the CSI use case but this is interesting to look at if you do not have a CSI enabled backup storage provider I'm sorry, storage provider itself good use cases of this is if you have Gluster say you have NFS volumes and you can't swing the volume definition over then RESTIC is a good fallback tool to use here there are instructions under the Valero docs use RESTIC itself you just need to set a specific annotation on the pod that's using it that's a good project check out as well I hadn't seen that one before that was a good one I don't see any questions in the chat there are 24 or 25 of you at any given moment are you doing awesome job demoing things or you've just shattered people's brains by facilitating awesome backups hopefully we'll go with the shattering of brains but I really think that one of the things that I always ask people is what's next what's the next set of announcements to this or where do you want people to come and give you feedback on this if they're using it and catching up with you thank you the ODP operator get hub is where we're tracking work in terms of enhancing the ODP API itself which really is Valero with some additions on top of it that data mover project is still in the design phase we don't have a timeline on when that would actually be usable by anybody yet but if you are interested in providing feedback on that probably talking in the ODP operator get hub would work we're also obviously in the core OS slack if that's a good place to chat with us but for anyone using the upstream effort if you have find bugs or issues or like enhancements to the ODP API in the operator itself please do go to that get hub link and open up some issues and I would mention as I showed basically the openshift container storage but really the components were from the RookSef version 1.4 upstream in terms of doing snapshot and clone using CFCSI so if you want to try it today you would need to be using RookSef we're looking to bring this downstream sometime around early November in the next version of openshift container storage and also integration is definitely going on with IBM Spectrum Protect Plus other backup vendors that are looking to integrate either the data mover or even just the CFCSI capability for snapshot and clone so they're building their own controllers, their own CRs to basically take the data and get it into their platform in addition to that we're looking at doing some work with Cloudpack for Data via IBM on their apps they're going to develop the hooks that Dylan showed how you could have a hook so before you do a backup you would have a pre-activity to freeze or to stop and then a post-activity to start again in any of these things I think the future development is really to get it into more integrated backup and restore solutions as well as possibly in the future even having one that's native to openshift I think is a possibility there is one question come in is asking if he can use data mover to copy the data from snapshot to another SAN or NAS that is a design goal of the data mover project yes absolutely so how existing data mover projects work right so for example Trillia has got an example of this others that I can't think of but the idea here would be that if you take a snapshot of a volume you may want to export it into some common format say like a QCAP2 image or something then you want to be able to take that more common format formatted file or state of that application itself and then convert it onto some other storage provider that is definitely use case of the data mover project yes couldn't you also Dylan take it mounted some off mounted somewhere and then copy the data out that way too actually how did a mover is intended to work so that is absolutely another use case as John has just posted in the chat here the data mover design work is still very much a work in progress and looking for people's feedback so the link here is the in the conveyor repo under data mover take a look at that you can add in your comments requests so I'm not seeing too many other questions here so you must have done a really awesome mind blowing job and I'm really thrilled with that so we may leave it at that today unless there's something else that you'd like to showcase or talk about today Raghu has sort of dropped off the video but I think we're nearing the end of this and it's always wonderful to have you guys here I always learned something new and get my mind blown a little bit and really backups are the essential element of everything so you guys are doing some great work there yeah thanks a lot Diane for inviting us and yeah I think having this actually work today as you're seeing again upstream would be work stuff right now but downstream will have this working and you know it's pretty exciting to that it's basically orchestrated now and OpenShift yeah no and to see it eventually be like native part of OpenShift would be really rockin' awesome too yeah I can see that in my future crystal ball right now that's gotta be coming soon well thank you very much Diane alright thanks and I'll add those links in the YouTube video that I'll upload if you guys send me your slides and if you could also send me the link to your video and that would be good too okay I'll send you a link with audio awesome that'd be great I'll try and conjoin all of those things together for everybody here who was probably taking copious notes as I'm trying to always take screenshots and catch things but I'll have it up on the YouTube channel that's RH OpenShift probably in the next I don't know four or five hours if all things collide well where I'm co-hosting right now on another screen the Latin American OpenShift Commons gathering so if any of you are Spanish speakers go to commons.openshift.org and look down the gathering list and you'll see what's going on I'm going to finish which has got subtitles whenever I'm speaking and Andrew Clay Schaefer delivered an awesome keynote this morning but he talks so fast the poor subtitled translators must have been going mad trying to keep up it's a busy day and thank you everybody for paying attention and being here virtually with us and we would love to get your feedback on this as it is an evolving project alright, thank you thanks again guys, take care alright, bye thank you