 Let's get started. Hi everyone. Thanks for joining us today. We're going to welcome to today's CNCF webinar, Data Protection for Applications Running on Kubernetes. I'm George Castro, Community Manager of VMware and a Cloud Native Ambassador. I'll be moderating today's webinar. We'd like to welcome our presenter today, Ravi Alubojina, Senior Architect at robin.io. A few housekeeping items before we get started. During the webinar, you're not able to talk as an attendee. There is a Q&A box at the bottom of your screen in the Zoom window. So please feel free to drop your questions in there and we'll get to as many as we can at the end. This is an official webinar of the CNCF and as such is subject to the CNCF Code of Conduct. Please do not add anything to the chat or questions that will be in violation of that Code of Conduct. Basically, please be respectful to all of your fellow participants and presenters. And with that, I'll hand it over to Ravi to kick off today's presentation. Hi, this is Ravi Kumar Alubojina. I'm the Senior Product Architect at robin.io and good morning to everyone. I work at robin.io and Robin specializes in running data heavy workloads on Kubernetes platform. So with that, let's start on our topic today, which is data protection for applications running on Kubernetes. So before we talk about data protection, let's take a quick look at the lay of the land as far as Kubernetes is concerned. Let's look at the state of the applications that are running on Kubernetes today. All of those are pretty aware of the first category of applications which are stateless applications. Here, Node.js, Python, and web application stacks, Nginx, HAProxy, etc. So these are qualified as stateless applications. But there is another big, big segment with three big pillars in them, which is your SQL databases, the traditional oracles of the world, SAPs, MySQL, PostgreSQL databases. They run majority of the data workloads in the enterprise. And the new players trying to offset some of the SQL databases are the NoSQL databases, which could be MongoDBs, Cassandra, Couchbase, Influx, etc. They are distributed in nature and they are trying to take off some of the workloads out of the SQL databases. The third and the biggest segment, which is loosely termed as big data, this is where we have Hadoop stacks, Elasticsearch, Elk, your Kafka pipes, and other database products. So this is a layer plan. They have different segments of applications and this is in green because Kubernetes solves it exceptionally well. Kubernetes is the de facto standard for managing the stateless applications. And it is making grounds into SQL NoSQL and big data applications. That work is in progress and Kubernetes is doing that, covering that ground gradually. So that is the state of the applications today on Kubernetes. So with that, let's look at an application composition in Kubernetes. If you take a simple application like MySQL, traditionally if you were to deploy it on bare metal or machine, you'd be concerned about storage and compute. Storage could be a direct attached disk or it could be coming from Saneray or you compute and you compute as a virtual machine or you install MySQL on a bare metal. So bottom line, you're only concerned about compute and storage. But when you come to Kubernetes platform, Kubernetes is a very, very open-ended framework. So you have to do things in a certain way and that certain way is this. If you were to run MySQL on Kubernetes, it is not just a PVC and a pod. It's not just storage and compute. You need to have all other elements to give you a fully functional MySQL application. It is, there's a deployment and there is a replica set, there is a service definition, there's a secret and config map. All of this would constitute a MySQL application. So if MySQL is that complex, let's look at the other applications. Is that MySQL? By any way, it's a distributed SQL. And MongoDB is a NoSQL database with lots of components. There are config routers, there are config servers, there are query routers and there are shards, replicas. And as you move up the application fillers, which is stateless to SQL to NoSQL to big data, the complexity only increases. If you look at Elk stack, there are several components in Elk stack. This is Elk stack, this is Kibana. Elastic itself is a distributed application. So applications on Kubernetes are complex, they involve lots of components. So with that, let's jump on to data protection. What does data protection mean and why is it different in Kubernetes world? When you talk about Kubernetes, it comes with a lot of advantages. It runs containers, it orchestrates containers, and containers by design will give you a lot of consolidation. These are highly virtualized and highly consolidated in mind. There are multiple layers to running an application. It's completely different. When you want to run it on a bare metal, you just take an RPM and install it. But whereas on Kubernetes, it is hidden by multiple layers. You start with Kubernetes and then Docker and then CRIC and then CSI contribute to that running of the Docker. And these are typically large scale applications. And we are also talking about multiple data centers. What Kubernetes allows us to do is stretch your clusters across GOs or across Amazon zones or regions. So if you're talking about multiple data centers here. And the application spectrum also is gradually changing from SQL to SQL database and big data. Applications themselves are very complex. So given this environment, what does protection mean? It means several things. It could mean encryption or replication. But we've got to talk about three big pillars here. We've got to look at it from three vantage points, which is, how do you protect the application deployment from poor resource planning? And why is it important? We'll get into why do you have to do resource planning and why is it difficult in Kubernetes? The second thing is, it is, user errors cannot be avoided. So how do you protect the application from user errors? And the third and the biggest bucket is, how do failures are bound to happen? This can fail, nodes can fail, and sites can fail. And how do you protect the applications from all these failures? Only three. Like I said, there could be many more. But in this presentation, we'll focus on these three. So what is resource planning? To explain this, let's talk about a distributed application like Cassandra. The way Cassandra is designed is, you need Cassandra as a distributed application. You pick up three nodes and we run Cassandra on these three nodes. Any data that goes to one node is replicated to the other two nodes. It is configurable, but typically it is a three-way replicated system. So, which means the compute is residing on three physical servers and the storage is directly attached to the physical servers. It means Cassandra is writing to three different disks. So that is a set amount of pain metal. Now, if you had to take the Cassandra and deploy it on Kubernetes, how would it look like? It is very well possible that all the three pods could be placed on a single server. That is the first problem. But Kubernetes provides a primitive using which you can spread these three pods onto three different physical servers. And that is by using node affinity or anti-affinity policies. But there is another problem. Even if the pods are running on three different nodes, which could be coming from a software-defined storage stack, which is interfacing with Kubernetes using a CSI plugin. So what happens in this case? Three pods are running on three different servers, but they are writing to the same disk because these are virtual volumes that were given out by Kubernetes. And what's the problem here? It is not tolerant to disk values. In a way, we have negated the assumptions of the application Cassandra. Cassandra assumes that I'm writing to three different disks. But because of the virtualized software-defined storage layer, those assumptions are violated. And how do we fix this? So anyway, let's protect Cassandra by doing a proper planning. Obviously, you'll try to split the volumes across multiple disks or multiple nodes. This is the right placement of Cassandra from storage point of view as well as compute. Now, that is Cassandra. Let's talk about Hadoop. Let's move to the extreme right side of our application spectrum. Hadoop is not an application. It is a platform. It is a stack. There are several components in Hadoop. Let's talk from Cloudera Manager, which is just an Uber UI layer which will orchestrate the entire Hadoop cluster for the Cloudera distribution, of course. So Cloudera Manager is not natively protected by that. I mean, it will use a MySQL database to write, to store its configuration metadata, but that database is not protected. It's not replicated or anything. So which means the Kubernetes has to provide a storage block which is internally replicated. Cloudera Manager requires a replicated storage. And there are data nodes. And data nodes are similar to the Cassandra nodes. The data node does three-way replication. And we have to make sure that all the three volumes that are supplied to data nodes are coming from three different disks or nodes. There is name node, which is active and passive, standby name node and the active name node. And there is ZooKeeper, which is latency sensitive. It has to make an IO in a very, very short duration. Otherwise, it will evict the node. So which means it has to be writing to a local disk or it has to see latency as close to the local disk as possible. And then other components, too, in Hadoop Stack, there is Kuru, there is Solar. So the point is every component in the Hadoop Stack will have different compute and storage requirements. So which means the stack has to be built or planned. The application has to be planned. So it is resilient to any of the faults. The building blocks for doing such planning are you have to worry about computer anti-affinity, location awareness, storage and compute affinity, and IOPS and, of course, the high availability of these spots. So what are the challenges in planning these applications? First, the summary of what we have discussed is data heavy applications deal with multiple volumes, like we have seen in Hadoop or Cassandra. Every volume will have different IO characteristics. Zookeeper is latency sensitive, whereas the data volumes for data nodes are throughput intensive. Consolidation of multiple applications onto a single Kubernetes cluster makes the problem even harder. And some of these applications have built-in replication, like MongoDB, Cassandra, and majority of the NoSQL applications do replication by themselves. So that makes the allocation tricky. So what are we looking for? How can we plan the compute and the storage? So all the applications are protected. So we are looking for application of aware storage provisioning stack. When Cassandra is asking for three volumes, the storage stack should understand the three volumes are related, that Cassandra treats these three volumes as special. The storage stack needs to know about it, but in the current state of Kubernetes, the CSI doesn't know that these three volumes are related. The calls that we get are create a volume amount of volume. We will not get a volume group. That is not the area. So what we are looking for is application aware storage provisioning. Let's move to the second segment of the data protection, which is how do we protect against user errors? We'll go back to our MySQL example, but the workflow applies to any of the applications here. MySQL is an easy example. The traditional way of protecting against user errors is taking checkpoints or volume snapshots. The question that we need to ask is, is this enough when we are running applications on Kubernetes? The short answer is no. Let me explain. Let's say this is our MySQL application. We deployed it on Cassandra using a Helm check. Now, right after installation, I want to take a snapshot. Let's say we take the snapshot of the volume. This is a volume-based snapshot of the storage arrays. So we take a snapshot of the volume. This is our initial snapshot. And after that, let's say we created a schema and added some data sets. And so that changed our PVC. The data goes into the storage. Now we take a snapshot again. So we captured all the data changes in the snapshot. Now, this is where things get tricky. Let's say if you want to change the password. In Kubernetes, the password changes in two places, depending on how the entry points are written. It is possible that the secret gets changed in two places. Now let's take capture the snapshot again. We did capture the secret change. Now, let's say we were to make some configuration changes. Again, depending on how the entry point is written, the config changes will go to config maps as well as the PVC. It could be persisted inside the database. So when we take the snapshot, the config changes are captured here. I forgot to roll back to the snapshot. So I want to get rid of all the data changes, password changes, and config changes. I want to get to my initial snapshot. Now, if you observe the picture there, the MySQL application, that is a rollback. So we rolled back to PVC. What is the problem? The final state after we rolled back the application, the traditional bare-metal inversion machine world, this is called config drift. The configuration of the system has drifted, and that is not captured in the snapshot. And that is the problem that existed for a long time in bare-metal world inversion machines, and there are tools designed to solve these problems. Chef, Popette, and there are several other tools which do configuration management. Is there an alternative to this? Can we protect the application better in the Kubernetes world? Of course, we can. Let's replay the whole scenario again. Created an application. After the creation, I'll take a snapshot here. This is the initial snapshot. Let's make some data changes, add some tables. Take a snapshot again. So we capture the entire application again. Make password changes, capture everything again. Config changes, but capture the config changes. So if you've noticed, what's happening here is we are not just capturing the data, but we are capturing configuration, which resides in config maps and secrets. We are capturing the topology, which is how many replicas, and what is the topology of the application. And we are also capturing the service definition. So we capture the application in its entirety. All the building blocks of the applications are captured in these snapshots. So if you are to revert to any of these snapshots, you'll get a complete view of the application. There is no conflict here. What did we talk about? We talked about how to take the snapshots to protect the applications better. And not all the applications provide checkpoints or snapshots. And typically applications are application stacks in real world. So you just don't run MySQL and call it an application. I mean end user-facing application, of course. The solution is built with multiple components in play. So you will have a web server, a middleware, and a database. So in that, all the layers may not provide snapshotting capabilities or checkpointing capabilities. So in Google, it is if you are to deploy the entire stack and if the platform allows you to take a snapshot of the entire stack, that will be much, much better. A piece of mind. And data snapshots are prone to conflict of issues that we have seen there. The conflict map, secret, and volume are at different points in time. And consistency group is very, very critical. And this is the core of a multi-volume snapshotting philosophy. When we snapshot Cassandra, which could be using 20 or 30 volumes, we have to snapshot all the volumes in one shot. Otherwise, you'll have consistency issues. Think about Oracle Rack. Oracle Rack is ASM, which is not file system-based. You'll have multiple redo volumes, which are ASM disks, and data volumes. So we have to snapshot the transaction log as well as the data block in one shot. Otherwise, even though the database is asset-compliant, you will end up losing data. So which means we will violate... There's a chance to violate the durability aspect of the database guarantees. And when we take snapshots, we have to float the application buffers and page cache to give application consistency. So what are we looking for again? To protect the applications on Kubernetes, we will need application-level snapshots, not just the data snapshots. So that is the takeaway from this exercise. So we've looked at two big aspects. One is application-aware storage provisioning to protect the applications. And we also need application-level snapshots, again, to protect the data-heavy applications on Kubernetes. Let's see the workflow that is built into Robin here. So you can take any application. You can create a helm-based application on Kubernetes. And with one command, you can take a snapshot of that application. Again, by snapshot, I mean capturing everything, all the aspects of the application, not just volumes. Data, config, and metadata. Snapshots are typically used for three purposes. One is rolling back to a point in time or creating a thin clone or pushing the snapshot to cloud so you get DR functionality. So with one command, you can roll back any application, the entire application, to a point in time. We can also push the snapshot to an external target to a cloud to protect the application again. And we can restore the entire snapshot back. So these are the flows that are built into Robin, but the concepts in general will apply. So Robin backups are fully self-contained. Like I said, we capture everything. And the entire application can be restored. It's not just the data or one or two tables. The entire application, including the topology, is rolled back. Let's look at the third and the biggest aspect of the protection, which is hardware and side failures. You need to protect against this value, no failures inside failures. So we've already seen some form of protection offered by the proper planning and snapshots, but this is the third phase of it. So why do we need backups? For the biggest reason is, I think, protecting against upgrades when the application is running, and also for compliance and auditing reasons. What backup allows you to do is, potentially, this form of backup allows you to do is it will make the process of hardware refresh easy, data center migration. You're not locked into a vendor, because as long as you have access to backup, you can recreate it somewhere else. So how do backups look like? So backups are based off of snapshots. Snapshots are local. Snapshots will give you a consistent point-in-time image of the application. We have seen the picture on the left side here. Initial snapshots, data changes, password changes, and conflict changes. These are snapshots. These are four snapshots that we have taken. Backups are off-site. They don't reside on the same media as the original data or the snapshots. So we can register a cloud repository and move the snapshots to the cloud. So when you move it to cloud, we have to worry about two aspects, time and cost, of course. So we want to move them to a cheaper storage, because the backups could be just a compliance or audit. That is a cost aspect. So you move it to S3. S3 is cheaper than EBS and EWS. And the second aspect of that is time. During the backup process, you only want to copy the increments. So you save time on the copy. So you always go for incremental backups. When you are resurrecting an application, which means you take an application in the... You take a backup in the cloud and you want to create that application in the cloud for several reasons. One, you want to migrate an application, which means you take incremental backups, put it in the cloud. When it is time, cut the pod and create the application in the cloud. So you have moved your application from on-premise into cloud. Or you have an application running on-prem, but you want additional compute capacity to run a completed, disjoint job. Let's say every quarter you want to run the books. So which means you have the data set, but you want additional compute capacity to crunch the numbers at the end of the quarter. So that is when you leverage your backup, create an application in cloud, and run your job there. So this is one of the benefits that the backup provides in this form, which is the application of the backup. So how do we save time during this process of cloning? What Robin allows you to do is sort of steering the S3 layer, which means if you have four terabyte database sitting in off MySQL database, or Oracle database sitting in the cloud, on S3, S3 is an object store. But application is run on top of EBS, which is a block storage. You do not have to rehydrate the entire four terabytes into the block storage for the application to start. That could be a waste of time and money, because you may not read the entire four terabytes of data. So Robin allows you to read only a subset of the data into EBS volumes by doing a storage steering. So I will go through how we do that a little bit. But the point here is, for the database to come up in cloud, you might need 100 megs of data. So we should always avoid rehydrating the entire four terabytes into EBS volume. That's what we do. So that is the time aspect when creating the Chrome. So this is one of the advantages of the backup procedure. So you can think of it as a git flow. So we have an application running for the Chrome. So we have an application running on-prem on any of the Kubernetes distributions, OpenStack, Anthos, OpenJos Kubernetes, doesn't really matter. You can create a snapshot. We can do a local clone of that application out of that snapshot, point-to-line snapshot. We can push the snapshot into the cloud storage, be any of the object storage systems, and we can take that backup and recreate the application in the cloud. So all of this is enabled because of the consistent snapshots and the backup of these snapshots, which are consistent point-to-time images. How did we do it? This is the Robin architecture overview. So we take Kubernetes as the centerpiece for orchestrating the pods, which is compute. And we quickly realized that orchestrating compute is not enough for hosting data heavy applications. It may be enough for stateless applications. For stateful applications, we need complete control over storage, compute, and network. So we built a distributed storage stack. This is a volume stack with all the enterprise-grade features that you expect from big storage vendors, such as snapshots, clones, quality of service, thin provisioning, compression, and encryption. So we have a complete storage stack. On the other side, we leveraged multiple networking stacks in a way we built a programmable network stack. So that uses control over storage, network, and compute. On top of that, the biggest piece that we have is application workflow manager, or you can call it lifecycle manager. Using that, we were able to provide one click deployment of the application, one click protection, just snapshots, one click clone, scale, upgrade, and backup. All the workflows are handled by the workflow manager. It will give you an experience of application as a service platform, like maybe Redshift, any of the AWS softwares. You can do it in the private cloud using Kubernetes and leveraging containers. And Robin is a software platform only, and you can deploy it on any of the native Kubernetes platforms, including all the Clouds, AWS Azure, and Google platform. Existing deployments, massive deployments, the application stacks vary from Oracle Rack to Hadoop Elasticsearch. So we have Elasticsearch clusters ingesting huge amounts of events, like 11 billion security events are ingested and analyzed today. And we have multiple petabyte scale Hadoop clusters running in a single server form. And the uniqueness of Robin is we can run Oracle Rack on containers. And we are running it at scale. We run 400 Rack databases in one of our customer sites on Google. Bottom line, so we need application of storage provisioning, application level snapshots, application backups. And of course, you need a storage stack that provides replication features, encryption and compression features to give an end to end protection for the applications running on Kubernetes. And that is robin.io. And thank you. All right, we're going to open it up for questions. Michael Ryan asked a question about 15 minutes and it's a stateful Petsets question mark. But Michael, if you could resubmit your question maybe with more context, that would be really good. I'm going to move on to Roberto's question as well. He asks, does Robin preserve the DB consistency? Yes, it does. Yes, it does. That's unique to Robin because Robin has a concept of application bundle. You can think of it as Helm++. Application bundle is a construct that will allow you to plug in. You could think of it as HelmChat with application lifecycle hooks. After the application is created, if you want to write some custom logic, let's say seeding the initial tables, that can be put in post create hook. So after all the pod creations, we go out on the post create hook. The consistency point comes from a post snapshot or pre-snapshot hook where you can quiet the database and then take a consistent snapshot of all the volumes in one shot. Again, note that because we own the storage stack, we can snapshot all the volumes in one shot. Okay, Roberto says, thank you. Very helpful. Do we have any other questions? We have plenty of time. Let's just give the participants here some time to ask their questions. Shansan, sorry. I can answer the maybe Kubernetes specs. Sure, sure. So Stateful Set does, is the first initiative taken by Kubernetes to solve the data heavy applications. But that is not a complete solution because what Stateful Set did was it offloaded the storage question to somebody else, which is the CSS stack. It handled, it very cleanly handled the compute part with additional primitives like node affinity, node anti-affinity and all that. But when it comes to storage, Stateful, Stateful Set has limited control on it. It cannot dictate where the block should come from. And there were additions made in Kubernetes like local persistent volumes. But that also didn't solve the entire problem because local persistent volume will take over the entire drive. But that goes against the concept of consolidation. So that also didn't solve the problem. Then I think there were strides made into solving the same problem using operators. And operators also had some difficulties in solving the Stateful applications. And this has been the flow and Stateful applications, and we are always there is more that Kubernetes has to do and it is doing to cover that area. Okay, next question is from Shamsen Song who says, does Robin work for VMware vSphere Storage Platform? There is, I'll take up the, I have a little confusion on the question, but I'll address both. Robin can run on VMs and it can use any storage that is plugged into a virtual machine. So it will just treat the virtual machine as a host and it can run on the top of that. Like it can run on AWS. There is a unique value prop on AWS and any standard storage because Robin can actually orchestrate the block devices attached to the hosts. For example, it can orchestrate the EBS volumes attached to AWS instances. If the question is about, can I use a Robin storage stack as a storage backend to my ESX or vSphere? The answer is yes and no. We don't have it today. And so we are planning, it's in the roadmap as a nice because we target. Okay, Govind asks, can you touch on the restore options like what needs to be done to restore some objects? Just one click. This is a short answer. When he says some objects, so you can, ideally, I think in order to avoid the conflict drift problems, you want to revert the entire application, which is using a single command, Robin restore or Robin rollback. If you're doing it from backup, you do a Robin restore. If you're doing it from a snapshot, you do a Robin rollback. But if you want to restore a particular object, there are ways to do it. Essentially, you need a running instance of that. You can create a think load, take it object and replace it. Because we, again, like I said, we capture everything. If you want to take just the secret, you can get the secret as well. You can create a think load, take the components that you need and delete the clone. And if you're talking about, let's say hypothetical case, you want to create or restore a particular namespace in Cassandra, or you want to create a clone of a particular namespace or a key space in Cassandra. Again, the life cycle who comes into play here, you want to clone a Cassandra database, but you want a particular portion of it or you want to do data masking. Then you will say, you will write a post clone. Robin will create a clone of the application and it will trigger the post clone, so it will go and do data masking. So many of the things will be enabled by the Robin application models and it is as easy as writing an Allen chart or in fact more easier easier than writing an Allen chart. Okay, Vikram asks, does Robin.io provide volume replication and a way to put them on the same host as per application localization as the original volumes are created in some host? Yes, we do. Yes, we do. And that is one of the interesting aspects. Robin has a very, very fine-grained policy engine. It supports replication. You can set the fault domain for replication. You can say my replicas need to be on two different drives or nodes or racks or data centers. That's one level. And the other aspect of that is I want my compute and storage to be coming from the same node, which is what we call compute and storage affinity. Let me see if I have the captions here. So in a consolidated storage place, a software defined storage place, the blocks could be coming from anywhere. Robin provides a policy which says, if my data node is running on a physical server one, make sure you get the data blocks from that same server. So that privileges. You can set storage and compute affinity, anti-affinity, which is type storage and compute. You can set fault domains for replication and you can set compute and anti-affinity policies and around Robin policies. You can also say that container X should not be running on the same node as container Y. I don't want to run data nodes and zookeeper nodes on the same server. That is a policy. And you can also say, I don't want to run data nodes and zookeepers. I don't want to carve out volumes for zookeeper and data nodes from the same disk or the same host. Very, very fine trained policies. And we have to design all these policies because we started with data heavy applications. And we do run customer environments which are highly protected. Long answer to a short question. Yes, we do. Jacob asks, is Robin available in a developer or trial license? Yes, yes. There is get.robin.io and we have a Slack channel available. Please visit get.robin.io. Okay. Govindan asks, how does Robin help in site failure kind of scenarios and also Robin's backups are certified by the software vendors like Oracle? Certified meaning, so we use Arman for certain kinds. But it's as certified as SAM snapshots. For example, if let's say you have an Oracle running off of SAM and if you're the snapshot SAM, if Oracle certifies it, we are like a SAM platform. And in that case, they are certified, I think we are certified as far as the technology goes. Yes. Okay. Won Chung asks, how does Robin.io compare to AWS ECS, Fargate and EKS and capabilities? So EKS, Robin come, you can install Robin as an operator on top of any existing Kubernetes. Fargate, I am not really sure. But EKS for sure. Yeah, so we support EKS, EKS and Pivotal Cloud on all hosted Kubernetes services. You can install Robin. Okay. And when you said EKS, did you also mention EKS? Sorry, I just want to make sure that we have that. EKS, yeah, yeah, yeah. Okay, good. All right. Azure Container Services. Okay. Roberto asks, what kind of persistent may I use to store the Robin backups as three Azure Blob, local NAS, et cetera, et cetera? So we can use any of the object storage systems. So you can use S3, you can use S3 Azure Blob Store, S3 Azure Blob Store and GCS, Google Cloud Storage. Okay. And we also support NFS if you have local, if you want to store local backups on it. Okay. Eric would like to know, what's Robin.io's pricing structure and or model? Based on storage and compute, but I suggest please contact our sales folks or just go to gendorobin.io, there's a Slack channel available and somebody will contact you. Okay. Jacob asks, how does Robin recognizes what an application, what is an application in Kubernetes? Very good question. So that is a secret sauce in a way, but I'll give you some of that away. So we use labels and selectives as one and there are many components because if you look at Kubernetes objects, it's very, they're very loosely coupled entities. You cannot tie all the objects together. You need to look at labels and selectives, you need to look at some annotations and you need to look at object mappings in some case. So we do all of that and we constitute an application. So you use a big HelmChart to create multiple stateful sets, replica sets and deployments and you give Robin a directive which says, Robin app register. And you can give labels and selectives or you can just say, this is my HelmChart. Robin will go crawl and figure out the connections between all the objects and create an application for bound review. Okay. And Robert wanted to specify better from his previous question, may I use a specific persistent volume to target backups performed by Robin? Again, application for Robin is a loose construct. So you can just put a single pv or a pvc in the application construct and it's a take a snapshot on backup. So because the application only constitutes a pv and a pvc, the backup can be done. Yes, the answer is yes. Okay. Vikram wants to know, are you using the CSI driver for shared as well as dedicated storage or do you have your own storage driver in addition to CSI? Right, to CSI. We write to CSI but we run as a demon set. So there are some functions which are not supported in the CSI stack or offloaded to our agents. Okay. That's what Vikram wants to know but we didn't write a CSI driver. So we just implemented the CSI that is published out there but we have an agent running. Robin runs as a demon set. Some of the functions are offloaded to agent. Okay. And that is, we are caught up on questions now so let's give people another few minutes. Here we go. Eric wants to know, that's just a reminder that you can keep on asking questions everyone. We still have plenty of time here. Eric wants to know, how do you support RWX access mode for the PV? Actually, we do support because we actually run Oracle Rack and the protection aspects of the RWX are offloaded to Robin. Let me explain. I think Oracle Rack will need shared access to the PV and you can always use our RWX mode but if I'm a single mounted, Robin provides that consistency guarantee that no two computers can write to the same volume. Those are fenced at our storage layer even in the network partition case. Okay. Eric, hope that answers your question. If not, please host a follow up. Café Wang, sorry if I mispronounce that asks, is the NetApp Trident supported? And we are caught up on questions. Do we have any other questions? Let's just give people some time there. I know there's a delay sometimes. All right, great. Well, thank you, Ravi, for a great presentation and for answering all the user questions. Thank you for joining us today. This webinar and the recording and the slides will be online later today at the usual places. We are looking forward to seeing you in a future CNCF webinar. And with that, thanks everyone and have a nice day.