 to the series on Valero that CloudCasa is doing. I'm Raghuram Naurakonda. I'm Chief Architect for CloudCasa. As you already know, probably CloudCasa is a Kubernetes backup service. So it works as backup as a service. But in this series, we really want to get into the details about Valero and there are quite a few topics that we have to cover. This is first in that series. And in this particular talk, I'm going to cover at a very high level what Valero is and how it works and things like that. But I do want to spend a few minutes on the CSI and in particular how to configure the CSI, what are some of the issues that we see from time to time, right? And if time permits, I also want to talk about the file system backups and restores supported by Valero, right? And before I just start the actual talk, I want to point out that everything that I show today in my slides and all the commands that I run, everything can be seen in this page. We will email you this link and this contains all the slides. As you can see, all the commands, pretty much everything. So you will be able to, if you have any questions, you can always get back to this page and follow along, right? All right, so with that said, let me start with this presentation. Again, there is quite a bit of material to cover, so I hope I don't overshoot the time. So I'm going to be switching between the slides and the demo because I want to really show the commands and how they work. And so we'll see the installation backups and restores. So many of you are probably aware of this. Valero is a very popular open-source project. It is a Kubernetes backup tool. You can tell that from many different things. You can see the activity in the Slack or how many pools are happening on the Docker Hub images, or for example, the activity in the GitHub, right? Valero GitHub issues and from all those criteria, it's kind of clear that there are a lot of people out there using Valero, right? And apart from your basic backups and restores, Valero has a couple of interesting use cases that you can think of on this cluster migration that you can do backups on cluster A, for example, and you want to migrate in a particular application to another cluster, maybe from on-prem to the cloud and things like that, you can use Valero for that. And also, for disaster recovery is another use case that Valero can be used. Let's talk about different types of backup because Valero does support different types. The most basic one is the resource data. So this is basically nothing but all your resources and their JSON specs. So Valero just picks up the JSON spec of every resource in the backup, puts them in a tar file, and copies that to the object store, right? That's the most basic backup that you can do. But typically you're going to do much more than that, right? You'll have persistent volumes in the cluster. You have application data residing in those PVs. So you want to back that up. And there you have a few options. One is you can snapshot the PVs. And if you're using CSI, that would be the CSI snapshots. But if you're not using CSI, which is getting rare now because CSI is very popular and that is definitely the future of storage in Kubernetes. So most people must be already using CSI storage in their clusters. But if you are not, for whatever reason, for example, if you're using the native EBS volumes on Amazon CKS clusters or Google persistent disks or Azure disks, those cases also, Valero can take the snapshot, right? Even if you are not using the CSI. And then the third type of the backup that I would like to talk about is what we call the backups, right? These are not snapshots. There is data transfer involved. So data gets read from the PVs and get moved to the object store. And in this category, Valero can use a couple of different tools. One is RESTIC, the other is Copier. These are completely different tools not related to Valero at all. Valero just makes use of them, right? And I'll discuss a little bit more detail in the next few minutes. The next thing that you want to know is the object stores, right? Backups obviously have to go into some storage. They need to be stored somewhere. And Valero supports a wide variety of object stores. And most of them you can consider as S3 compatible storage. Obviously AWS S3 is one of them, but not just the one, right? There are quite a few S3 compatible storage out there. Even Azure Blob storage, Google storage. So there are many different kinds of object stores that Valero can back to. So let's talk about installation. I think once I talk a little bit about installation, I'll go to terminal and I'll show you how to do the installation. Now, there are again multiple ways of doing that. Valero comes with its own binary called Valero. So that's the CLI method. You can use not only to install, but pretty much to manage your entire Valero installation in the cluster. So that's what I'll be doing today. I'll be using the CLI, but there are other ways. For example, Helm is one way to install, but there are other ways out there. And to use the CLI method, the first thing you would do is go to the Valero GitHub page, go to the releases section and you can download the binary here, right? So I have already done that, so I'll be using that. Before you start the installation, there are a few questions that you have to ask yourself, right? Because depending on the answers to these questions, the installation command or the installation mechanism is going to be different, right? The first and the most important thing is where do you want to store the backups? Like I just explained different types of S3 providers that Valero supports, but at this point you have to make a decision which particular one you want to use, right? Then do you want to use CSS snapshots? I would say just enable it all the time because almost definitely you will be using CSI storage or are you going to use non-CSA as well? So these are all the questions that you need to prepare for you need to answer before you go ahead with the installation. And finally, do you want to enable the file system backups? In most cases, I think the answer to snapshots and resting backups or file system backups is going to be yes. In our particular case, I'm going to use AWS S3 bucket for the backup. I will use CSI snapshots. I'm not going to bother about non-CSI in this talk. We will definitely do one more webinar on that because it requires more details to be covered. And finally, I want to enable resting backups. So with those answers in mind, let's go back to the terminal and go ahead with the installation. For all my demonstration today, I'm going to use a mini cube cluster. It is a local cluster which makes things much simpler. MK is my alias for mini cube. So that's what you will see me using. So if you do list the clusters, you can see that there is one cluster running. Valero, and again, I have an alias for cube control. So if you see the namespaces, it's a bare minimum cluster, right? So this is the install command that I'm going to use. I'll briefly explain what this is doing because we decided that I want to use the AWS bucket, right? So you have to select the plugins based on that. So I selected the AWS plugin. I want to enable CSI. So I enabled the CSI plugin as well. One important point to note here is what version would you pick, right? So I'm actually using Valero 1.10. So depending on the Valero version, you need to pick the right version of the plugin. And how do you know which version to pick up? So each plugin has its own GitHub repo. For example, this is the GitHub repo for Valero plugin for AWS. And you can see on the GitHub page there's a compatibility table. And for 1.10, Valero version, you need to use 1.6, right? That's the plugin version. And in the releases here, you can see there is 1.6.1. That is the latest plugin available. And you will see that's the tag, Docker Hub tag that I'm using. Similarly for CSI plugin, this is AWS, right? So this is the CSI plugin. The table says we need to use 0.4.x and the release seems to be 0.4.2. So that will be the Docker Hub, Docker tag image tag that we'll be using. The next option is use node agent. This is basically telling Valero that we want to enable file system backups. So this would install node agent on each node in the cluster. We want to enable CSI feature. And the rest of the flags are all saying, I don't want to configure any backup storage. Unfortunately, Valero command requires you to configure some backup storage at this point itself. But personally, I don't like combining the installation with configuration. So we are going to say, we don't want to configure any backup storage at this point. We're just happy installing Valero itself, right? So let's go ahead and do the install. I have another alias here. This is basically cube control, the check things in the namespace Valero. So let's first see what namespaces we see. You can clearly notice there is a new namespace created called Valero. Let's see what is in that namespace, right? So you can see a Valero pod up and running already and you see another node agent. Let's go ahead and see more what other things are there in that namespace. So you can see a deployment called Valero, which is running the Valero pod. That is actually the Valero controller that monitors all the Valero resources and then acts on them, right? And then you have a demon set called node agent. And this is a one node cluster, but if you have multi node cluster, there is going to be one node agent per node and that is required for the file system backup, right? So this is the installation, it's up and running. So let's move on to the next step, the process, right? Once you install the software, the second thing you want to do is to configure the backup storage. So we picked AWS S3 bucket and the way you do that is by creating a resource called backup storage location, right? While creating that, you give the configuration details. So let me go back and create that storage location. You can do that using cube control if you want or you can use Valero CLI. Again, I'll be using Valero CLI throughout this talk because it makes things a little bit simpler. Now to configure the AWS S3 bucket, there are a couple of steps to do. The first thing is you need to pick what identification you want to use for the S3 bucket. There are different ways of doing that and I again pick the simplest method which is access key and secret key based, right? There are other ways of configuring AWS S3. For example, there is a method called IRSH which is IAM rules for service accounts. Definitely, this is a recommended approach. This is more secure than using access key and secret key but this is quite involved. And in fact, incidentally we published blogs just few days back on how to do this. So Valero does support this mechanism and I highly recommend that you go and check this out. For today itself, I'll use the access key and secret key and for that method, the first thing you need to do is to create a secret, right? So I have a bucket. I just want to quickly show you bucket that I have going to use the AWS command line. So this is the name of the bucket that I'm going to use. Right now the bucket is empty. There are no objects there, right? So let me go ahead and create the bucket. But before you do that, the first step is you need to create a secret which contains the credentials for your bucket, right? And before you create the secret, you need to create a file. The file is going to look something like this, right? So default and access key and secret key. So I already created that file with my access key and secret key. So we'll go ahead and create the secret. The details are in this file, AWS credits, go ahead. So the secret is created. And the next step is creating the backup location, right? So let's do that. So this command is saying, I want to create a backup location. This is the bucket, right? This is the bucket name is intro demo, right? Let's do that. And then the credential is the secret that we just created. The provider is AWS. It's in US East 1. And the name of the backup storage location, I want to call it AWS demo, right? So we created that successfully. Let's see there is a command backup location get that is going to show you the details of the storage location you just created. It's telling you that it's available and the access mode is read, right? So by default, we need to be able to write and read. So that is the access mode we used. And this phase is very important. If for whatever reason, there's a problem in accessing the bucket, maybe you don't have sufficient permissions. This phase is going to be not available or pending or something like that. But in this case, you can see the bucket, well it will find a bucket accessible. So it says the phase is available, right? So we have the bucket now, backup storage is all done. So let's go back to the next step. The next thing I want to do is do a very basic backup, very simple backup that is consisting of the resources. There are different ways of including what you want to include in the backup. Typically I recommend doing a full cluster backup because you should include as much as possible in the backup, but at the restore time, you're always more selective. You want to restore only what you need. You have the option, you can say I want to backup for namespaces or you can say I want to backup only if it can be given label or even given resource types like PVCs or parts and things like that. For our demo, we will just do backup for entire cluster. And this is just a basic resource backup because we don't have any persistent volumes to begin with, right? So let me go ahead and create this backup. I'm calling it basic backup. And do note that I'm explicitly providing the storage location here. That's something that we created just now. There is no need for you to do that if you configure the storage location as default. There is a way to do that. But again, explicitly providing that it is always better, especially when you're doing demos, right? You don't, you want to have as less magic as possible. So here, let me go ahead and create the backup. It created and you can always do backup get. That's going to tell you how it is doing, right? So here it's telling us that backup is in progress. It's completed. I mean, there are not that many resources on this cluster. It's a very small mini cube cluster. So it's no wonder it finished in pretty quickly. Now, there are different ways of finding more details of this. So you can do, for example, describe on this and that will tell you how many resources are backed up, timestamps and things like that. And here it's telling you CSI volume snapshots. Obviously there is nothing included because we don't have any PVCs, right? Even there is a, even more interesting way to see more details about the backup that is you pass the details option. And here you're going to see the full list of resources that are included. So this is all the resources that are backed up and in this particular back, right? And finally, I want to show you how things show up on the bucket itself, right? So this is the bucket that we used. And you can see that will appear a bunch of objects here. And the most interesting of this is here the basic backup.tar. This is essentially an archive of all the JSON specs of the resources and all the lists that you have just seen. The JSON spec of all those resources are part of this tariff. The other you can notice is the logs. So you can, there is a way to see the log files for this backup. The others are practically nothing, right? There's very few bytes because we don't have any volume snapshots included in this backup, so you don't see anything else. I want to show the logs as well. So let's go and run that command. So if you do this command, it's going to show you the log file. So essentially it's basically pulling the log down from that S3 bucket and ensuring the log. So in this case, everything is completed. So you probably don't worry about the log, but in case of any errors, any warnings, sometimes the backup completes partially. In those cases, you can just do the logs and see if there are any errors, right? That's about the basic backup. Let's go to the next step, which is the CSI. So I do want to spend a few minutes here because the primary focus of this talk is how it works with persistent volumes, right? You're not going to have a cluster without persistent volumes, right? In most cases, there is a persistent storage. There is probably stateful applications like databases. So it's very important that you backup the data. And CSI is a standard to expose all the storage systems to the Kubernetes cluster, right? So with this Kubernetes implemented CSI, I think sometimes starting in 1.13, so it's quite a few releases back. And with that support, you just request the storage for the containers and your parts by creating these resources called PVCS, Red Persistent Volume Plates. And the recent CSI versions, and when I say recent, probably three or not even more, maybe five or six versions, you can, the CSI spec interface supports snapshotting the persistent volumes. And that is very, very important for the backup applications, right? But remember, one important point to note here is not all CSI drivers implement these snapshot functionality. The example here is AWS type EFS, the elastic file system. It does implement the other parts of the CSI. You can create the persistent volume claims, you can position the storage, but you will not be able to take the snapshot because snapshot is an optional functionality of the CSI, right? Configuring the CSI is not a joke. It's not part of the core Kubernetes as I mentioned here. So when you do the default installation of any Kubernetes distribution, whether it is cloud provider or on-prem, you are not going to get the CSI artifact. So you have to do something extra. And that extra includes installing the CSI customer resource definitions. There is something called external snapshotter that provides some common components to all the CSI drivers. So you need to install that as well. And finally, the CSI driver itself that interfaces with your storage, right? So as you can see, there is quite a bit of configuration that you need to do to get the CSI to work properly. In cases of cloud provider clusters like EKS, Azure clusters and GKE, the process is a bit simpler, definitely. But when you're talking about on-prem clusters, you need to check the documentation for each distribution for each provider and see how you need to do that because I can almost guarantee that the configuration is going to be tricky. And if you don't do that configuration correctly, what happens is that the snapshots will not work. So you can create a snapshot resource, you can request a snapshot, but nothing will happen, right? And in case of Valero, what happens is that Valero would wait for 10 minutes. It creates a snapshot, it issues a snapshot request, waits for 10 minutes to see if something is happening and if snapshot is not ready in that time, it will simply time out. So Valero will log an error message and we'll just move on with the other resources. And in fact, we see many users complaining about that. We see many cloud CASA users complaining about CSI snapshots timing out and we see Valero users complaining about that as well in the GitHub form. And we try to answer as much as possible what to do in these scenarios. And in almost all cases, this turns out to be a problem with your CSI configuration. So in order to make this process a bit simpler, right? We answered many questions, these are the steps that you need to do to verify the configuration. So based on that experience, we went ahead and developed a shell script which automates the process and makes things much easier for you. So this is the link that describes the details on how to do that. For example, I'll quickly show you that. Okay, let me hear. So this is the page that talks about how to use this shell script and run it locally. I'm going to quickly show that here on my Minikube cluster. So this is how you run it. Right, directly from online, you can run it. If you want, you can download it locally and run it. You can take a look at the script, what it is doing. The script will tell you what it does, but I'm just going to say yes. And the script failed right away. It's telling you that the CRD is volume snapshot and these things are not there. So there's nothing for the script to do. So I'm using Minikube and on Minikube, the way you do that is enable the add-on called volume snapshots. So I'll do that. But again, if in your distribution, it's going to be different, right? You have to follow the instructions for your distribution. So let's go ahead and run the script again. And this time you can see that it found the CRDs, right? So it went past the step and now it's saying I don't find any CSI drivers. So let's go ahead and do that as well. And again, on the Minikube, the way you do that is enable the CSI hotspot driver. So let's do that. And while this is happening, it'll take probably a few seconds or maybe one minute. So let's go back to this page. Not this page, yeah, here, right? So here, the page will tell you what exactly it is doing. It is creating a small PVC and then it'll issue a CSS snapshot for the PVC and it'll check if the snapshot is working or not, right? Okay, we can see that this is finished. So let's run the curl command again. And this time it found the CSI driver, found, so it's going ahead and it's creating a PVC, a pod. Now in that particular test path, it just means that it's able to create a pod and PVC and now is testing the volume snapshot creation for that PVC, right? And here it's waiting. It created the PVC on one volume snapshot and is waiting for the driver to do that. And actually you can see that the test has passed, right? And which means that our CSI configuration is in good shape. So now we can go ahead and further do the CSI backups with Melero. So let's go back to our presentation. The script is just cleaning things up so it will be done soon. So Melero's support with the CSI, so Melero supports taking the snapshot of CSI volumes but there is a little bit of more configuration that you need to do. So to take a snapshot, we're using CSI, you need to have a volume snapshot class. This is very similar to the storage class that you have with persistent volumes. You need to have a corresponding storage class, right? To provision dynamic storage. Similarly to create a CSI snapshot, you need to have volume snapshot class. And then you need to set this particular label on that and so that Melero knows which one to use because in theory, you can have more than one volume snapshot class. I would also like to point out that in the new versions of Melero, we are people are discussing how to improve this, right? For example, if you have only one volume snapshot class, maybe you don't need to set the label, right? Because you can automatically use that. So in the newer versions of Melero, this process would be a little bit simpler but for now with 1.10, we need to do that. And also I just want to quickly point out that when you restore from the CSS snapshot, Kubernetes allows you to create the PVC directly from the CSS snapshot. So there is no real restore happening and there's no data movement in the restore. And that is what Melero does. It creates a new PVC and it'll mark snapshot as the source of the PVC. So let's go back to the cluster again. I'll show you the one that I have the storage class. I mean the snapshot class. So you can see one snapshot class, the deletion policy is delete. It's highly recommended that you keep this deletion policy as retain. Otherwise, in some cases, you may lose the snapshot. So I'll go ahead and edit this snapshot class and change this from delete to retain. Sorry, let's do get again. And you can see that change took effect. And I'm also going to set the label here on this. So this is the CSS volume snapshot link, right? So now with that done, we can go ahead and create the backup, right? So let me do that. Oh, actually before you go ahead and create the CSS backup. We don't have any CSS part, right? We need CSS PVC. So I have a small test PVC and part using that. So I'm going to do that. So I'll quickly show you what I have here. So I have a small PV claim. It's coming from the storage class. You can see this is the CSI storage class. There's a part where I'm mounting this PVC and slash data. The part really doesn't do anything. Just sleeps for some time, right? So we're not interested in what the part does. We just want to see how the PVC is back up. So let me apply this file. And we will notice, we'll see this part coming up. You see the new namespace test at CSI. And let's see what is there in that namespace. So you see the part is running. Let's see the PVC, right? The part, the PVC is bound. So let's actually get into this part, eject into this part. Here I'm going into that and slash data. That's where the PVC is mounted. You see there's no data now. So just for the purpose of verifying restores, what we will do is create a very small piece of data, right? So you see a file here created and all you see is just one line with the dates down. So just make a note of this because when we restore this PVC, we should see exact same data, right? So we have the CSI now, so let's create a CSI backup. Yeah, so this is exactly the same as the previous backup command. So when Vero sees a PVC, a CSI PVC, and it finds a snapshot class, it would automatically create the CSS snapshot because we already included the CSI plugin, right? So let's go ahead and create this. So backup is created. Let's see what's going on with the backup. It's in progress. Let's see what's going on. Should be finished pretty quickly because there's not much data. So as you can see the backup has completed. Again, I'll quickly see the describe command with details. Sorry. So you can see that it is clearly showing you that one CSI volume snapshot will be taken, right? So let's go ahead and do a restore now from this. And here I want to briefly talk about the options. Here we're saying I want to restore from this backup. The backup is just created. And like I said, your restore is usually more narrow compared to in scope, right? Compared to the backup. We did the whole cluster backup, but I want to restore this particular namespace test and CSI. And not only that, I want to restore to an alternate namespace. So this also shows a cool feature of Wellero where you can back up, you can restore one namespace to with a different name. Here I'm saying I want to restore test and CSI to a new namespace called restore CSI, right? Let's go ahead. Restore is created. And let's see what's going on with this. It's already completed. And let's see what namespaces we see here. So you see the restore CSI namespace here. And if you see the pod, part is up and running. Let's see the PVC, right? And there is a PVC created. Let's again, get into this pod, exit into this, go into the data. And you see that the test file is there, is restored with the same data. I quickly want to show the PVC here. And I want to point out this particular one. So when Wellero created the new PVC in the new namespace, the data source is pointing to the snapshot. So this is what I meant when I said you restore from the CSS snapshots by creating PVCs from the snapshot. There is no data transfer involved in this, right? Okay, that completes what I want to talk about this CSI. Let's move on to the file system backup, which is third kind of the backup, right? So this is a different type and there is actually a data transfer involved. So in this type of the backup, Wellero reads the data from the PV and the data is transferred to the object storage. And for this purpose, it uses one of the two tools that you can use either RESTIC or Copia. Both are kind of similar. They both support compression, deduplication, encryption. There are some limitations. You can read them on the Wellero docs. And in fact, we will do one future webinar on in more detail to cover both the tools. We'll talk about the differences and we'll talk about how you can choose or what fits your scenario, right? In this particular demo, I'm going to use RESTIC. That is the default with Wellero. And one other point to note here is, in some cases, when snapshots are not supported, for example, if you have an FS or EFS type of persistent volumes, there are no snapshots possible with those PVs. And in those cases, file system backups are pretty much the only option that you have. And when final point is you cannot combine snapshot and file system backup. So you can either do snapshot backup or file system, right? Not both. So let's go back and see how they work in practice and then we'll come back to the slides. So we will use exactly the same PVC, the CSIPVC and the save pod to backup using the file system backup, right? So let me create, yeah, I call this FS backup. And you notice there is an additional option here called default volumes to FS to backup to true. So basically we're telling Wellero that in this particular backup, any persistent volumes that are found in the backup should be backup using a file system, right? That's what this option does. So let's go ahead, create that. And let's do get on FS backup. Progress. So this will take a few seconds. Again, it's the same data, right? So not much data, so it should really finish pretty quickly. And once this is done, I want to show the contents of the bucket again and talk about what object you will see on the bucket. Yeah, so while this is going through, again, if anybody does have any questions, feel free to throw those up in the chat. But I know there was one question there from Antonio. Yes, so we'll share this and probably also post these generally on the site and on YouTube, Antonio. So if you do have any other questions or want to see anything further in depth, by all means feel free to ask and we'll certainly share that with you. Yeah, thank you, Evan. Yeah, and I want to also show this again. This is the link that we will send to everyone. So all the presentation, all the commands that I'm running right now, everything is documented here. So in great detail. So please feel free to take a look at this. Okay, so the backup is completed. And again, let's do the describe command. So keep making this mistake. And you can see here, it's showing you that RESTIC backup is done. And I want to show the bucket. And suddenly you're going to see a lot more objects here. And you can see something created called RESTIC. And underneath that, you will see one, they call it repo. So RESTIC has a repository and all the backup goes into that. So Willow creates one repository per namespace. So you can see one for restore CSI, FS backup. I mean, these are all the namespaces, right? So you're seeing all these objects created underneath those repositories. Okay, so that's the backup. Now we will go ahead and do the restore exactly the similar scenario we did for the CSI. So let me go ahead and create that. Yeah, this is the restore command. We are saying I want to restore from FS backup. Again, restore only one namespace and I want to restore to a new namespace. This time I want to call it restore FS. Go ahead, created that. And then let's do get on the restore object. So it's restored in progress. It's completed not much data really. So again, let's see how many namespaces are showing up. You do see the namespace called restore FS. Let's see what is there in that. So you see the pod running and you see the PVC. And I want to show this PVC and distinguish this from what we saw before. You know, contrast this with the PVC we saw from the CSI restore. So with the CSI restore PVC, you saw the data source, right? Because the PVC came from the snapshot. But in this case, you don't see any such thing. So what happens here is a PV is dynamically provision. That is a blank, right? MTPV. And then the data is restored on top of the PV. Actual data is moved red from the object store and copied to the PVC. So again, let's eject into this. This is the restore CSI, right? This is, yeah, this is the restore FS. And again, we'll go to that mounted directory and you see the test file. And you can see the date, right? So the data is correctly restored. All right, that's about the file system restore. Let's get back to the presentation. A couple of key things that you want to notice. You want to make note of with the file system backups. So the data is being read directly from the PV, right? There are no snapshots in the picture here. So if the data is changing, actively changing, it would change probably if you're using applications like databases or any other applications which will be frequently or periodically changing the data on the disk in the PV. So your file system backup may not be consistent, right? Because the PV backup is happening directly from the data, live PV. So for this reason, you would like to use what are known as hooks. Valero supports this hooks to freeze and unfreeze or freeze and power the application. So before you do the backup, you want to freeze the application at which stays all the data is flushed into the storage and probably the application is put in some backup mode. Then you do the backup. And once the backup is done, you take the application out of the backup mode, right? And this is very, very important. So you should take a look at Valero hooks to make use of that, to make the backups consistent. That's about file system backups. Let's briefly talk about some limitations about Valero. Again, we talked about this already. But one important thing I want to point out is Valero is a single cluster solution, right? So it always talks about a single cluster. You install one cluster, you create all the custom resources on a single cluster. Backup storage location is also tied to the cluster. So everything that we discussed so far, everything is happening in the context of a single cluster. But it's very rare that people will have only one cluster, right? In fact, the recommendation is to have, people always talk about multiple clusters, multi-cluster solutions and things like that. And in fact, we see from Valero forum questions and things like that, people are using tens and possibly hundreds of clusters. And so what happens is that you will end up writing some tooling on top of Valero. You will never directly use Valero to manage individual clusters. It quickly gets out of hand. So what we noticed is that companies and users are writing their own scripts and even sophisticated tools on top of Valero. So most probably you would end up doing the same thing. And the other limitation I want to talk about is backups are not done from snapshots. So the ideal way to do the backup is snapshot the PV and then you read the data from snapshot, right? That makes the backups consistent. But unfortunately Valero doesn't support that at this moment, right? So I'm happy to say that we are thinking of working with Valero maintainers and in helping address these limitations. So stay tuned for updates in this direction. So we definitely want to help Valero project address this especially the second part, both parts, but the second part is something that we already solved with the CloudCast solution. And CloudCast already supports multiple cluster management, right? And we also solved the problem of backing up from the snapshot. So we want to share that knowledge and contribute that back to the Valero project so that everybody can get benefit out of that. So stay tuned for that. These are just the links that we already, some of the projects that we talked about. Right, in conclusion, I want to say that Valero is definitely a very popular open source option for you if you're thinking of backing up your Kubernetes clusters. There are a lot of people out there using that. So you should definitely give it some thought. But all I would say is read the Valero docs. There are a lot of things that you need to really think through before you go ahead and start implementing the solution. And like I said, you should give thought to implementing something on top of Valero to make this process a little bit more convenient for yourself, right? All right, that concludes the presentation. Hopefully I didn't overshoot too much. The time given to me. I finally want to mention about the KubeCon Europe that is coming up next month. Aaron, you want to talk about this? Yeah, so just real quick, I'm sure a lot of you will be attending or will be attending virtually the upcoming KubeCon in Europe. So that is April 18th through the 21st. We will be doing some virtual passes in the coming webinars and stuff, but stay tuned for all of that. And again, in the upcoming series for the Valero, titles and everything that we're going to be doing over the next coming months. So if you do have any questions on anything as far as the events, KubeCon, Valero, the webinars, Cloud Casa, feel free to always throw those up in the chat and we'll certainly get to them. But with that being said, I think that's everything for today. So I appreciate Ragu going through the presentation and demo and everybody that is on with us today and will continue to do these over the next coming months and weeks as well. So again, thanks everybody, thanks Ragu and we'll catch everybody next time. Okay, thank you, thanks everyone. Thanks guys, appreciate it.