 All right, I think we're going to go ahead and get started. I'd like to thank everyone for joining us today. Today's webinar, Kubernetes Backup and Migration Strategies Using Project for Lero. I'm Subri Blackman. I'm CNCF Ambassador and Security Engineer at Docker. And I'll be moderating today's webinar. We would like to welcome our presenters today, Steve Chris and Tom Spoonmore from Project for Lero. A few housekeeping items before we get started. During the webinar, you will not be able to talk as an attendee. Everyone is muted. There is a Q&A box at the bottom of your screen. Please feel free to drop any questions in there, and we'll get to as many of them as we can at the end. With that, I'll hand it over to Steve and Tom to kick off today's presentation. Great, thank you, Steve. My name's Tom Spoonmore. I'm a product manager at VMware. I'm a Valero product SME. And joining me today is Steve Chris. Hi, everyone. I'm Steve Chris. I'm a software engineer at VMware. I joined about six months ago as part of the HEPTIA acquisition, and I'm the tech lead for Valero. And also joining us today on keyboards is Carlesia Compos. Carlesia will be answering Q&A throughout the webinar today. Glad to be here and help out. OK, so today's agenda is we'll be talking about Kubernetes Backup and Restore migration use cases, how to think about backup and restore in a cloud-native world. There'll be an introduction to Valero in the project. We'll talk about strategies for protecting your applications in a cloud-native way. Steve will also be doing a demo of migration and Valero backup and restore. So Kubernetes is a great platform for running your applications. And as with all data processing data applications, you're going to need to provide some data protection capabilities for those applications. So there's a number of use cases that you need to be prepared for, disaster recovery, data protection, data migration, and Project Valero is one way to help do that. So in the disaster recovery use case, you'll be considering ways that you can rebuild your clusters if something happens to your cluster or if you choose to rebuild it to take backups of your applications, point-in-time backups, restore those backups for a variety of different reasons, or provide recovery in the case of a failure of some kind. On the data protection side, you have to be concerned about data losses with any data application. It's possible you could have data corruption. Applications or hardware could go bad, could create corruption. You could have other reasons why your data becomes corrupted, and you have to go back to a certain point in time of restoration. And for a number of industries, archival retention is critical in the legal industry and other regulated industries. Just having archives of data is a regulatory and compliance issue that has to be planned for. And in the world of public cloud and private cloud, we see a data migration use case from our customers as well. And for data migration, these are times where you want to move from one cloud platform to another, perhaps. Sometimes just the complexity of Kubernetes upgrades causes you to choose to rebuild a cluster rather than doing an upgrade to a cluster. And in those cases, you'll want to move your data and your applications from one cluster to another. The other use case that we're seeing quite often with users is the need to hydrate or bring data into one of their tests or staging environments from a production or other environment for testing purposes, for development purposes, and other reasons that they want to have data available across a number of environments. And so these are all reasons that you would want to use backup and restore technologies to be able to provide responses for each one of these use cases. So Project Valero, just a little bit of background Valero Valero is an open source tool for backing up your Kubernetes clusters and persistent volumes for your applications. It works both on-premise and in public clouds and natively on Kubernetes. So Valero, unlike many backup and restore utilities, isn't just concerned with passing a file system and picking up the files. But it also is aware that Kubernetes itself is really a running state machine that is managing the state of all of your applications, the deployments of the deployment manifest for all those applications. And all that is considered data to Valero that needs to be backed up. Currently, we're well over 2,500 stars on GitHub. We are at version 1.0. And as of this week, we're 100 contributors strong in the project, and we're super happy about that. Valero itself has a couple of different components. One of them is a command line interface that you can use to install Valero, to also initiate backups, to create backup schedules, to add backup locations and targets, as well as to perform restore operations. So then on the server side is a server application, a controller that operates on a number of CRDs that are created by the Valero command line that give instruction to the server as to how it should go about performing backups and restores. So Valero seeks to address each of the use cases that we spoke about before. So on the disaster recovery side of things, Valero reduces the time to recover your clusters in the case of some kind of infrastructure loss data corruption or other kind of service outage. Valero backs up all of the objects in your cluster and can, in a very easy way, recover the cluster to a new target cluster. For data migration, Valero does have a lot of capabilities to allow portability of the objects on your cluster from one cluster to another. There's also techniques, which Steve will go into a little bit later for doing things like transforming data objects as they move from one cluster to another, perhaps data objects like storage classes, et cetera, need to change from a source to a target cluster. Valero has all of these kinds of capabilities and an understanding of how to make those migrations successful. The other thing Valero is, like Kubernetes itself and many other cloud-native tools, is it integrates through DevOps workflow. So you can use GitOps kind of strategies to drive Valero. You can also create kind of ephemeral clones of namespaces. So you need to do an operation on a namespace or you need to transform a namespace from one cluster to another or otherwise make it available to another team or set of users. It's very easy with Valero to use the backup and restore of a namespace. And Valero really understands how to deal with that and how to understand what's in a namespace, all the objects, how to reconstitute those objects and data volumes into another namespace successfully. And so this kind of sets us up for what we really think is kind of a future for ephemeral clusters. And Valero provides a capability to really rebuild clusters in a very easy and scriptable way so that we can start thinking of clusters in ways that we are thinking of cloud-native applications where the application can be completely removed, restored, or components of the application are resilient because of strategies like consistency and declarative YAML. So Valero allows you to start thinking of your entire clusters in this way so that it unlocks the abilities to be able to easily reconstitute clusters. And again, as I pointed out before, you can think in terms of these clusters being so easy to reconstitute that you don't really even need to do upgrades. You can just build new clusters and transition your applications from one cluster to another. So how do we think about the new paradigm of backup that Valero is coming with as it relates to maybe the way that we've traditionally thought about backup and restore? So a traditional backup and restore, you're really backing up an application. And the application is really the hardware that it's running on. So in a pre-cloud-native world, you're backing up a server. Applications ran on a server. Server had an OS, configurations of the application, file system structures, et cetera. All of these things need to be backed up in a very kind of infrastructure-focused way. But as we move into cloud-native applications, we're not so much interested in the underlying infrastructure as that's been largely abstracted away. So we need to think about things a little bit differently. And particularly as it relates to Kubernetes, like I said before, it's a deployed application itself. It is a stateful operations platform. So there is data associated with that. So your application can be deployed across a number of nodes. It can, the state of that application, both the desired state as well as the operation state, are stored in a database called SCD. And we are concerned now with not just backing up the application, but how we deal with that, how that application is distributed across nodes, applications that are storing state in SCD, the Kubernetes itself storing of its own state in SCD, and then, of course, the application deployments themselves, so the deployment resources themselves and how they're handled. So we need to think about how all of this stuff is deployed and tied together in all of the various objects that are underpinning an application, how they are related to one another and how they can be reconstituted in a consistent way. And this is what Valero endeavors to do. So with that, I'll let Steve talk a little bit about Valero's technical approach and how we actually do that. All right, thanks, Tom. So things are definitely a little bit more complicated in a Kubernetes environment than in a traditional IT environment when it comes to data protection. And so the first step to developing our data protection strategy is really to think about where is the state within a Kubernetes system? And so the first place that most folks think about when they think about state in Kubernetes is that at CD database that Tom alluded to. So this is the backing store for the Kubernetes API. This is where all of your deployments are defined, your config maps and secrets and pods, et cetera. And so this is a dynamic store of data that's going to be changing as users are deploying applications to your Kubernetes platform. And so this is going to be super important to be able to protect and to have backups and to be able to recover. And then additionally, the second main store of state within this environment is your persistent volume. So this is really where your applications are storing their data. And they're doing this in a Kubernetes native way using declarative definitions for these volumes. But we really need to be able to back up not just the configuration of these things, but the data inside them as well in a way that's linked back to the Kubernetes application definitions. When we think about the nodes that make up the cluster though, so your masters and your workers, there's not a lot of state that's inherent in these things themselves. So each of these nodes are gonna be running instead of the core Kubernetes components. So on the master, we'll have the API server running, we'll have the controller manager running, and the workers will be running the KubeLit and KubeProxy and other things like that. But these really aren't changing very much over time. They're essentially a set of commodity components that can be used to define a cluster. And so when we think about the approaches to kind of data protection for each of these components, it's gonna vary based on whether they are in this kind of stateful category or in the stateless category. So next slide please, Tom. All right, so starting on the left when we're talking about the nodes that make up a Kubernetes cluster, it's our view that it's less important to think about backup and restore here than it is to think about really using automation to make it very trivial, very straightforward to provision new infrastructure as needed to replace existing ones. And so whether you're using Ansible, Puppet, Chef, et cetera, the focus I think here is really on making it super easy to stamp out a new copy of a worker node or a master node so that if you need to replace one in your cluster, you're able to do that. And this applies not just for individual nodes, but for clusters as well. So Tom alluded to the kind of concept of ephemeral clusters a few slides ago. Cluster API is a great example of an effort that's going on in the Kubernetes community to really make it very straightforward to stamp out new clusters based on a predefined template. And so we think that's kind of the right approach to think about how to deal with needing to replace individual nodes or clusters within a node. And then once you do that, there are tools that Kubernetes provides, that Kube control provides for safely taking unhealthy nodes out of your clusters and swapping in a new healthy node that you've been able to stamp out from your template. On the right side though, things are a little bit more complicated. So with Etsy and persistent volumes, you can't just reproduce these from a template. These are, they store dynamic data and they're changing over time. And so we do really need to think about how to be able to back these up and recover two specific points in time. Next slide. All right, so let's dive into each of those stateful components a little bit and talk about some strategies for data protection here. So starting with Etsy D, there are a number of different approaches we can take to protecting this data. And the first two, block and file system backups basically mean that you're taking either block or file level backups of the device that Etsy D is storing its data on. And this is kind of a good basic approach. It's kind of the lowest common denominator and it'll get you a certain level of data protection for that data. But Etsy D itself is a distributed system, typically runs multiple replicas. And so if we really want to get application consistent backups of this data, it makes sense to look at the Etsy D CTLs capabilities for doing snapshots of the cluster. And this will give you application consistent backups of the entire dataset that's stored within Etsy D. But there is another approach to backing up this data. And so we know that within Kubernetes Etsy D is used to store the data for your API resources. And so another approach to backing up this data is to use the Kubernetes API itself to actually access the data that's in Etsy D and store it in a backup. The Kubernetes API provides something called a discovery API, which enables you to kind of dynamically discover all of the different resource types that exist within the API. And then from there, you're able to capture the full definitions of each of those. So this is another viable approach for accessing that data that's in Etsy D to back it up. And then on the persistent volume side, again, we have a few different approaches. So the first one is to really rely on your storage provider. This may be a public cloud provider or maybe an on-premises provider, but to rely on snapshots as a way to capture point-in-time references to your data and then to be able to potentially create durable backups from those so that you're able to restore if you need to. This is typically kind of a block level function, but you can also look at taking file system backups of the persistent volumes that you're using. This would be particularly useful if you're looking at doing kind of single file recovery if that's an important use case for you. And then a third option here that's emerging is looking at the container storage interface, which is becoming a standard for Kubernetes and other container orchestration systems and looking at the emerging snapshot APIs that are being added to those. Now, those are currently in an alpha state. The upstream storage community is looking at moving these to beta relatively quickly, but this provides another kind of integration point for being able to take point-in-time snapshots of the persistent volumes you're using. Next slide. All right, so with that background, let's talk a little bit about Valero and its approach to data protection. So first of all, Valero enables you to back up both your Kubernetes objects, so all the Kubernetes API data that's stored in LCD, as well as your application's persistent volumes. So Valero covers both of these staple components that we've been talking about. As far as the Kubernetes objects go, Valero uses the Kubernetes Discovery API to access this data, so it's not talking directly to LCD to back up this information. And there are a few different reasons that we chose this approach. The first one is that on a lot of managed cloud provider Kubernetes platforms, you're not gonna have direct access to LCD. So in things like Google Kubernetes Engine or Azure's Kubernetes Engine as a user, you're not gonna have direct access to LCD, and so there's not actually a way to do a direct backup there, so we needed an alternate approach. Beyond that, though, the API gives us a lot of capabilities that we wouldn't have, even if we were doing direct at-cd backups. So using the API, we can do sort of fine-grained filtering and selection of the resources that we wanna back up. And this also means that when it comes time to restore, we can be much more precise about which applications, which particular resources we want to restore. And then using the API also enables us to kind of transform this data as we're doing backups or restores. So if you're in a migration scenario, you may actually need to make some changes to your YAML manifest as you're restoring data into a different cluster. And so using an API-driven approach gives us this opportunity to transform these resources while we're restoring them. And then Valero stores all this data in an object storage system, typically in the cloud. And so this makes the lifespan of these backups exist beyond the lifespan of your individual cluster so that if you lose a cluster or you need to migrate to an additional cluster, you have kind of a central location where that backup data exists. And then as far as persistent volume backup and restore, so Valero uses the cloud provider snapshot APIs. Valero has an extensible plug-in system so that additional providers can be added as supported Valero providers without needing to modify the core Valero code. But this is really our primary approach to backing up persistent volumes. We also have beta support for a file system level backup solution. We use an open source tool called RESTIC, which is a generic file level backup solution. And so this can be really useful if you're maybe running on a storage provider that doesn't have snapshot APIs. Maybe you just, Valero plug-in hasn't been developed yet for your provider. Or this is also really useful if you're doing cross provider migrations because you do get a generic provider agnostic format for your backups. And then beyond that, Valero has a variety of different features that help with your backup and restore workflows. So it supports both ad hoc and scheduled backups. As I mentioned before, there's some sophisticated filtering that you can do as you're backing up or restoring resources. So you can select specific namespaces to backups, specific resource types. You can also use label selectors to identify just a subset of resources that you want. And it's also possible to restore resources into different namespaces then you back them up. And so this can be really useful if you're trying to clone an existing environment, if you're trying to migrate something from maybe a dev environment to a QA environment that uses different namespaces. And then as I mentioned a minute ago, Valero is also very extensible. So we have a plug-in mechanism so that new object and block storage providers can be added to Valero. And we also support two additional types of plugins called backup and restore item actions. And these allow you as a user of Valero to write custom bits of logic that can actually transform resources as they're being backed up or restored. So this helps you handle any individual use cases that you may have that aren't covered by the core Valero platform. All right, next slide. So just a kind of a quick overview of what a standard Valero workflow might look like. So in this diagram, we have two different Kubernetes clusters, one on the left, one on the right. And starting on the left in Kubernetes cluster A, we're using the Valero CLI to take a backup. And if you look at the command text just above the cluster box there, you'll see we're creating a backup. We're including one specific namespace called example. And we're specifying a flag that indicates that we wanna snapshot our persistent volumes. And so the Valero controller that's running in that cluster will be communicating with the Kubernetes API there, fetching all the resources that need to be backed up, and then reaching out to the cloud provider's API to take snapshots of any persistent volumes that are being used by those resources and storing them in an object storage bucket in the cloud here in the center. And then on the right in cluster B, we have another instance of Valero. And so in this case, we're doing sort of a migration. So this cluster is configured to be able to restore backups from the same location where cluster A was creating them. So on the right, we're running a Valero CLI command to create a restore. We're specifying the name of the backup that was created from cluster A. And the Valero controller that's running in cluster B here will be reading the contents of that backup and applying all of the API resources to the Kubernetes API within cluster B. And additionally, we'll be taking the persistent volume snapshots, reconstituting volumes from those snapshots, and then attaching them correctly to the persistent volumes within cluster B. And so with that, this kind of sets us up for a couple of demos. So I'm going to share my screen. So the first thing I'll show here is a backup and restore within an individual cluster. And so what I have here is a cluster running in GKE. You can see from my command prompt here that it's labeled as cluster one. And within this cluster, I'm running an instance of the WordPress application that's using MySQL as the data store. And so the first thing I have showing here is that we actually have a service up to access that WordPress instance. So if I jump over to a browser window, you can see this is my WordPress blog. It's called Steve and I have a Hello CNCF blog post here. So jumping back to the terminal, the first thing I want to do is take a backup of this application as it exists now. So I'm going to use the Valero command line and I'm going to say Valero backup create. I'll just call it WP for WordPress. And specifically I want to include the WP namespace. So this is where all of my deployments and persistent volume claims are defined. And I'll also use the dash dash weight flag so that I know when this backup's done. And so Valero now is talking to that Kubernetes discovery API, accessing all of the resources within that namespace and also taking persistent volume snapshots of the volumes that are used. So the backup's complete now. We see that that ran pretty quickly. And so if I use Valero backup get, I can see here's our backup titled WP, it's completed. And I can also run Valero backup describe WP and I can use the dash dash details flag to see some additional information about it. So there's a variety of information that's shown here. In this top section, we just see some information on the kind of the spec for the backup that we created. So the main thing to see here is that we have a namespaces section that defines which namespaces were included in the backup. And since I used that flag while I created the backup we see that the, that was the included namespace. And then scrolling down towards the bottom, you'll see the section here which describes the persistent volumes that were backed up as part of this Valero backup. And so Valero found that there were two different PVCs that were being used by that WP namespace. And so for each one it created a snapshot using the GCP APIs. And so here we have references to those snapshot IDs and some additional metadata that'll be useful when we need to restore those. So the next thing I'm gonna do is sort of simulate a disaster. And so the most straightforward way to do this is just to delete the namespace, the WordPress namespace. And this really, this is gonna take a minute to delete but really you could imagine a number of different scenarios here. Maybe a user accidentally applied a configuration to this namespace that messed up your application definition and you want to be able to roll back to a previous state. Maybe there was another incident and your persistent volume data got corrupted and you want to restore. And so there are a number of different ways that you can think about this use case but in general we look at just deleting the namespace as kind of the primary issue that we would need to recover from. All right, so the namespace has been deleted and I'll just take a quick look. So we no longer have that WP namespace and I'll just pop back to my browser window and do a refresh and you'll see that we're not able to connect at this point in time. The service no longer exists, the pods no longer exist. So going back to the terminal, what I'd like to do now is actually restore that namespace using Valero. So I'll just take another look at the backups I have. So my one backup is called WP. So I will now say Valero restore create and I'll say from backup WP. And I'll kick that off. And now while this is running we can get a little bit of information on what's happening. So if I put a watch on the persistent volumes in this cluster, well, you'll see that they actually were already created. So that happened pretty quickly. You can see that they're created within the last few seconds. And so now if I look at the pods within the namespace WP, you'll see that they were both recreated within the last few seconds and they're now in the process of creating. And so if we give this a couple more seconds, the restore should be completed and the pods should be up and running. I can use the Valero restore get command to look at the status of my restore and we can see that it's completed. And let me take one more look and see if the pods are up and running. They are, they're up and running. And so now what I need to do is get the IP address for my service. This will change since we restored it. And so WordPress now has this external IP address. You can see it was just recreated under a minute ago. And so if I go back to the browser, pull up this IP, we're now connected back to a WordPress site. And so if I look here, we'll see that that post that was there in my previous copy of the site is back up. So we've been able to successfully restore from that incident. All right, so that's great for kind of restoring within a single cluster. But I also want to take a look at another scenario which is migrating an application, not just to a different cluster, but also to an entirely different cloud provider. And so to do this, we're going to use Valero's RESTIC integration, which I talked about a couple of minutes ago. So the first thing we need to do is we need to tell Valero that we want to use RESTIC to back up the persistent volumes rather than using GCP snapshots. And we need to use RESTIC in this scenario because I'm going to be migrating this application into an Azure cluster. And Azure has no way of knowing how to restore a GCP snapshot. So RESTIC will give us kind of that provider agnostic backup of our data so that we can restore it into another platform. So the first thing I'm going to do is actually, in order to tell Valero that I want to use RESTIC, I need to add some annotations to the WordPress and the MySQL pods. And so I have some YAML definitions here. So I'll just uncomment this annotation which tells Valero to use RESTIC to back up this persistent volume. So I'll do that from MySQL and also for WordPress. And then I'm going to go back here and just reapply those manifests. We'll actually get a new set of pods that are scheduled. So just put a watch on that so that I know when those are completed. But so what's going to happen from here is that we'll take another backup and this one will actually use RESTIC to back up those persistent volumes. And then I'll switch over to my Azure cluster and show you how to configure it to be able to pull the backup from the place where this GKE cluster is currently putting backups. All right, the new pods are coming up. So I'll just give it a couple more seconds and we should be ready to take a backup. Sometimes this goes a little faster, sometimes it goes a little slower. So apologies for the delay there, but it looks like the pods are up and running now. So I'm just going to clear my screen. So at this point, I'll take a new backup. So I'm going to use a similar command to last time. So I'll say Valero backup create. This time I'll call it WP-RESTIC so that we know we're using RESTIC backups. And again, I just want the WP namespace, all the resources in there and I'll use the dash dash weight flag as well. So Valero is going through a similar process of talking to the Kubernetes Discovery API at this point, but instead of using the cloud provider snapshot APIs, we'll actually be executing RESTIC to take file level backups with the contents of these persistent volumes. All right, so we have a backup now. I'm going to actually use this command that's given to me here to look at some of the details of those. I'll use the details flag. And so the most important thing to notice that's different here is on the bottom instead of having a list of persistent volume snapshots, we actually have a section called RESTIC backups which tells us the pods and the volumes within those pods that were backed up. All right, so we have our backup now. So now we want to switch over to the Azure cluster that we're going to restore into and get it configured to be ready to restore. So you can see here that I'm on cluster to Azure and if we look at the namespaces, we'll see I don't have a WordPress namespace right now. So the first thing I need to do is create what's called a backup storage location which tells this cluster where these backups are stored. And so to do that, I can say Valero backup location create. I'm going to call it GCP. Need to give it a provider, which is GCP. I also need to tell it the name of the bucket which is Steve Valero backups. And then the final thing which is a new feature that hasn't been released yet but will be coming out in 1.1 is an access mode. And so I want to set this location up to be read only from the perspective of the Azure cluster because I only want to be able to restore these backups. I don't want to be able to create new backups in this bucket or to expire, delete any of the backups that are in there. So this can be really useful for a migration scenario. So, and then the other thing I need to do is if I switch back to cluster one, if I look at the storage classes that I have in this cluster, we can see that there's just one and it's called standard. And then if I go back to my Azure cluster where I want to restore and see that I have three storage classes but none of them are called standard. And so when I do a restore, I actually want to change the storage class of the persistent volume so that they're no longer called standard but instead they're called managed premium. And so we have another new feature which will be shipping in 1.1 which allows you to do this really easily. And so it's implemented as a plugin for Valero but there's a config map which will be fully documented with 1.1 which allows you to specify remappings of storage classes. And so what we want to do here is just in the data section for this config map add a new mapping from the standard storage class to the managed premium storage class. So I'll save that. So we should be set up now. And so since we added that backup location pointing to the GCP bucket, we should actually be able to see now if I run a Valero backup get those two backups that we just created from the GCP cluster. And so this WP rustic backup is the one that we're going to want to restore here. So now I'll execute a restore. So I'll say Valero restore create from backup WP dash rustic and I'll start that. And so now the Valero running in this Azure cluster will be downloading that backup and going through a restore process. So we can take a look at what's going on with the persistent volumes while this restore runs. All right, so we see that we just actually got the two persistent volumes created in this cluster. And so if you look here under the storage class you'll see that those were successfully remapped. So they now have the managed premium storage class within the Azure cluster. And then take a look at the pods that are running in the WordPress namespace. So these were just created and you can see they're currently running in a NIC container. So this is part of the rustic restore process that Valero is executing. Now this may take a minute or two. In the Azure cluster were dynamically provisioning empty volumes were then running the rustic restore process to actually take the contents of those persistent volumes from the GCP cluster and restoring them into these new empty PVs. So this may take a minute or two. So just bear with it. Just going back to the storage class mapping. This new feature that we added in 1.1 is something that we heard a lot from users particularly for this migration scenario because a lot of times as you're migrating across different providers the storage class names don't line up. And so you need a way to be able to modify those as they're coming into the new provider. So we're pretty excited to be able to support this natively going forward. All right, so let's see. So it looks like one of our pods is up and running at this point. It looks like the MySQL pod is running and it looks like the WordPress pod is still restoring. So we can also use this Valero restore describe command to get a sense of what's going on with the restore. And so you can see that the MySQL restore has completed and the one for the WordPress itself is still pending. So it looks like this one's just taking a little bit longer to provision the disk, but it's finally coming up. So all right, so now that WordPress pod is up and running. And if I just take one more look at the status of the restore we'll see that it's completed. So that's a good sign. And so now the last thing that I need to do is actually look at the services in the WordPress namespace because I need to get the IP address for this new service that's running in Azure. And so I'll take that and just switch over to a browser window and I'll pull it up and boom, there we go. So we have our WordPress site up and running. We have our CNCF blog post and this is now being hosted from Azure. So yeah, so that pretty much concludes the demo. I'm gonna stop sharing my screen. Tom, do you wanna reshare the presentation? We should see the presentation again. And yeah, can you just jump to the next slide? There we go. All right, so just quickly. So as I mentioned, we're currently on Valero 1.0. We added a handful of stability and usability features in that release. So one of the main ones was the Valero install command to make it really easy to get up and running with Valero. We also did some work on the community-supported Helm chart. So we're now officially supporting the Helm chart going forward as an installation mechanism. And then we also improved our plugin integration so we improved stability and the developer experience there. And our RESTIC integration for doing file-level backups is considered beta as a 1.0. We're currently working on 1.1. So this should be coming out by the end of the summer. We have a handful of different features here. So we have been doing some prototyping of an integration between Valero and the CSI snapshot APIs. So the CSI snapshot APIs are alpha. And so our integration is alpha as well, but we're exploring this and looking at how to support it going forward. And then we're also making a number of RESTIC stability improvements with an eye towards moving this integration to be generally available. And we've also used added the support for read-only storage locations, which I demoed. And we're adding better support for defining resource usage requests and limits for the Valero pods that are running. And so with that, I'll hand it back to Tom to wrap up. Okay, fantastic, Steve. Thank you so much. The demo was really great and glad nothing got stuck. So that was a live demo, folks. So one thing I wanted to draw your attention to is we're very involved with community. We encourage community involvement with Valero. And one of the things that we're doing right now is it's launching literally today is we're doing a data protection survey. So this is a data protection survey for cloud-native backup. So we're really interested in getting your feedback on how you're thinking about backup and restore, what providers you're using, et cetera. You can find links to that survey at Valero.io slash survey. We will be sharing this back with the community. So once we get the results and we tally those, we'll share those back and make it available for everyone to data mine and take a look at. So it would be a great opportunity for us to learn just kind of what's going on in the community and how people are thinking about backup. And clearly from a Valero standpoint, we'll be taking a look at it from the perspective of trying to find out what the best way to prioritize our feature development going forward. So please look for that survey. And in a few weeks, we will close the survey and tally everything back and post that back on the Valero.io website. So speaking of the Valero.io website, that is where you can find Valero and any blog posts and other valuable links like to our GitHub. You can find us on GitHub at Heptio Valero. Also, please follow us on Twitter at project Valero for the latest news. And I'm certain that we'll be tweeting out about the survey and also once the survey results are posted, so go ahead and follow Project Valero there for all the latest news on what's going on. And in terms of getting questions answered as you're using Valero or thinking about using Valero, please join us on the Kubernetes Slack. You can find us at the channel Valero. And all of us are on a daily basis and looking forward to interacting with everyone and with the community. So the other thing you can do is you can join our mailing list at Project Valero, which is on Google Groups. And the Valero, we also have community meetings twice a month. We just had one a couple of days ago. They're the first and third Tuesday of every month. And you can find out more about that on our GitHub at Heptio, our GitHub Heptio Valero community. So with that, I think we have a little bit of time for some Q and A. I know that Carlicia has been answering a bunch of questions on the chat. So that's fantastic. Carlicia, are there questions that should be answered verbally at this point? Yeah, a lot of it, if you could answer these questions, we have some good ones. So we have about eight minutes in our lobby. If we could blow through all of them, we have about six. I'm going to try to start with the straightforward first. So if we need time for discussion, we have a little bit of time. So as I say that, I get lost. Okay, so what are we using CSI for? Yeah, I can tackle that one. So we're looking at integrating Valero with the snapshot APIs that are being added to CSI. And so the snapshot APIs mean that for anyone, any storage provider who's implementing CSI, if they implement the snapshot API, then any consumer of CSI will be able to take advantage of that. And so we're looking at basically adding integration between Valero and those snapshot APIs so that if you're using CSI to provision your volumes within your clusters, then Valero will automatically be able to take snapshots of those volumes using that API as part of its backups. Yeah, I would just add to that. You know, in a CSI world, it vastly increases the number of storage providers that can be, can underpin your Kubernetes clusters, which also means that Vasily expands the number of storage providers that we can provide backup for. So CSI is very promising. We're following it very closely. Great, next questions. Actually, Tom, would you mind backing up to the slide that has the contact information? I think that would be a more useful thing to stare at. Okay, sure. Cool. All right, next question. How do we do backup and restore on-prem? Yeah, so on-prem there's a number of ways to do it. First of all, backups are kept in an object store. And so on-prem you can run an S3 compatible object store or if your storage provider has an S3 compatible object store, you can store your backup data there. You can also install an open source project like Minio, for instance, to provide that backup location and run that either in your cluster or alongside a cluster in your on-prem environment. Okay. Next one. How does Villero do a restore? How does Villero ensure the order of that the Kubernetes objects get created? For example, if user accounts needs to be created first, then the pods. Yeah, Steve, you wanna feel that? Yeah, I'll take that. So there are a set of kind of standard orders that need to be respected during a restore. So for example, you typically wanna restore a persistent volume before a persistent volume claim. And so Villero is aware of those, but it's also user-controllable. So there is a flag to the Villero server command so that if you have any sort of custom orderings that you need to follow, that you can specify those there. And then I guess one other angle to that is that if you're writing a plugin for Villero or restore item action to customize how Villero restores a resource, you're able to have that plugin kind of return additional related resources that should be restored. And so using that, you can have some additional control over how things are restored. So happy to dig into that some more in detail offline, but that's kind of the overview. Cool. Next one, does Villero support OpenShift specific objects, for example, security context constraints? Yeah, so Villero is definitely capable of backing up custom resources. So to the extent that those are defined as custom resources within your cluster, those will be available through the Discovery API and will be backed up and restored. Yeah. There's a good product related question. The question is, is there plan to make Villero more complete by having features to support snapshots, retention policy and templating? And as far as templating, it would be so that you'll be known what it's being backed up and the steps that needs to be performed in the container before the backup is being created. Yeah, actually. So we have features to solve it for a number of those. We have for deciding what is included or excluded from a backup. We do have included and exclude actions that you can use. We also support snapshotting, both on cloud providers today, which some of the features that Steve showed in the first example of the demo today. Also, our CSI support is looking specifically at doing snapshotting, for instance. So I think, and Carlisle, was there another thing besides snapshotting? Retention policy. Yeah, so retention policy is really good. So you do have the ability within Villero to put what's called a TTL, or Time to Live, on your backups. And Villero will, when it backs those things up, will automatically do garbage collection and remove those ancient backups so that it's effectively cleaning up your backup targets as it goes along. So if you wanted to save something for 30 days, that's the default TTL. If you wanted to save it for longer, seven years, or a year, or something like that, you just place that TTL on the backup and Villero will take care of managing that retention for you. All right, we have one more minute and I have two questions. First, will later versions always support restore of old backups? How is this assured? I missed the middle of that. Will later versions always support restore of old backups? How is that assured? Yeah, so I would say within a major version, so all 1.x versions will support restores of any backup created with, say, 1.0. It's possible there could be breaking changes over major versions, but I think we'll always provide data migration scripts if that's the case so that users always have a path to be able to restore their older backups. Yeah, we've done that in the past. We have... Yeah, I was gonna say, I think we did that with 1.0, didn't we? Yeah. All right, last question real quick. How frequently should we be backing up? Wow, yeah, so that's really gonna be dependent on your own domain knowledge of your applications. So I would encourage everyone to take a look at RTO and RPO, which is the recovery time objective and recovery point objective. Wiki PDF, you can find out on Wiki, you can find those things and use those concepts to make a backup plan that suits your own particular use case and then use Valero schedules to effectively codify that policy and Valero will adhere to that and be able to manage to that particular yourself set of recovery time objectives. Let's have questions, thank you. Okay, I wanna thank everybody very much for attending today's webinar. It's been fantastic for us to get the word out. Please take the survey. Love to hear more from the community on how they're thinking about backup. Visit us on Valero.io and for additional questions, you can come visit us on the Kubernetes Slack channel at Valero on the Kubernetes Slack. We'd love to interact with each of you. Thank you very much for your time today. Thanks a lot everyone. Goodbye. Great, so thank you guys for joining us today. The webinar recording and slides will be online later today. We are looking forward to seeing you at a future CNCF webinar. Have a great day, thanks you guys.