 Hello everyone, thanks for joining us today. Today, Xin and I will give a deep dive and introduction to the data protection working group. My name is Zhang Qian, exercise please. I am a software engineer from Google. Hello everyone, my name is Xin Yang. I work at VMware in the cloud storage team. I'm also a co-chair in Kubernetes 6 storage and co-chair of the data production wing group with Xiang Qian. Thank you, Xin. Next slide please. Today, we are going to go through a couple of topics. First, we will reiterate a little bit what is the motivation of this data protection working group. And then we'll go a little bit to recognize over who are the organizations that get involved in this effort. And then we talk about what we think about data protection is in the Kubernetes context. And then what are the existing building blocks to support data protection in Kubernetes? And what are we missing for the building blocks to support this feature better and how to get involved? Next slide please. There were not sufficient fundamental Kubernetes building blocks to easily allow users to do backup and snapshot of the stateful applications as of today. They want operations for deployment, management and rollout of stateful workload are well supported. There are volume operations either through CSI driver or entry drivers. And they are workload APIs, deployment stateful set, et cetera, to support that. With more and more stateful workload moving to Kubernetes, the desire of providing fundamental building blocks to allow users protect the data is getting headed. GitOps could potentially be used for rolling back of stateful workloads. It is still insufficient for stateful workloads as the data cannot easily be managed by GitOps. This working group has been built, targeted at designing and implementing Kubernetes native building blocks to allow backup windows to quickly build backup solutions for the users. That's to support the data operations for data protection specifically. And this working group has two major stakeholders, SIG apps and SIG storage and potentially SIG node. Next slide please. So who are all in world? This is a weekly meeting and those are just probably incomplete list of companies that are supporting this initiative. We saw a lot of contributions from storage companies, cloud providers, backup windows, et cetera, et cetera. And thanks to all the contributors, we are making good progress over there. Next slide please. So what exactly is data protection in Kubernetes? The main purpose of data protection is to make sure that an application and its data can be recovered to a previously preserved state. We call it backup for the state after any corruption or losses. That can include badly rolled out software costs, data corruption or it can be disaster recovery when you lost the whole data center, et cetera, et cetera. In the Kubernetes context, it mainly involves two pieces of data. One is the API resources that describes your workload in the Kubernetes cluster and the other is the persistent volume data that application uses. This is a complicated and layered problem. Likely including backup and recovery at a couple of levels. Persistent volume level, application level and the cost level to satisfy different goals. Next slide please. Part of this working group's charter or emissions is to define a list of Kubernetes native constructs to enable protection at different levels. At the volume level, there are volume snapshot features and we're looking into volume backup features, volume restore. In the application level, we are looking to define what consists of your application and how to acquire an application such that application-consistent snapshot can be achieved. She will give more details later on and just glance in through this concept. The cost level, what if I want to backup the whole cluster and being able to restore into a completely different cluster. Next slide please. Here, this diagram shows you a pretty typical stateful application backup workflow. A user generally starts a backup either with your COI or something. It has two missions to do. One is to backup the Kubernetes resources and get them archived or exported to an external externally managed storage, typically object storage systems. And then the other piece, the data backup. As you may aware or some of the stateful workloads like MySQL, Postgres, Radius, etc., they have this so-called application-native data dump. So what they're capable of is an COI tool which allows you to download the current state of the application into a local file. And this local file can be later on used for restoration. In this case, if such application has a native data dump, a user can choose to use the native dump and export the dump file into the remote bucket, object bucket. If there's not, there's a controller coordinate backup workflow that typically will happen. Where does it typically quiet the application and take a wild snapshot or backup using the Kubernetes construct and unquiet the application? The reason why you want to quiet your application is to make sure during the process of backup the data is being maintained in the consistent application level. Next slide, please. The restore workflow is the reverse. Users start to import the backup, including both the Kubernetes resource piece as well as the data piece and the PVC data needs most likely to be handled specially. PVC and the PV data restoration, again, it branches into two pieces. If the backup turns out to be a native application dump, then use the native application tools to restore from the dump, otherwise just rehydrate the PVC from the volume snapshot or volume backup. Next slide, please. So we talk about the workflows. What exactly is there as of today to support these workflows? For application, they reach sets of workload APIs, state, for-set, deployment, demand-set, et cetera. There's also an application CRD, which groups a set of resources together and abandon them as an application. In the storage piece, we have volume snapshot feature, which is already G80 and 1.21. Next slide, please. So how does this work? How does this building blocks fit into the workflow? As you might see the green box over there, workload APIs and SIG apps application, CRDs can help the backup system understand what are the Kubernetes resources this backup should contain. Next slide, please. Volume snapshot feature majorly sits in the controller or coordinate workflow where the controller can literally create a volume snapshot object, and this will trigger a snapshot over the system of the twice cookies called. Next slide, please. In the restoration workflow, the same thing. Volume snapshot can be directly used to rehydrate PVC from it. Next slide, please. We talk about existing. So what are missing? This is just a short list of missing features. We're looking to design a build. Volume backup, volume snapshots. In some of the windows, they are stored externally. In some of the windows, they're actually locally stored in the primary storage system. Backup repository. Where do we put or where do we put our backup manifest? Quiet and quiet hooks. How do you trigger a command in an application such that it can quiet itself and unquiet itself? Application snapshot and backup. This is basically an application level of extraction, including controller that allows you to do application consistency, consistent snapshot and backup. And there are more. Next slide, please. This is a slightly busy slide. If we take a look, the boxes in orange are the missing blocks. And boxes in yellow are the ones that are under design or work in progress. Boxes in green or blue. Boxes in green are existing ones. So the volume backup, just like volume snapshot, fits into the picture of controller coordinated backup process. And a backup repository serves as the storage place location for when you try to expose the backup. An application backup is a higher level construct, organize all this and coordinate in all this process. Next slide, please. In the restore workflow, a pretty similar thing. Backup repository this time serves as a source of the backup to be pulled from. And there's a COSI cap, which is to build a generic interface to all the object providers, to object providers, which is in progress. Application restore, in this case, plays a role of orchestrating the restoration process from the application level. And the volume backup again falls into the same box as volume snapshot. I will not go too much detail into this and I'll hand it over to Xin to cover the details of those missing blocks. Xin, please. Thanks, Zhang Xin. So I'm going to talk about what are the missing building blocks. The first missing building block we identified is volume backup. We needed this because we need to extract data to our secondary storage. We've already got a volume snapshot API, but there's no explicit definition made in the design to have snapshots stored on different backup devices separate from the primary storage. For some cloud providers, a snapshot is actually a backup that is uploaded to an object store in a cloud. However, for most other storage vendors, a snapshot is locally stored alongside the volume on the primary storage. So without a volume backup API, backup vendors can handle different snapshot semantics differently. For storage systems that take local snapshots, you use the volume snapshot API to take the snapshot and then have a data mover to upload a snapshot to your backup target. For storage systems that upload snapshots to an object store automatically, the snapshot can be treated as a backup. Alternatively, backup vendor can also move the snapshot to a different backup target. We just started discussions about this in the working group. In this diagram, volume backup is next to volume snapshot. We put it in an orange box to indicate that it is a missing Kubernetes component. We have started discussions about it, but there's no concrete design yet. The next missing building block is CBT and change file lists. Without the CBT change block tracking and change the file list, backup vendors have to do full backups all the time. This is not space efficient, takes longer to complete and needs more bandwidth. Another use case for CBT is snapshot-based replication where you take snapshots periodically and they replicate it to another site for faster recovery purpose. So what are the alternatives? Without CBT, we can either do full backups or we can call each storage API individually to retrieve CBT, which is highly inefficient. We are currently doing design for the CBT feature. As shown in this diagram, the CBT box is next to volume backup and volume snapshot is used to make backups more efficiently. It is a yellow box now as we are currently working on the design. The third missing building block is the backup repository. Backup repository is a location or repo to store data. This can be an object store in a cloud, on-prem storage location or a ANSI-based solution. There are two types of data to be backed up that we need at the restore time. The first one is Kubernetes cluster metadata. The second one is snapshot data. We need to back them up and restore them in a backup repository. Currently there is a project for object store backup repository that is object bucket provisioning or COSI. This introduces object storage Kubernetes API to support orchestration of object store operations for Kubernetes workloads. This brings object storage as the first class citizen in Kubernetes, just like file and block storage. It also introduces container object storage interface or COSI as a set of GRP interfaces for provisioning object buckets. Kubernetes COSI is already a sub-project in six storage and it is a pretty active project. It has weekly design meetings and stand-up meetings. It is targeting afar in 1.22 release. In this diagram, COSI is in a yellow box indicating it is a working progress Kubernetes component. This is an object store backup repository. It can be used to export backup and store the data. Let's look at the restore. COSI is used to import backup data at restore time. The next missing building block is the volume populator. Currently, we can only create a PVC for another PVC or for a volume snapshot. But what if the backup data is stored in a backup repository such as an object store? The volume populator feature allows us to provision a PVC for an external data source such as a backup repository. In addition, it allows us to dynamically provision a PVC having data populated from the backup repository and honor the wait for first consumer volume binding mode during restore to ensure that volume is placed at the right node where part is scheduled. There isn't any volume data source alpha feature gate which was introduced in 1.18 design and part tapping for volume populated implementation or in progress. New repos are created. One repo is for Shell library for use by volume populators. The other repo is for the controller responsible for validating PVC data sources. This is targeting beta in 1.22 release. As shown in this diagram the volume populator is needed at restore time. Volume populator is in a yellow box indicating that it is a working progress Kubernetes component. It is used to rehydrate PVC from a backup repository during restore. The next one is quiet and unquiet hooks. We needed these hooks to quiet application before taking a snapshot and unquiet afterwards to ensure application consistency. We investigated how quiet and unquiet hooks work in different types of workloads. They have very different semantics. We want to design a generic mechanism to run commands in containers but we want to mention that application specific semantics is out of scope. We currently have a proposal called container modifier. It is submitted and it is being reviewed. Let's take a closer look of the container modifier cap. This cap proposes to introduce the mechanism to notify a selected set of parts to run pre-specified commands defined in line in those parts specification. This is achieved by adding an optional modifier field in the container to allow users to specify commands or signals which can be executed or sent. We are using a core API object pod notification to trigger specific command signals specified in a pod. We plan to introduce the changes in multiple phases. In phase one, we propose to add an optional field notifiers which is a list of container notifiers in the container. Add a container notifier handler which defines a command at a core API pod notification which defines a container to trigger the execution of container notifier in a pod and add a single trusted controller a pod notification controller to watch pod notification resources executed commands and update their statuses. In phase two, we propose to add an API called a notification and a controller which processes notification resources. In the notification spec, there is label selector to select pods. There is also a pod selection policy. If policy is all pods, it means the controller will keep processing all pods matching the label until the notification object is deleted. If policy is pre existing pods only, it means the controller will only process pods created before the notification object is created. In phase two, we also propose to add an inline pod definition for signals and allow the API object to send a request to trigger delivering of those signals and also move logic in the pod notification controller into Qubit so that Qubit will be watching pod notification objects, runs the commands and update statuses of pod notifications. In phase three, a probe may be added if needed as an inline pod definition to verify the signal is deleted or whether the command is run and results in desired outcome. Let's take a look at this diagram. Continuity fire is mainly used at a backup time to do choirs before taking a snapshot and unquires afterwards. The next one is consistent group snapshot. We talked about the container modifier proposal which tries to ensure application consistency. If you can't choirs the application or if the application choirs is too expensive so you want to do it less frequently but still want to be able to take a crash consistent snapshot more frequently. Also an application may require the snapshots from multiple volumes to be taken at the same point in time. That's when consistent group snapshot comes into the picture. There's a cap on the group and group snapshot. It proposes to introduce a new volume group CRD that groups multiple volumes together and a new group snapshot CRD that supports taking a snapshot of all volumes in a group to ensure right order consistency. This cap is being reviewed. Let's look at this diagram here. We don't have content modifier to do choirs here but we have a consistent group here that facilitates the creation of a snapshot of multiple volumes in the same group to ensure right order consistency. We have snapshot APIs for individual volumes but what about protecting a stateful application? There's a cap submitted that proposes the Kubernetes API that defines the notion of stateful applications and defines how to run operations on those stateful applications such as snapshots, backup and restore. This is still in a very early design stage. In this diagram, application backup handles the backup of our stateful application. It can leverage content modifier to choirs and use COSI as the backup repository. Similarly here we have application restore that handles the restore of a stateful application. So this are all the missing building blocks that we have identified and are working on. Next, I'm going to talk about how to get involved. As discussed in previous slides, this or in group is working on identifying missing functionalities in supporting data production in Kubernetes and trying to figure out how to fill those gaps. We are also working on a white paper on data production workflow. We have bi-weekly meetings on Wednesdays. If you are interested, please join our meetings. We also have a mailing list and a Slack channel as shown here. This is the end of the session. Thank you all for attending. If you have any questions, please don't hesitate to retell to us. Thank you.