 So, I will introduce you to the CSI storage driver for LVM-based local storage. This driver has the capacity-aware dynamic volume provisioning feature. So, it's a agenda. First, I will introduce you to the motivation to develop our storage driver for LVM-based local storage. Next, I will introduce what is local storage and how local storage works. And, at last, I will show you the next plan of local storage. So, first, motivation. So, let me introduce our company Cybos. Cybos is a leading cloud service provider in Japan. And we provide software that support teamwork. So, this kind of software is also known as groupware in Japan. So, Cybos has a Kubernetes cluster. It's an on-premise Kubernetes cluster in our data center. So, it's under development, but we will use this cluster to host our service after two or three years later. In this cluster, we also have a storage system on top of this Kubernetes cluster. And we have two kinds of storage. The first one is distributed block and object storage. And we also decided to use Rook and Safe for the distributed storage. The second one is a local fast storage. In this case, fast means local NVM-AESST storage. So, at the beginning of our development, we needed to select the best storage driver for local fast storage. So, let me introduce the requirement for local storage, Cybos local storage. So, we have two requirements. First, users can create arbitrary sized volumes. Typically, local storage is disks or partitions, and they are fixed-sized. So, but fixed-sized disks or partitions are, in other words, fixed-sized volumes are inconvenient, especially for Kubernetes users. And the second requirement is volume should be spread over nodes based on free storage capacity. In other words, use storage capacity for each node, Emily, to utilize the full local storage. So, we should decide, we should decide and to use what was the best storage driver, what is the best storage driver to use our Kubernetes cluster. So, unfortunately, there was no CSI driver that met all our requirements. So, we decided to create a new CSI driver, Topolvm, a very simple story. There's no driver we create driver. So, next, I'll introduce what is Topolvm. About arbitrary volume size, it's one requirement, it's a first requirement. Topolvm deals with LVM volume groups prepared on each node. And Topolvm creates an LVM logical volume for each persistent volume resource. And probably you are a bit confused because what PV means. But in this case, PV is, in this case, in this slide, PV is not means LVM's physical volume, but persistent volume. So, let's see the top side of this three year. It means the Kubernetes resource, there are three persistent volume. And the bottom size means hardware configurations. There are node one and node two, two nodes. And there are one VG for each node. And this volume groups are prepared by administrators. Topolvm doesn't touch volume groups. And there are logical volumes for each persistent volume. So, let's see the port scheduling and volume provisioning of Topolvm. So, let's see the top side of Kubernetes resources. There are three Kubernetes resources, port, pvc, and storage graphs. The first, port consumes a pvc that request 10 gigabytes. And this pvc points to the storage graphs for Topolvm. That means this volume is created through the Topolvm. And there are three nodes, node one, node two, and node zero has 15 gigabytes of free VG storage. And node one is 100, and node two are just five gigabytes, respectively. So, let's assume in this case, I have two assumptions. So, pvc is dynamic volume provisioned in this case. So, it means the pvc corresponding to pvc will be created at port scheduling. So, it means in this figure, pvc is not exist yet. So, the next assumption is that storage class volume binding mode. In this case, volume binding mode should be wait for first consumer. So, when we create the pvc, pvc is not created. So, pvc is bound to pvc at port scheduling. So, both pvc and pvc are created on port scheduling. So, of course, Topolvm also supports not dynamic, so static volume provisioning and volume binding mode immediate. But it's the best choice to use dynamic volume provisioning and wait for first consumer. So, as a result of Topolvm volume management and port scheduling, the port is scheduled to the node having the largest free VG space as possible. In this case, node 1. Node 1 has 100 gigabytes. It's the largest value among three nodes. And the volume is provisioned on the same node as the port. Volume and port are scheduled to node 1, both scheduled to both 1. And the port is on node 1 and LV is created on this volume group on node 1. And it's bound to this port and pvc and pvc and the port resource. Topolvm, that was the main feature of Topolvm, and Topolvm also has many other features. So, first one is the volume type. For file system volumes, we on Topolvm support EXT4, EXT4S, and PDI4S. So, it covers most main three file systems in Linux. So, and Topolvm also supports low block volume. In our case, Topolvm is used for getting storage for block-safe distributed storage. So, we also, Topolvm also supports other features like generic FMR volume and volume expansion and shin volumes. Shin volumes also supports snapshot and shin snapshot and shin clone features. Interestingly, this shin volume feature was not created by cyborgs but was created by Red Hat. So, they need to implement this feature. So, how about community of Topolvm? There are many known cyborgs users and developers. I guess there are over 100 or 200 people, the Topolvm user. And there are over 10 people, 10 known cyborgs people are developing Topolvm. And some companies use Topolvm in their products. So, it can be said that Topolvm is used in the production environment, not only in cyborgs but also in other companies. Especially, one good example is about Red Hat. Red Hat OpenShift embed Topolvm in their OpenShift. So, one very interesting story is that the first patch from Red Hat is about to support their mainframe system in Topolvm. All my team members are very surprised because we don't have mainframe system. How to test this patch? So, after discussion with Topolvm maintainer and Red Hat developers, we decided to merge this patch to tell the truth. We don't test it, we can't test it, but probably Red Hat test this patch in their environment. No problem, yes. So, it means that their OpenShift and their Kubernetes cluster can be run on top of mainframe, very surprising. So, it also means that there is a Topolvm container working for mainframe. I didn't touch it. So, next topic is how Topolvm works. So, there are two challenges to implement the Topolvm port scheduling and volume provisioning. So, one is the scheduler port to the node having as large free space as possible. To accomplish this purpose, we decided to use the scheduler extender of Kubernetes and its official Kubernetes feature. And the second challenge is provisioning the volume on the same node as the port. So, we use CSI topology, it's also an official feature of Kubernetes. So, but these features are optional. So, the user or in this case Topolvm or CSI driver can implement their logic to use scheduler extender and CSI topology. So, I'll introduce the scheduler extender. So, let's see the bottom side of this slide. It means the step to schedule a port into a node. It's the Kubernetes scheduler's work. The first, just start port scheduling. And in the second step, filter out node that doesn't match some conditions defined by Kubernetes scheduler. And in the third step, Topolvm, no, no, no. Kubernetes scheduler squaring the remaining node. It's not filtered out by the Kubernetes scheduler in step two. And the fourth and last step is the port is scheduled to a node that has the highest score. And in this case, users can extend the Kubernetes scheduler by scheduler extender. It's implemented as a webhook. So, this webhook can implement two logic. The one is in the filtering step. And the second is in the squaring steps. The first one is filtered node which don't have some conditions defined by scheduler extender. And the second one is at the factor of squaring node. It's also defined by this scheduler extender. So, how about Topolvm scheduler extender? The first step is about filtering. Filter out node which don't have enough free space. And the second step is at the factor of squaring to prefer node having large free bridge space. To implement these logic, the two parameters of the Topolvm scheduler are necessary. So, the two parameters, first one is free bridge space for each node. And the second one is total requested Topolvm volume size for each port, which using the Topolvm volumes. So, Topolvm manages annotations for these parameters in node resource and port resource. So, for free bridge space for each node, now Kubernetes has storage capacity tracking features. This feature can also be used for extending the scheduler, but it can only be used for filtering. It cannot be used for scoring. So, the Topolvm also can use storage capacity tracking, but for now, the Topolvm original annotation is the best choice. So, about CSI Topology. CSI Topology is a feature of Kubernetes, and this feature scheduler port to one of the nodes where its volumes are available. And where volumes are available means zone or node or something. So, this feature is used for zone local storage or node local storage and so on. So, Topolvm, in Topolvm, Topolvm creates a volume on the same node as the corresponding port. So, let's see the example of scheduling a port using the Topolvm volumes. So, top side is Kubernetes resources, and the bottom side is hardware configuration. There are three nodes, node 0, 1, 2, and node 0 has 15 gigabytes free bridge space, and node 1 has 100 gigabytes, and node 2 has 5 gigabytes. And there are three node resources that exist for each physical node. And this node has, oh, no, no, no. This node has annotations that are named capacity. So, this annotation means real VG volumes. So, this annotation is acted by Topolvm. So, let's create both port and PPC resources. So, then PPC requests 10 gigabytes volumes from Topolvm, and the port is created to consume this PPC. And then, when creating this port resource, Topolvm will be hooked up at an annotation named capacity. This capacity, this annotation means total amount of volume size consumed by this port. In this case, just 10 gigabytes. So, next step, KubeScheduler started to run its filtering process. The first filtering process traversed the node resources and read capacity annotation, and found that node 2 doesn't have enough free bridge space. So, then, in filtering process, node 2 was filtered out. Node 2 is filtered out. So, next step is squaring. There are two remaining nodes, node 1 and node 2, and the Topolvm scheduler traversed two nodes, and found that the largest capacity is 100 gigabytes, and node 1 is the best choice. And the schedule this port to node 1. And the next step, thanks to CSI topology, the volume is provisioned to this node, node 1. First, the port is scheduled to node 1, and the corresponding logical volume is created on top of node 1, and this is bound to port and pvc. So, the last topic is about the next program of Topolvm. So, next, we have two plans. The first one is implement the Kubernetes official capacity-aware port scheduling. So, it's because setting up a schedule like standard is a bit difficult. I said a bit difficult, but too difficult. Very difficult. So, we'd like to prepare CAP to implement the capacity-aware features to mitigate this situation, because there are many, many, many issues to claim. I failed to schedule like standard or configure a schedule like standard or something. That's the worst case. We cannot use schedule like standard at all due to the limitation of cloud provider or something. So, the last plan is to donate Topolvm project to CNCF because now Topolvm is created by many companies or many individual peoples. I and all Topolvm member and sideboards consider it's the best choice, I believe. So, the conclusion is that Topolvm is an LVM-based CSI driver. Volumes and corresponding ports are evenly spread for each node. It's accomplished by the schedule like standard and CSI topology Kubernetes features. So, we will come new users and contributions. So, do you have any questions? Thank you for your good presentation. So, I have one question. Why did your company decided to develop it as an OSS? Not only Topolvm, but also... Topolvm? This software. It's just because there is no suitable driver and we are able to create any software. Yes, we have many skillful engineers. Is there any benefit to create to develop it as an OSS? Because you can develop it as a closed source code? Ah, okay. I understood what you said. So, he said why we created Topolvm and publish it as an open source, right? Yes, right. Okay. It's because my team first decided to publish all our product, all our product about the infrastructure in public as an OSS. Because it's for many purposes. The first purpose is if someone thinks our product is very easy, very good. It's very happy for us. And probably, as you know, the software about the infrastructure is not the source of our benefit. So, it's not difficult to publish our products. And the last reason is if we publish our software to OSS, someone read our source code. And we should write this source code to public. And we can write very good code than the in-house software. It's very good. And I recommend you to write your published source code as OSS. Okay. Thanks. I understand. Any other questions? Nothing? Okay. Thank you very much.