 Okay, we, I think, all of our audience has arrived. Our topic today is HDFS CSI, and also how can we speed up the Kubernetes in on-premise environment. Of the big data clusters, first I want to introduce myself, and here is also my colleague Chen Yi. And she is the software engineer from Tencent and Apache Hadoop Committer and PMC. I was working for a long time on the Apache Hadoop, HDFS and also. And my name is Zhu Jinping, and I have done a lot on the massive data storage and computing. I'm the chair of Tencent Open Source Committee and also the Apache ASF member and also the Apache Hadoop Committer and PMC. First, I want to talk about the big data and its history. Maybe you are the experts on Kubernetes, and I'm going to talk about the workload challenges on the big data and also different interface and solutions. And also some options for you to choose from. First is the history of big data. As you may know, it's first started from GFS of Google and Magnetist and also started from 2003. And Abduce Establishers, the founder of it, has proposed a Hadoop project from its original project and later is developed into the Apache Hadoop and also Hype as well. And he has built up the whole ecosystem of Hadoop. And in 2018, there were a lot of releases called Dairy Company Establish. And a lot of the companies that is related to this field also established in 2012, 14 and 17 has pushed the development of big data into the 1.0 period. In the 1.0, the foundation has been more mature and 2.0. It has the scheduling and also the factoring of the abstract resources. After it has been split, it can be used on more complex big data calculation and computing as well. And also it's going to integrate with AI and deep learning. And the direction and chance in future is utilizing this spark technology. And in 2016, it started to come into being and now has become a very popular engine for computing as well. One of the major events last year in the big data field is two giants, HotWars and Cadela, has built up the new Cadela. So it has entered the 3.0 era for its iteration. So this is the history of big data development. From the perspective of the big data stack, it has been separated into different levels. That's how we have got mature big data first is the NFS. And HDFS has become the foundation of all the big data. Of course, we utilize the others like S3 on the cloud and also on-prem. Maybe YouSeph that is supporting the blocks and files, documentations and different type of storage classes and also the big data workload. On the other prospects, we also have other scenarios that have been supported like supporting the ORC, incorporating and the MECC for its storage. And in the computing level, including its management and orchestration, it needs another resource negotiator. It's like the operation system of the big data systems. It's going to orchestrate those resources and also to manage some of the containers and abstractions of those containers. And it also tries to achieve the consistency of all of these systems. And then it's the computing system in computing layer including the processing of the data and computing and also the hype engine like the SQL and Android and Spark as well. We also mentioned it's very popular and also the mainstream technology and it's utilizing Intel computing technology to support many different type of computing scenarios like machine learning and also some independent scenarios as well. And now there is another one that is very popular. It's the real calculation and it's very popular now. So all of these scenarios here will support different type of business scenarios of big data including interactive, acquiring, etc. There are different types of trends like streaming and also the big data and traditional big data and its integration with deep learning. The strategic development of big data is like that. So how about a chance? First, it's going to migrate to the cloud. A lot of the big data workload has been migrated to the cloud and operated there. One of the core reasons is that the cloud computing is very important. Second is that the operations and maintenance of big data is very complicated. You can see the stack is very deep funded and also usually it has a very big foundation. If you want to manage it, there is a lot of challenges on operation and management and maintenance. So it has to migrate to the cloud and also the integration of KBS because a lot of the workload is transferring to the container. A lot of them are microservices and before that a lot of batch mode and utilizing different waves of iterations to deal with their workload. In the future, if there is a whole set of the projects that utilize the microservice or the traditional ones, then it will increase the efficiency of the utilization of resources and integration of the clusters as well. As for the layering of the big data system and its power I think the enhancement of these functions is that I know how can I schedule my computing and also so that my computing happened close to my data. And now I separated computing and storage. Computing could be elastic, but storage is very difficult to be elastic. So if we combine these two parts together, it will affect our business. How can we realize this elastic computing and separate this computing from the storage and also prepare for the replicas? I think there will be a very big room for us to innovate. Also, we are migrating to the cloud. There were some of the on-prem scenarios we have fixed clusters. If it has a peak times, we can flexibly expand it to do some deployment. This is another trend for it. And just now I mentioned containerization. It's a very important direction. The benefits is very obvious. First of all, the hardware utilization will be higher compared with the VM. The hypervalue can save a lot of resources and reduce the consumption of resources as well. And also it's for the split of resources or segregation isolation of the resources or from the perspective of security, it will be better compared with the traditional isolation from the perspective of network, namespace, CPU, GPU. It will have better states. And it also provided packaging solutions. So all the containers are able to seamlessly operate it on different nodes. You don't have to care whether you're utilizing the native languages or not. And this is related to the independent states of different nodes. And here is the whole framework. The data in the Kubernetes architecture is master, slave, and the master level is CTCD, APS server, and scheduler to control the resources. And also there are separation between different parts. And also the parts can talk to each other. So this is the overall architecture. Here are some challenges for big data. The first one is the orchestration and deployment. The optimized is using them through microservices. And there are many big data workloads. It's around and around. So it's not that you can finish it once and for all. It's round by round distribution model. And second, the big data workload sometimes is using an IOP. It's different from the traditional microservices, different manners. So the different types of workloads mix them together to distribute. The good thing is they can more efficiently integrate resources. The con is that if you can't segregate them properly and SLA management to achieve the desirable level, it is a challenge or an important challenge for distribution and management. And thirdly, for some scenario, it's tuned. But for some un-prem environment, it's not tuned. In some containers, it's used to unsound the story. But they are temporary things. So I don't care too much whether they are boring or not. So overall, in terms of sustainability, it's different from the traditional SWH or other persistent storage. So this is some capability. So this is the background and challenges later on. My colleague Chen Yi will talk about CSI and other matters in design. Overall, in in-house and un-prem, using KPS to replace the traditional models and big data storage is still using HDFS. So how can HDFS cope with big data workload while at the same time other workload on the platform? So regarding HDFS as a singular integrated storage, currently we provide the external storage interface at CSI. So I say a few words about CSI. CSI, the whole name is a container storage interface. It's an identity service. And it's called SWON. Together with the industry, they jointly developed this interface. It includes the on-spec and the purpose. The purpose is to bridge the container orchestration between different suppliers from different storage providers and for providers, as long as they follow this spec to make the CSI driver of their own, then it can be utilized by other providers. It's not unique to one platform. And the container orchestration provider, he only needs to provide this interface, CSI interface, so all the CSI storage can be used in this cluster. So it is an interface layer where different systems can interoperate. And it's not locked on one vendor nor one container orchestration provider. It's a neutral interface. And another feature of this spec, it pays attention to the control flow of the container orchestration using the volume, managing the volume. And for IOF, the spec does not include an IOF. The two designing criteria is to keep it interface as simple as possible. And second, the distribution, the using of the interface. Multiple distributions should provide the same result. CSI divides the service into three groups, identity service, identity service is used to expose the generic information about the driver, like the name, the version of the driver, and the capability of the driver. So this is a must service. And the second service is a controller service. And the main function of the first service is to cater to the user's demand to create and manage volumes. And the third service is node service. For some of the disks, cloud disks, this function enables the attachment or the attachment with the VM. So this type of a service is the only need one instance, and that will be sufficient. And regarding node service, the main function is to, you know, the volumes created by controller service based on the user's needs to mount it on the designated node. So it's a node-based service. In Kubernetes, on every node, we require an instance on every Kubernetes node. Besides the basic functions, the mounting and controlling, now the version 1.9, it supports many advanced functions. One is the broad volume. Back then it's based on files, FS volume, and now it supports the creation of snapshot. So it can create a snapshot for a certain volume and at a certain time slot, based on this snapshot, you can recover or restore the previous state. And thirdly, it's topology support. CSI supports these storages. Basically, they are mounted on their network-based volume. It's also distributed topology, so it's sensitive to the network. Under the same network upon creation, this volume is only suitable for your company, Nekus. You know, the nodes under the same network. And the nodes in some other networks might not be friendly to your applications or not compatible. So it's still optimized topology support. Before the volume does not support besides. So the volume is immutable, and now it supports the resize of the volume. Currently, it only supports to expand the volume currently. And another feature is interim volume. Before, the volume's lifespan is coupled, is not coupled with your system. If your volume still stands, when the project is finished, you need to manually delete it. So it requires a manual deletion. And this type of volume, the lifespan, will be the same with your part. When your part is released, then the part will also release the volume as well. And the cycle of the volume, just now I said the lifespan of the volume involve the controller service and node service. And for controller service, it is responsible creating and deleting the volume while node service will ensure that they are attached to the corresponding node, even multiple nodes. While creation, it only requires a very simple operation. So this is a CSI, two ways to deploy CSI services. Back before, I said your node service on each node is a must-have. And the difference between the two methods is whether you need one or multiple volume. One method is you only need to deploy one. It satisfies the above functions. Second one, if you combine the three services and deploy it on each node, it's also acceptable that within the cluster, you only need one active controller service. If you deploy so many controller services on each node, you need a strategy to maintain only one active container at one given time, just like the leader of our HDS. You need to maintain one active service. And this deployment will be more intricate. So we would recommend the above deployment approach. So that you can maintain a simple architecture. Next, let's review in Kubernetes storage support the history. Earlier, all the Kubernetes supported storage, they are all maintained and developed by Kubernetes, the code or all owned by them. But with the population of Kubernetes, it needs to dock more and more providers and vendors. So it came to the epiphany that the model is not sustainable. So they changed, they shifted from the previous model to the out-of-tree model to dock the vendors. One way is called flexor volume, another is called CSI. So far, Kubernetes no longer supports. That means if you just freshly docked the two Kubernetes, it will no longer recommend you to using Kubelex. It will instead recommend CSI. So for some of the intrigue rankings, Kubernetes is doing some work, selling some to out-of-tree and CSI model. So this is the future. Kubernetes will use CSI to dock third-party storage. So itself will not maintain the storage drivers. CSI driver spec is for the storage vendors. Storage vendors will follow the specs to deliver, to make their own systems. So how to acquire this spec, how to utilize the spec. Actually, there's still a gap in between. This is the driver's deployment. It's not a static process. It's a dynamic one. So that means how can Kubernetes dynamically identify that in this cluster. Currently, there are how many CSI drivers because it's continuously changing and how to identify how many existing storage drivers that are accessible and how to develop the interfaces. So Kubernetes provides, in the middle here, the green ones. There are many auxiliary containers or components that are taking the form of containers. And these components, the functions, are assisting Kubernetes to identify that in this cluster, there are some newly registered units and using Kubernetes code to distribute the interface of CSI driver. Actually, the interface of CSI driver can be directly used by the code but through an intermediate an ancillary component as agents. So through these external components, these auxiliary components, Kubernetes can dynamically identify the CPI drivers and leverage them, dynamically creating new drivers corresponding to the volumes or in a dynamic manner, just like how Kubernetes supports other functions. It's dynamic, no need for static configurations. So far, there are over 60 CSI drivers supported. So in Kubernetes, you can visit both to find out more for CSI support history. Let me add. It started from 1999 and after a few versions iteration, now it's a version 1.13 CSI part. The GA is 1.13 and it supports 1.0 as standard version. Speaking of CSI, I must mention within Kubernetes, the volume support the KPVB and storycast. And for PP, the perceived volume, it represents storage resources and PVC, a persistent volume claim. It's a user's demand, a description of a user's demand for the storage resources. It's described in the port. When you describe the port, you need to also define the PV and PVC. They're sharing the same match. When the port is scheduled on the load and it's going to generate a PVC if you need to store, it's going to find the corresponding system volume, consistent volume to satisfy your needs. For example, I need 100 gig, then it's going to check whether they can find the consistent volume that can satisfy this demand. There are two ways. And for the bonding, it means that the resources need the immune to manually manage it, to establish it manually. If there is no matching resources, it's going to process it in the other way. Another way is that we don't need the manual establishment of it. When we have this demand, it is not satisfied and it can dynamically create a consistent volume to it. And to build up the bonding relation between them, the key thing for the direct bonding is the story path. Storage class is related to the list of the storage. In its regression, it may say that talk about what kind of things it needs. So everything is defined in the storage classes. Next is the suggested helper container. And it should help you to deploy and also the way it is used to deploy. And on the left-hand side, it's the port. It's suggested the node driver and also deployed it as a demand sector. So on each of the nodes and the Kubernetes nodes will have this kind of node service instance. And on the right-hand side, it's a controller service and Kubernetes accessory containers, attach container, provision of containers, etc. to build up into a port and deployed as a sector. So this is a suggested deployment method. And our demand is that HDFS want to become an in-house cluster and provide the unified solutions for it. Then that will depend on the HDFS architecture. If you have listened to other speeches, maybe you have been quite familiar with this already because HDFS has three replicas by default. For its data storage, so it's very reliable. And HDFS has about more than 10 years of experience and history. So it has the history of market deployment and operations and utilization. It has been a stable storage approach and it's a very good candidate for Kubernetes storage. Now the default cluster's number is very large and also there are a lot of nodes as well. Single clusters may have more than 4,000 nodes according to our understanding. But there are some Federation cluster mechanisms. We know that there are more than 20,000 can easily get by several dozens of thousands of nodes. I also want to mention Apache Ozone. It's a very popular topic as a storage project in our ecosystem. And their developer actually the developer of the Hadoot HDFS. So they have a lot of experience of different scenarios and operations and maintenance experience as well. Ozone is an object storage system. So its goal is to solve the scaling problem of HDFS so that it can achieve cloud-native storage. This is the structure of it. If you want to put HDFS in Ozone as its persistent folder, is there any path first? It's utilizing the HDFS NFS gateway. The NFS gateway is supporting that you will not belong to the HDFS clusters. But maybe you want to utilize some of its nodes. Then on these nodes, you can deploy an NFS gateway and then put one of the root of HDFS to root to the local and take this as a local volume. And next is utilizing the root. So different storage will have different roots. And also it needs your deployment on different nodes because the gateway should be deployed on each of the nodes. And also should provide some different services to it and a lot of configuring as well. So the process configuration would be a little bit complicated. But now we have CSI, the auto identification mechanisms and mounting mechanisms. It will be much easier. So I think in Kubernetes, which is quite suitable, we can use the NFS gateway to develop CSI based on the NFS gateway because Ozone is supporting S3. So it can utilize its method to mount. So it can also base on the S3 gateway. This is the utilization of Ozone on the Kubernetes. And it can also make it into a block device. We know that the NFS file system is very clear, but it's lacking a device like that. So they want to make a block device based on Ozone. And Cordura is also here. If you're interested in it, you can have a look. So what is the benefits of utilizing this as the storage on it because it has three replicas and also it's reliable during its existence. And it has very strong enforcement. Once you store this data, then you can ensure the consistency. It's impossible that you write it in this way and then its reading is different. And the speed of reading is very high. It's also able to route different paths on your local. Of course, you can route different paths. So it's suitable for different users because they have different authentication. They may see different directory. So it's able to route different users to different places. And also to utilize multi-nodes mounting and read and write. The multi-nodes read and write has one limit. You cannot write the same file at the same time. It's not allowed. As for snapshot, you can do recite. Dynamic recite is not supported here because the snapshot is not supported here. And this is the deployment framework based on the NFS gateway. It means you have to deploy NFS gateway on the nodes and also to utilize the CSI driver node and then mount it on the local. And then it's able to provide the storage to you. And next is based on Ozone. Ozone is also recommended by many communities because it has object storage that can include the file storage. It's still developing its block storage. Next is some result. So time is limited. I'm not sure whether you have any questions. Please use the microphone. So it's a question is how to separate the workload on different scenarios. And usually it's used at IO and also the bandwidth split or isolations we have addressed it here. HDFS integrated with Kubernetes into one product and it's not trying to solve the native problems. But we haven't find solutions on the cloud native. In the traditional fields, we have actually isolated different fields and so that they are not going to influence each other. This is one of the reality. NFS gateway will support mounting one of them. So if you have several, then you need more. There is no limit. What I mean is that for example, POT, maybe on one POT, one node may have 100 POTs, different POTs using different volumes. And of course, on this POT, we'll have different NFS. If it has several NFS, it won't have any problems. For example, if you have 100 volume, we haven't produced it in the larger scale because for some of the applications, we are different to control which POT is utilizing, which volumes in these occasions, what should we do? I think this is the practical usage. It's difficult to validate at the moment. I think the volume may be able to share between different ports. I think it's okay. Thank you. So this is the end of our speech. We can communicate.