 Stephanie, everyone, this is Xin. Hello, everyone. Xin and I are very honored to have this opportunity to present our talk in Shanghai, KubeCon. My name is Xin. I'm Google engineer. I work for GKE, Google Kubernetes Engine Storage Lifecycle Team. Xin is OpenSS Lead Architect. OpenSS is Open Source Project and the Linx Foundation. So we are also proud Kubernetes Open Source contributors. We lead six storage snapshot workgroups. Hello, everyone. I'm Xin. I'm the head of OpenSS. OpenSS is an open source project under the Linx Foundation. Its purpose is to solve some of the difficulties in the survival of the environment. This is Xu Jing. She is the engineer of the Google GKE survival management team. We are all contributors to the Kubernetes community. We lead the workgroups of storage snapshot. Today, we are very honored to have this opportunity to give you a speech. We will use English to speak. Welcome everyone to chat with us later. So today, we are going to talk about CRD, custom resource definition, but what it does and why it's important to talk about here. So we want to share a story from us using CRD in our project, which is Faraday's talk. So our journey starts about a year ago when six storage started a snapshot project. You know, storage is for storing data. It is important that you don't want to lose it. But things might happen, things might fail, data might be corrupted. So snapshot gives you an easy, quick way to recover your data. It's very much like taking a picture of your data. And later, you can recover your data from that point. So we work very hard for this project since it provides such important feature for storage. So we define API for volume snapshots, you know, like other Kubernetes APIs. It tells system what to do and the controller manage the behavior based on the API. And then we present the design to seek architecture. You know, there are a group of people have this power to decide whether to accept your API into Kubernetes core. So we hope to get their approval so we can add snapshots into the core. The user can start using snapshot API. Guess what? We got rejected. So they told us no. And to use CRD said, at that time, we had many questions. Why? Why push us out? Why we have to do this out of trade with CRD? Isn't for like third party application thing? That means our feature is not that important and should not be part of Kubernetes. And seeing the remember, we were very hard trying to change their mind. Yes, of course, I remember that. We tried many different ways trying to prove that snapshot API should stay in tree and that seek architecture was wrong. But after many, many rounds of discussions, we finally changed our mind and we agreed with that decision. To understand from seek architecture's point of view, let's go back in time to the very beginning of Kubernetes. In 2014, the first PR was submitted into the Kubernetes repo. Kubernetes was born. In 2015, we had the Wanda.org release. Early this month, Kubernetes just passed five years old. Happy birthday, Kubernetes. Kubernetes grows fast. When it was just a little baby boat, it had a very small number of APIs and controllers and supported very limited features. Now, at five years old, it has grown into a much bigger boat with a lot of APIs and controllers and a lot of functionalities. Do you know how fast Kubernetes has been growing? To find out how fast Kubernetes has been growing, let's take a look of the number of API types in Kubernetes core to measure the growth rate. In release 0.4, there were only 19 API types. Since then, it has doubled and tripled. Do you know how many API types were in release 1.10? It is 110, so that's almost six times as many as what we had in release 0.4. With this growth rate, can you imagine how many API types are in the latest release 1.15? I will tell you the answer later. Are we expecting Kubernetes to grow into this scary Kubernetes giant? How can we possibly manage and maintain such a giant? Obviously, we can't. It's time to put a stop and say no more. We can no longer keep adding APIs into Kubernetes core like this. Instead, what we need to do is to make sure Kubernetes is truly modular, extensible, easy to add new features, and maintainable with a minimum core. So how can we achieve such a goal? Gene is going to tell us how CRD comes to the rescue. So yes, to explain CRD, I want to use this Kubernetes institutions book. Anyone read this book before? It's fun, right? So I want to use it here not because just the cute pictures. It's because the book explained the concept in such an interesting way I want to share with you. So imagine Kubernetes is a big zoo full of animals. We have API to represent animals, and then we have controller to monitor them. You know, case always want to see something new, something funny, and then we keep adding new animals into the zoo. But how about something not even exists? Like this hippo giraffe, like this snake with big ears. How are we going to add them? So here it is. First we call something, the API already in the net is resources. You know, like Power, CFO sets, deployments, right? And if you want to define something new, something on your own, we have custom resources. So you can define any way you like. You can define something really weird, like a Power, CFO sets, or non-processed module. And CRD, custom resource definition, is used to define those CRs. Let's take a look at an example with this cute little panda. You know, everyone loves panda. So left side CRD, right? It tells you we're going to define a new kind called panda. And we want to add it into animals group. On the right side, CR, it defines the detailed spec of this new resource, panda. And you can give a name. You can define the spec like first date, place, wait. Now with CRD, you can add the custom resource into the API. So you can use it just like other Kubernetes API. But it doesn't do anything yet. Until you add a custom controller to manage it. With CRD and custom controller, you get something so-called decorative API. That is really what Kubernetes is all about. You know, the API spec defines the desired state, what the system will look like, and the controller will move actual state towards desired state. I hope I explained CRD good enough. And Xin, do you want to talk about our journey? Yeah, sure. So now we understood better why CIG architecture want to push Snapshot APIs out of the tree, and how CRD can help us extend Kubernetes API. We started to work on the Snapshot feature. We created our repo under Kubernetes CSI. We added CRD definitions. We added a controller to watch the Snapshot API objects. We added deployment YAML files and documentation. There are lots of details here. But as a user, you really don't need to know. From a user's point of view, it works the same as any other Kubernetes API. In Kubernetes core API, we already have PVPVC, HOD, Node, and so on. Now we have also added wanting Snapshot APIs into the API assembly. You can just use it the same old way. You can do a cube cut or create. To create a Snapshot API object, you can do a cube cut or delete to delete it. The Snapshot controller will be working hard behind the scene to making sure that user's request is taken care of. Now you may wonder, does that mean CRD is exactly the same as native API? We have to say that there are differences. CRD is still beta. The plan is to bring it to GA in the next couple of releases. CRD already has many features available in native APIs. For example, before team validation, open API with three schema definition, and so on. Knowing the effort to bridge the gap between the two. CRDs are becoming more and more like native APIs. What is Snapshot CRD is part of Kubernetes. It's owned by Sieg storage. It has entity and testing. It has official documentation. User can use it just like any other Kubernetes storage feature like persistent volumes. And more and more CSR drivers have implemented Snapshot feature. People also started to implement more advanced functionalities like data protection and disaster recovery on top of that. So now you have heard our story. We successfully added one in Snapshot CRD Auto Tree as a core Kubernetes feature. So great, finally our story has a happy ending. Now you probably get interested in CRD and want to try it yourself. Like how about to build a CRD for this cute panda and create a controller to monitor it. Is it easy to do so? To tell the truth, if you do it manually, and from scratch, you have to go through a lot of steps and have to understand code structure very well. The good news is we have some excellent tools to help us. So here, for example, QBuilder, one of the most commonly used tools to help you to create, publish your CRD APIs and build controllers. With these one, two, three simple steps, you can create a project. You can define a new CRD API and you can build and run your controller. That easy. So to prove we are not lying, we want to show a demo with this panda. People from Shanghai might already know her. She was born last year and her name is little princess, Chinese name Tiki. And seeing, do you want to show everyone that how we can build panda CRD and controller in two minutes? Sure. So first we create a new project for the Kubernetes Zoo using QBuilder init. With this one line, we will have a new project created. Step two is to define new API for panda in the animals group using QBuilder create API. After this, we already have a project and with files populated in that project. Next, we add CRD definitions. So in panda types.go file, we will modify panda spec. We added a birthday, our birthplace and birth weight in the spec. And in panda status, we added age, weight, member consumption. So later on we can check whether she eats well and grows well. Next, we add controller logic. So we go to the panda controller. Here we go to the reconcel routine. Now we find panda based on panda's name. And then we just update panda status. So after that we are done with the controller logic. Now we install CRDs into Kubernetes cluster. It's installed. So now let's take a look of the CRD manifest we just installed. So this is the CRD definition for panda in the animals group. Next, build around the controller manager. Now we have the controller running. We can test it. So we have a spec for panda chi chi. It shows that her birthday, she weighed only six ounces at birth. She was tiny. Let's check how she's growing now. We're going to create a panda API object for her. It's created. Now let's take a look of her status. It shows that she is 11 months old now. She eats about 10 pounds of bamboo each day and her weight is 60 pounds. So she has grown a lot since birth. She's healthy. Great, we did it in two minutes. It is easy. Please try it yourself if you're interested. So before finishing our talk, we want to review CRD's history to get a full picture of it. You know, at the very beginning, CRD has a different name. Third party resource, TPR. You know, people want to try out their ideas. TPR, for the first time, gives you a way to do it outside of Kubernetes without adding your API into the quiz core. Of course, at that time, it has limited features. Now over time, teams has improved a lot, add a lot more features, and now people start to realize the benefit of using it and they start to create application operator to manage their own logic. Today, not only just for prototyping and for application operator, we have shown how to use CRD to extend Kubernetes API core. With this capability, we no longer need to worry about seeing that Kubernetes joined, right? What we are going to see is this cool, nice, Kubernetes fleet with the core set in the center, which is easy to manage, easy to maintain, and we can add all the components alongside. We add volume snapshot, and we are going to add a screen hook, volume group, application snapshot, just a name of few. Imagine we are going to, if we are going to add all of this into the core, it will eventually blow out, right? So remember seeing, gave us the question to guess the number API type we add in current release. What number did you guess? Seeing are you ready to give out the answer now? Yeah, sure. It is 113. So it's actually almost the same as what we had in release 1.10. That's because CIG architecture has already started to limit the number of APIs that can be added to the core. Let's look further into the future. We have a vision. This is from Tim Hawken, the Kubernetes pioneer. He said that eventually everything will become our CRD, except things to run CRD. Of course, it's still a long way before we can really realize this vision. There are version compatibility issues between CRD and the core Kubernetes. And also installation of CRD is a separate step. So we need to figure out how to better handle CRD's dependency on Kubernetes core and so on. But we hope that with hard work, eventually we will get there. I believe so too. And I hope you learned something from our journey too. We cannot cover all the details in 20 minutes. So here, a list of references that you can check out later for more details. Of course, just asked to present the work, but it is definitely a teamwork. We want to thank everyone who worked on it. And last, not least, this is our contact information. Please join us for further discussion. Thank you. Thanks.