 Hi folks, welcome to my presentation room in the Skopika. By the way, it's been two years since the COVID situation and the Skopika events are still virtual this year again, so hopefully just everything will be over really soon and I do hope everyone to be safe and well wherever you are living and are working at. Okay, let's get out of our business today and the topic of my presentation is API Server Builder extending COVID-19 spare and aggregated API Server. My whole presentation is going to start with a very detailed introduction about the technique of API Server aggregation per se. In the next one, I'm going to expand the details about the subprojects named API Server Builder and also its successor new subprojects named API Server runtimes. And in the end, there will also be a Q&A section in which I will be addressing all the questions from you if there's any. And the time is limited, so we can probably extend our Q&A sections to the Slack channels. Over there you can just DM me or whatever you want. Okay, I'm going to start with a brief introduction about myself. My name is Min Jin and I'm a software engineer working at Alibaba Clouds and I'm also a Kubernetes maintainer working in CIG API machinery. I'm the subproject owner of multiple Kubernetes subprojects and I'm also the speaker of Kubica 2019 Europe 2019 of America 2020 of America. And besides Kubernetes, I'm also actively running the project of open custom management. I also maintain the API Server aggregation related libraries for most failures. And you can contact me via my mailbox or just mention me in the GitHub or DM me in the Slack channels or Discord. So I outlined my presentations into basically five parts. The first is the background around the API Server aggregation in which I will be sharing the basic fundamental knowledge of the technique. And then we will sort of the history of API Server aggregation subprojects on the evolution of API Server and co-builder and the controller runtimes. And in the third part, I will announce the new project of API Server runtime, which is incubated by the API Server builders similar to the relationship between the co-builder and controller runtime. And in the first part, I will have a demo of how API Server runtime works. And there are also a few projects I will show you. In the end, I will share with you the future roadmaps of these projects. Okay, let's move on to the second part to the first part of my presentation. It's the background about API Server aggregation. Over the years of community support, I found many developers are stuck in a builder complaining that a technique is too complicated to understand. And most of them are stuck at the, for example, the installation of API Server obligations or the management of certification or something like that. So today, in this part, I will be elaborating the details of how the API Server aggregation works and the share share the detail of the fundamental knowledge because so the AA is actually the acronym of API Server aggregation. And it's an API that's beta in 1.6 and graduated into GA since 1.10. And from the perspective of a user, the AA works by we just bring one resource named API Service into the Kubernetes cluster. And in the, in the spectrum of API Service, you can specify the endpoints of the AA Server instances, or you can just passing in reference of the service, then the Kubernetes cluster will automatically resolve the endpoints for you. As for how to build an AA Server, there are basically three physical approaches. The first one is just copy and pasting from another official projects named sample API Server. And, but the projects, it's actually hard coded. So you will have to edit the, for example, the resource name of the, the, the aggregate, the AA Servers. And the second approach is extending the API Servers where the scuffle beams provided by the API Server Builder, which will be similar to the user experience of how could we do the works. And the third phase, by the way, will be just importing the libraries of API Server runtime, which I will elaborate later. You can just initialize a brand new Cologne projects and just add the API Server runtime into the dependency list and the magic will happens. As for, as for the insights of API Server applications, we're going to expand the internal technique from four aspects of the authentication and authorization, emission controls, and last but not least, the API quality fairness. By, by leveraging the technique of AI, you can basically do any kinds of customization, if you want. For example, you can even replace the persistent layer for an API Server. The normally, the storage layer of an API Server is supposed to be ETCD, but you can, by, by, by leveraging AI, you can just replace it with any kinds of further storage if you want. And this picture shows how an AI works in the architecture. So, an AI Server is a standalone API Server outside of Kube, Kube API Server. And by the spec of API Service, you can make the traffic going through the Kube API Server successfully proceed to the downstream of AI Servers. And you can even develop your custom controllers working based on the AI Servers. The custom controllers are supposed to connect the AI Servers while the proxy of Kube API Server instead of requesting direct speed. However, direct requesting will also be working, which I will also be explaining later. So, before we dive into the authentication of API Server creation, we have to understand how the TRS, the general technique of TRS works. Basically, there are two kinds of TRS in the world. The first is vanilla TRS, or we can call it the simple TRS, which is widely used in the in the HTTPS Servers. In the vanilla TRS, the only verification that happens is that when the client is trying to request download the contents from the server, then the server will be returning its server certificates back to the clients. Then the clients can check the signer CA, the authority of the server certificate, and check whether the CA is seen in the client's trusted list. Also, the clients can verify the server certificate's SAN, which is the SAN stands for Subject Alternative Names, which is a list of domain names or IP addresses. So, the SAN basically restricts the usage of the server certificates. And as for the mutual TRS, in addition to vanilla TRS, the mutual TRS has some other rounds of verifications, which is the server will also verify the client's certificates from the clients. So, there's a trusted authorities in memory inside of servers. The server is supposed to reject any request that doesn't contain, doesn't have the client's certificates signed by that CA. So, this is basically how the mutual TRS works. In the end, there's also an optional step of verifying the client's SAN. But actually, in the Kubernetes native libraries, it's skipped by defaults. Okay, after knowing how TRS works, let's take a look at how authentication works. This picture is the typical case when a client is trying to request a server by the proxy of Kube API server. So, the normal authentication happens between the Kube console and the Kube API server. The Kube API server will check whether the client certificate is signed by a CA authorities, which is specified by a flag named dash dash client CA file. This is rather simple, but, however, let's take a look at the authentications, the mutual TRS happens between the Kube API server and the aggregated API server. The AI server is actually, sorry, the Kube API server is the client when processing the request to the AI servers. So, the client verification happens when the AI server returning the server certificates back to the Kube API server. So, the Kube API server will check whether the server certificates of the AI server is signed by the CA bundles specified in this API service. And the AI server will also perform another round of server verifications by checking whether the client certificates from the Kube API servers are checking the client certificates. It's signed by the CA, CA authorities specified by the dash dash request slider CA client file with the flag. And optionally, if the AI servers are deployed inside of Kubernetes clusters, you can also looking for a config map under the interface of Kube system named extension API server authentications. There's a key named request header CA file that the content will be the expected signer of client certificates. And this is the indirect approach when a client is accessing AI servers where the proxy of Kube API server. However, a client can also access a AI server directly without the result proxy by the Kube API server. In this case, the case can be different when the client, if a client is requesting using any signed certificates, then the Kube API, then the aggregated API server will just checking whether the client certificates by the CA authorities passed from the flags dash dash client CA file, which is identical to the flags of Kube API server. And the AI server will also be looking for the config map under the Kube system, the same config map, but the key is client dash CA dash file, which is happening inside of one minus deployed inside of Kubernetes clusters. And repeated, repeated, okay, the servers will just, which is verified whether the client certificates is signed by this CA. And however, if the client is the credential, if the credential is service account tokens, the AI servers will do not understand how to verify the essay hook and solving some reasons. So instead of verification, manually, you just delegate the verification of service account tokens back to the Kube API servers, where an API named token review, and after getting the response from Kube API server, the AI server will know whether the tokens are valid or not. So this is how the authentication works between Kube API server and AI server, which is a bit complex than the simple Kube Kato Kube API server authentication. Then after the authentication, the next thing happens is the authorization. The authorization works inside the AI server is pretty easy. First of all, for clarification, the Kube API server will just forward all the requests to AI servers without performing any kinds of authorization check. So all the authorization checks are happening inside of the AI servers, but the AI server doesn't contain any logic of authorization. So again, the AI servers will be delegating the authorization back to Kube API server, where in dedicated API named the subject access review, and the results of authorization check will be cashed in inside of AI servers. And also we can also wipe off the authorizations in the AI servers, but this is strongly not recommended because this will be leaving the AI servers unprotected, can be attacked by hackers, and similar to authorizations. The admission controls inside of AI also happens by delegating the admission control request back to the Kube API server. The Kube API server do not perform any kinds of admission controls upon receiving requests. Instead, they just proceed redirect the request to AI. And after that, the AI servers can either execute the built-in admission controllers, such as namespace controllers or resource coders, or the AI servers can just delegate the admission controls back to Kube API server. They are in similar API named admission review. And we can also wipe off the admission controllers inside of AI, but this is still not recommended. But disabling the authorization and admission controls can sometimes can be useful, helpful when you are debugging in your local environment, because sometimes you just don't need that security check. You just, when you just want to verify the logic of your API servers. And this is how everything works in a piece on the host bill. When a Kube Cato sends a request to Kube API server, this is how authentication works. It just works in the flow. The mutual TRS between clients and Kube API server, then the mutual TRS between the Kube API server and the AI servers. And then upon authorization, the AI servers will be delegating the authorizations back to Kube API server and returning the responses. Then in terms of admission control, aggregated API server delegates again, then the responses are returned. And finally, the AI servers will be returning the actual results back to the clients. And there's also an advanced feature named APF added to Kube API server and AI servers since 1.18. And it's also a beta since 1.20, which means that the feature will be enabled by default. So this can be useful for you to understand how APF works between the Kube API server and the AI servers. So in the original design of APF, we disabled the flow controls by defaults so that in order to avoid cyclic blockups or the cyclic dependencies between Kube API servers and AI servers. So every request will be Kube just for only once inside of Kube API server or aggregated API server. As it's shown in the picture, the request will be Kube upon its first arrival at the Kube API server and Kube again inside of AI servers upon arrival. But the however delegated authorization will never be throttled. And because of this will be classified into the Excel products across also the omission control will not be throttled. Okay, after understanding the details of implementation of AI servers, let's take a step back and take a look at the CRDs in a higher level. Let's take a comparison between the CRDs and AI. The CRD is basically a declarative API, however, the AI is in collaborative. Though by the declarative API we can easily define the schema of CRDs by using the grammar of OAS or the OAS server. But there's some inconvenience when we are trying to do development in multi-version CRDs. This CRD has a mechanism of conversion by both collaboratives can be somehow difficult to develop one because it requires much more background knowledge of how Kubernetes API works. And as for sub-resources CRDs only have skill sub-resources and status sub-resources. So when you are trying to extend your arbitrary sub-resources, then AI will be probably something you're looking for. And the technique AI can help us to replace the ETCD as the precision layer. We can build an AI working based on MySQL or other kinds of SQL storages or even you can be working on a local file pass. But the top slide of AI is that we're going to define our schemas in Golang and it requires actual work of coding. It can be more costly comparing to just writing a YAML in your local computers. So after the comparison, I have some personal suggestions for choosing between CRDs and AI. The rule number one, if you are extending a trivial custom resources, for example, if you are just trying to structuralize the schema of your custom resource, then you can try CRDs. However, if you are looking for offloading API costs from Google API servers, for example, for example, to get some data out of the main ETCD clusters, if you want your AI servers working on dedicated ETCD clusters, and this will be offloading some ETCD within write costs. And also, if you are using the cost, the burden on Google API server will be reduced in a sense. And also, if you are trying to develop a specialized API, for example, extending custom subresources such as proxy, the proxy subresources is already used in the Kubernetes such as pulse proxy, service proxy. So if you are looking for a custom proxy, implementation, or other any kinds of subresources, then you should probably try AI. And also, if you want to get rid of ETCD storage, I think the AI will be the only choice to go. So by replacing the storage layer of the ETCD, we can easily encapsulate a third-party API into Kubernetes resource, and also adopting other kinds of other types of storages, and even integrate your legacy systems into the Kubernetes concepts. And let's move on to our second part, the evolution of API server to Kubernetes from the control runtime to API server runtime. So in the past, there are a bigger general debate between CRDs and NA. And at that time, the CRDs and NA, they are neither mature and developers are having trouble getting on board any of techniques. So the Kubernetes is actually trying to provide users a converging obstruction layer that eliminates the gaps between CRDs and NA. So the users will be just facing running on the obstructions provided by the Kubernetes builder or API server builders, and let the builders do all the rest to do the dirty things for you. The second goal will be scaffolding. The scaffolding basically covers, first of all, is the templating of your projects, no matter if it's a controller or an AI servers. And also, we need a scaffolding to setting up the local testing environment. In the past, setting up having a local environment, they say, actually a luxury thing because it involves too many dependencies. But for now, we have the KNB projects, which help us to easily customize, develop over on Kubernetes clusters. And the builders, the API server builders are also trying to provide the users user-friendly API concepts that's hiding some complicated details from the users. So as you can see in the peak, actually, the Kubernetes falls from the API server in the very beginning. The picture is the very first development commit of the Kubernetes builders. So the original idea is that the Kubernetes is supposed to running all the defects, running the fading lessons from the API servers, then have a better viewing toolings for the developers. And after the Kubernetes builder, then we have the controller runtime. This picture shows how the obviously controller runtime is moving over time. In the very beginning, the only thing we have is a project named sample controller. The sample controller is very well documented, but it's not user-friendly for developers because you will need to read the internals and every developer may have their own understanding of how a controller works. After a period of time, we extracted a Go template from the sample controller. So we don't have to manually copy and paste it from the sample controllers anymore, but it's still not user-friendly because we don't have the unified obstruction of controller. So in the end, we initiated the project controller runtimes. If you notice, many packages inside of controller runtimes are defined as go-longs internal packages. And that's the point of hiding the internals, hiding the underneath details from the users as much as possible. And the controller's controller runtimes have a standardized interface of a controller. It's an easy SDK for work for developing and testing. Yes, the testing is also covered in the controller runtime. However, the dark side of controller runtime is that some packages are conflicting with the native libraries from the Kubernetes. But sometimes the diverging can be misleading because sometimes the obstructions in the controller runtimes are just not correct anyway. And by learning from the success of Koobi Builder and the controller runtime, we made a new project named the API Server Runtimes. So the name is similar. So I think you can get it. The first drop-level API Server Runtime is drafted in 2018 by the founder of controller runtime, Soli. And the source is still available in his repositories. And now it's incubating into an official Kubernetes 6 sub-project which is now publicly available here. And the different thing comparing to the controller runtime is that the API Server Runtime is completely honoring the original implementation of the sample API Server. It works by just importing the sample API Servers into the API Server Runtimes. And then the API Server has its own lightweight overriding layer or obstruction layer over the sample API Servers for users to be looking on. And as you can see in the link, the sample API Servers is unaligned into API Server Runtimes as an internal package. So users will not have to care about the API Server Runtimes but still getting help from the sample API Servers. And after telling the evolution of the projects, let's clarify the scope of API Servers and API Server Runtimes. Overall, the API Servers will be a project of doing scoff loadings for API Server Runtime, which is similar to the relationship between controller and controller runtime. And also the API Server Builder provides users a set of useful utilities that can analyze to operate the API Servers inside your Kubernetes clusters. As for templating or the code generations, note that the API Server Builders will only do those one-shot code generations for you, including the project initializations or adding your custom resources. All of these work will just happen once and it's not consistent. And as for the scope of API Server Runtime, it's aiming at being a user-friendly library for building aggregated API Server. And it's also providing one lightweight toolings for code generations, which is the consistent code generations, not comparing to the API Server. The API Server does one type of things, but in the API Server Runtimes, the code generation is actually referencing the original code generators such as the copy generator and conversion generator, client generator, informer, and list generator. The aim of API Server Runtime Jam is just saving your time, but you don't need to download all the binaries on your own anymore. Then let's take a look at how the original sample API Server works in the code from the code wise. In the code, you can see that the rest storage implementation is initialized, is instantiated manually in the code. You need to organize all the resources into a Golan map, which is a bit hard to understand. Then you can just install the API groups in generic API Servers. But ideally, we hope users don't touch any internal details like this. Instead, in the API Server Runtimes, we developed a Fluent Builder Flow for developing an aggregated API Server. We call this the Fluent Builder Flow or Wildlife Flow. This is pretty common in the Java world. We found this, it's really useful. We decided to move it to the Golan world. As you see, we can just build our own custom Servers in one line by registering the resources and tuning a few configurations. We can also adjust the flag registrations. There are many, the extensibilities are sufficient for your customers. Also, by the API Server Runtimes, we can develop our new Kubernetes resource by one line API Service Interface Assertions. If your ID is supported, you can just assert your custom API types with a resource store object interface and assert your list type with a resource store object list interface. Then after you make the interface, as I said, the new Kubernetes resources are supposed to look like a charm. Optionally, this is the mandatory required step of developing a custom resources. However, there are some optional steps of advanced customization. For example, the validators, you can assert your custom type with an interface named validator. In the implementation of validator, you can do advanced projects like cross-bill validations or any kinds of non-trivial validations if you want. This picture also shows you how to develop non-ATCD API Servers using the API Server Runtimes. As you can see, in the builder flow, we can call without ATCD methods. Then this will be completely disabling ATCD-related settings and flags from the custom servers. And then if we replace the storage layer with something other than the ATCD, then the server will be running completely ATCD-free. Then let's move on to the demo. By the API Server Runtimes, the cluster of gateways will be easily developing any kinds of sub-resources. This time, it's going to be the connector sub-resource or the proxy sub-resource. As you can see in the picture, I asserted the sub-resource with an interface named connector. Then in the implementation of connectors, the connector method will just return an HTTP.handler, which is the API that's under this group. After setting up the AAT connection between Kubi API Servers and AAT, then the Kubi API Servers will be understanding. Inside of AAT, there's a new resource named the cluster gateways, and there's a sub-resource named proxy. First of all, I'm going to show you how to probe the happiness of cluster when I'm dead using the proxy sub-resource. Okay, this is how the proxy sub-resource works. The proxy sub-resource will be redirecting all the requests under the proxy sub-resources to the actual real cluster. Okay, I will leave my time to the last Q&A sections, so if you have any questions, I think this is a media record, but during my presentation, I will also be available in the presentation room, so if you have any questions, see if you like. Thank you.