 Hello, everybody, and welcome to the Cloud Provider AWS update and roadmap at Fuqon Europe 2021. We'll start off going through the agenda that we have for today. So first, I'll be talking through the background of the Cloud Provider extraction migration and the status of the Cloud Controller Manager. Then Nicole will be talking about the Cloud Provider V2 effort. After that, Aburk will touch on the credential provider. And finally, Yang Yang will give us a little bit of information about the AWS Load Balancer Controller and the roadmap there. So let's get started. OK, so to touch on the background of the Cloud Provider extraction effort, this is something that's been going on for quite some time in the community. The original implementation in Kubernetes had a Go interface called the Cloud Provider interface that was built into Kubernetes along with each Cloud Provider's implementation. So the initial versions of every Cloud Provider was actually compiled into each binary. But the community quickly realized that this was already causing and just going to continue to cause significant bloat as more Cloud Providers were added. And the maintenance effort was just going to be too great. So the effort was started to extract all of the Cloud Provider-specific code. So that's like each Cloud Provider's SDK and any logic that is specific to a Cloud Provider would be broken out into separate repositories and compiled into separate binaries, not using the main build system. So this really began a multi-year effort to extract all the code. And that's still going on today. So I just wanted to quickly go over each of the components that had Cloud Provider code or have Cloud Provider code and the extraction effort related to those. So first, the QController Manager, there was some Cloud Provider-related code in four of the controllers that would become the Cloud Controller Manager. That includes the node, the node lifecycle, the service, and the route controllers. We'll talk about those in a little bit more detail a little bit later. And then there was also the volume control loops. And those are being replaced by the CSI driver effort. So for AWS, that would be the EBS CSI driver or the FSX CSI driver. And another component is Cubelet. Cubelet has a couple of areas of provider code, one of which is the Docker image credential provider. So this was some code that was built into Cubelet. And for common container registries like ECR, there was a package there that would provide the Docker image credentials when Cubelet was doing a pull image. And that is being replaced by a external binary that Cubelet will exec. It's very similar logic. It's just outside of Cubelet. Cubelet execs it during pull image and gets the credentials back. Additionally, the other area of code in Cubelet was fetching node addresses, which is being replaced in the Cloud Controller Manager and the node controller. A couple other areas that are worth mentioning is the API server had the capability to do SSH tunnels. But this was related to cloud provider code because the API server had some flags that it took that enabled SSH tunnels only on GCP. And so that effort is being replaced by the network proxy effort. And finally, the CUBE API server initially had a persistent volume mutating web hook and persistent volume labeling mutating web hook. And that is being replaced by or that was already replaced by the persistent volume controller. So all the cloud code was removed from that. So I wanted to give a timeline from AWS's standpoint, but these are estimates. And this is only going to be true if everything goes perfectly. So it's relatively unlikely to look exactly like this. But this gives an idea of what the order in which things have to happen and what the earliest possible dates for some of these would be. So you can see for the features, the first is the HA migration framework. This is something that is actually, it just was merged in 121 in alpha state. And this allows HA clusters to migrate from just running the CUBE controller manager to running both the CUBE controller manager and the cloud controller manager using a migration lock. So the plan for this is to go beta in 122. Then we have the credential provider framework. This was alpha in 120. And along with the AWS ECR credential provider, that was alpha recently. Actually, I think it was kind of in between 120 and 121. So ideally, both of those, along with the AWS Cloud controller manager, would be beta in 122, which would allow all of them to go GA in 123. And what all this means is that the earliest, according to AWS Cloud provider timeline, the earliest possible date for the entry code removal is 124. But that will probably be later just because of other dependencies. So in terms of the AWS Cloud controller manager specifically, the upstream features are no longer being accepted. Pull requests are limited to bug fixes. So if you are opening a feature pull request, you should open it on the Cloud provider AWS repository where we've duplicated the V1 provider code there. We currently have alpha builds for the Cloud controller manager. And we're working on beta right now. And if you do want to go check it out, there's a couple of different ways you can install it on a cluster. If you use COPS, you can configure it in the COPS config, the COPS cluster config, to use the external Cloud provider. And there's also a Helm chart provided in the repository, as well as just some YAML files that you can use to install it. So what are the responsibilities of the AWS Cloud controller manager? For us, it's the node, the node lifecycle, the route, and the service controllers. The node controller is responsible for updating node status and node addresses. The node lifecycle controller acts on node events. So you might have a node deletion where you actually need to go and delete the node object that would fall under the node lifecycle controller. Then the service controller simply manages Cloud load balancers for load balance services. And currently, it handles ELBs. And it has some NLB code. But we're talking about moving the NLB code to the AWS load balancer controller, which Yang Yang will talk about later. And then the route controller manages a Cloud route table. If you have nodes that are using one, if your nodes are set up with one sider, one pod sider per node. I'm sorry, I'll hand it off over to Nicole to talk about Cloud provider V2. So hey, I'm Nicole from VMware. I will talk about some updates on the current AWS Cloud provider V2 implementation. We will continue to support the existing AWS Cloud provider in out-of-train mode. This new version will address many issues and gaps in the current V1 provider implementation. So for example, we are looking into allowing node names that are not the private DNS of the EC2 instance. For now, AWS Cloud provider forces the node name to be the same as AWS private DNS name. For example, this leads to the node being removed if a different node name is used. So it should be allowed to use any name as the node name. More descriptive node names are generally helpful and useful. Currently, load balancer can only be created with a machine generating name, giving load balancers easy to rate names while greatly enhance the user experience of Kubernetes on AWS. We also consider adding something to the service annotations and allow users to directly manage this. So a new V2 implementation will try to address these limitations. Also, we need this since the V1 legacy implementation is going to be removed soon. A V2 implementation is in alpha state, and after the launch, it will be supported for new clusters. So to develop Cloud provider implementation, we need to implement a few common interfaces, like instances, load balancer, and zones. So for example, the instances and zones interface methods will be called from the node and lifecycle controllers. The load balancer interface methods will be called from the service controller. For instances, we have an initial implementation of instances V2 interface. Instances V2 is an implementation for instances and should only be implemented by external Cloud providers. Implementing instances V2 is behaviorally identical to instances, but it's optimized to significantly reduce API cores to the Cloud provider when registering and syncing nodes. For example, we have implemented interface methods, like whether to check whether the instances exist or is shut down, where we use the node name or provider ID files to find the node in the Cloud provider. As I mentioned before, we are looking into allowing nodes names that are not the private DNS of the EC2 instance. In the near future, we will support node naming policy other than private DNS names. Implementation of this interface will disable cores to zones interface. So zones will be is deprecated in favor of retrieving zones or region information from instances V2. This interface will not be core if instances V2 is enabled. The existing zones interface were mainly created for the class to interface with the local metadata server to fetch a nodes zone and region. With external Cloud providers, it makes more sense for the zone or region logic to be coupled to the instances interface. After Kubernetes version 1.20, zone and region information should be added in the instance metadata type. We are using as part of the new instances V2 interface in the Cloud provider. So in most cases, the zone or region information is content in the gate gets instance API core of a Cloud provider. This change will reduce the extra core for zone or region in the Cloud provider. So for load balancer, we have an initial pass of the load balancer interface that is currently working in progress. There are some key issues we need to address. So like first, right now load balancer can only be created with the machine generator named, making it very hard to distinguish in AWS if multiple load balancers have been created within an account. Also, the default load balancer named for the Cloud provider from the Cloud provider package, which is currently in use, have been deprecated and needs replacement. Giving load balancers easy to read names well greatly enhance the user experience. It would be nice if we could pick a template to use for naming. For example, a combination of cluster name and service name minds work well. And prefixing the load balancer name with some meaningful name, like the environments, the app stack, et cetera, will be very much helpful. And we also consider adding something to the service annotations and allow users to manage this. Another thing we are looking into is we should build more expressive APIs for load balancer configuration options. Right now, it's an explosion of annotations that are really hard to comprehend. So I'll pass to Arbuck to talk about credential provider. Thanks, Nicole. My name is Arbuck, and I am a software engineer at AWS working on EKS. And when we talk about the production provider extraction effort, it provides some context and then briefly explain what it actually means for the cluster operators. So far, we have been mostly talking about the responsibility of the functional plane. But as Nick mentioned, Toblet also includes some cloud-specific code. I'll be talking on one of those specific bases where Toblet runs cloud-specific code. And that is fetching image credentials for non-public registries. Whenever Toblet needs an image for a non-public registry, it somehow has to fetch the credentials so it can actually get that image. And currently, Toblet does this itself. As you can imagine, each registry has different APIs, different expectations, and Toblet has specific code for each different registry to fetch those credentials. And that is actually not what we want at this point because we're extracting cloud-specific code. So somehow we have to get rid of this responsibility. And in order to remove this responsibility from the Toblet, a new cap was proposed. And this cap basically proposes a new plugin-based approach where Toblet basically talks to these standard binary plugins to fetch the credentials for itself. So how it happens is basically these plugins are nothing but standard binary running along with your Toblet on your nodes. And whenever Toblet needs credentials to fetch an image, it will basically communicate via start.it. And then it asks this binary to fetch the credentials for itself. They use well-formered JSON. It's versioned and serialized. But again, everything is happening through a start-to-start input and output. I strongly suggest reading cap. If you'd like to learn more, it's very well written and it explains how this all connects. Next slide, please. So, OK, but it's not going to fetch these credentials for us anymore, but what does it mean for us? Do we have to do anything? And the answer is yes. It means cluster operators will have to configure a Toblet so it knows what plugin to use or what image. Obviously, initially, Toblet will have no idea about for what image it needs to call, what kind of binary. So what they achieve is there are two new flags you have to pass to the Toblet. So the first one here you saw on the screen is basically the location of the binaries and primary plugins. And this is the folder where your binaries are located and is where the Toblet is going to be looking for these binaries. And the second one is a pretty important one. It is the conflict file you pass to Toblet. And this conflict file will tell Toblet which binary to use for what kind of images. It is basically a list of regular expression. To plot provider name pairs and provider name here is the name of the binary. And Toblet basically, whenever there is a match with the regular expression, you'll use the given binary for fetching images. That being said, it has to be in a very specific format and you can read the cap to learn about this format. For specific AWS users, we already have the ECR plugin implemented already. You can see it on our repository. We are not shipping it with the cloud provider yet, but we are planning to ship it in the future. And we also have more detailed examples and instructions on the documentation. And we also have a more important, we also have an example config you can just use. So basically, your ECR logic just keeps out of the box. And with that, I'm going to give it to Yang to talk about the controller. Yeah, Yang. Sorry. So I'm Yang from Amazon EKS team. So today I will talk about the advanced controller. It's a controller to help manage elastic load balancers for Kubernetes cluster, which satisfies Kubernetes ingress resources by provisioning application of the balancers and satisfying Kubernetes services resources by provisioning network of the balancers. Formally, it's known as AWS ALB ingress controller. We did a rewrite and rebranded to be adapt from the balancers controller in October last year. So I'll talk about our supported country. So currently with ingress resources and application of the balancers, we support instance mode and IP mode. For instance mode, the node will be registered targets. Traffic will reach the node code for your service and proceed to your containers. And for IP modes, traffic will reach your containers directly, but your C&I must be using Amazon's VPC C&I. And we also support a feature called ingress groups, which allows users to specify a set of ingresses to be supported by a single application load balancers, which can help customers to save costs. For services resources with network of the balancers, we currently only support IP mode, which is targeted in the container cluster. And we provide a compatible API with the entry service NLB support. Along with these two features, we also support a new series called target group binding, which exposes applications with an existing target group, like ALB target group or NLB target group. So this allows users to bring their own load balancers and target groups and just let the Kubernetes manage the targets inside high boosts. So customers can use external tools like confirmation, CDK or Terraform to manage their load balancer infrastructure. So what's coming next? So we will read new versions shortly. The new version will include support for NLB instance mode, which will be similar to what we have in the entry controller. After this feature is released, we recommend to use an ADAPT in the master controllers to manage the NLB for your Kubernetes service, which is more flexible. And we will support new NLB features more up to date. And it also addresses some technical limitations in the existing cloud provider. And we will also support user to specify the state IPs for NLBs. And we will also add a support called nodes and netters for instance mode, which allows you to specify the set of nodes used at the back end for your service. So you can use a label's netter to match the nodes. This feature will be supported for both Ingress and service and our target group binding CRD. For Ingress, we will add a new CRD called Ingress Cast Parameters. So it's for Ingress Cast support added from Kubernetes 118, where you can specify Ingress Cast for your Ingresses, which contains additional settings. And along with the Ingress Cast, you can specify controller-specific settings using Ingress Cast Parameters we introduced. So inside Ingress Cast Parameters, administrators can set up additional restrictions on the ALB settings, such as whether the ALB should be public or not. And also since like Ingress Grouping or IP address types. Also for Ingress, we will add support for the new pass type added from Kubernetes 118. And we will also offer a feature to simplify the SS reduction configuration for Ingresses, which customer can use a single occasion to configure the ALB to redirect all HTTP traffic to HTTPS. That's all the new features coming for AWS in the beta component. And hand it over to Nick. OK, so yeah, with that wraps up our presentation on the CloudFighter AWS. So we have some links here. You can see the CloudFighter AWS repository, AWS load balancer controller repository. The first link is to the documentation for the Cloud Controller Manager. And any questions that you have, we would be happy to answer either now or on. You can always find us on Kubernetes Slack. We have our Slack user names linked down below. So thank you very much. Appreciate you attending the talk.