 It's the Kube API server proxy service. So I'm an engineer at Google. My GitHub is Chef Taco. And I'm the SIG Cloud provider TL. I'm also one of the principal contributors on SIG API machinery. My name is Yong Kun. I'm a world's colleague and also working at Google, mainly focused on networking. So Kube API server. We're adding a new proxy service. It's an extensibility point. And the questions we're going to be focusing on for this are, why are we doing? How does it work? How do you use it? What's next? And can you help us? And by the way, spoiler for the last question, yes, you can help us. So the Kube API server, we were adding this network proxy. We also call it connectivity service. And one of the first questions I tend to get is, is this a requirement? And the answer is yes. Follow-on is, of course, why? Why do we need it? So there's a few reasons why we need it. The first one, as SIG Cloud provider TL, nearest and dearest to my heart, is that there should be no preferential cloud providers in Kubernetes. Today, we have eight cloud providers that get to live in the main repo. And no other cloud provider gets that advantage. And that advantage needs to be broken. Unfortunately, we have particular bits of code, specifically having to do with the SSH tunnel, but insert propagation that cannot be removed until we have a replacement. And this is that replacement. Another big thing is the Kube API server should always be able to talk to the cluster it's serving. Now, most of the time, communication is in the other direction. Most of the time, the cluster talks to the Kube API server. But I'll get into details of when the Kube API server needs to talk to the cluster. And if you're thinking a standard easy configuration, what you do on your box, that's pretty easy. It just talks. But there are interesting scenarios, such as hybrid clouds, where it's actually not that easy to have the Kube API server talk to the cluster. And then the last thing on the requirements is there are some interesting vulnerabilities that exist today that we would like to close. So if I look at certain functions of the Kube API server like proxy, it's actually possible to configure the proxy to act as a channel that can then be used for an attack vector against the control plane. So if I know where the traffic is intended to go, it's intended to go to the cluster, and I stick a firewall between the cluster and things that the cluster shouldn't be talking to on the control plane, like the LCD server, then I'm providing a significant increase in the protection that is provided to Kubernetes. So requirements, but do I want it? I mean, we need it, but do I want it? And I'm going to claim that, yes, I actually, in addition to needing this solution, I want this solution, and you should want the solution too. One thing is SSH tunnels. So today, when the Kube API server wants to talk to the cluster, there are two solutions. You either just drop the traffic onto the network and it gets there, or you use this very archaic, unpleasant solution called SSH tunnels, which requires that you run an SSHD demon on your cluster that you generate certs to those. And it's a mess. You don't want to do it. This is a much cleaner solution. It's also an extensibility point. So if I want to be able to do something like talk between the Kube API server and my cluster on a VPN that my particular IT department tells me I need to use, this is a mechanism that will allow me to do that, in which the SSH tunnel solution doesn't have, and there is no other way to do it. With this, we give you the extensibility that would allow you to do that. So I promised to tell you about when the Kube API server wanted to reach out and talk to something else. So the first and primary thing we think of that it wants to talk to on the master is at CD. That's where it stores its data. That's the primary thing it's going to talk to. But also on the API server, also on the control plane, other things the API server needs to talk to, including quite a few things, webhooks, aggregated API servers, et cetera. There are also quite a few things on the cluster that it needs to talk to. And the sharp eye of you will have noted that in fact, there's a couple of duplicates here. We see things like admission webhooks that are both on the master and on the cluster. How do I know which one I want to talk to? It turns out that the configuration of the webhook will tell me whether I should expect that admission webhook to be running in the control plane, i.e. it's the sort of webhook that the cloud provider is putting in, or it's the sort of webhook that the customer is putting in. And this configuration on the CUBE API server will route the traffic appropriately to either the control plane or the data plane for the cluster. So what does that look like? So I've got SCD, I've got the master, I've got the cluster, and I've got traffic. So we see the API server request like CUBE Cuddle Proxy comes into the API server. And the API server has the option of routing that traffic. And I could either just route it the old way like we see between the API server and SCD, where it's just a direct connect to the SCD, or I can choose to route it through a dedicated connectivity proxy. And here we have an example of having one for the master and one for the cluster. And those connectivity proxies know how to get the traffic where it belongs. Thanks, Water. So, sorry. I shouldn't move it, my bad. So let's talk into some of the technical details of how we implement this proxy server. So if you look at the top box, we have a proxy server. And in proxy server, we basically expose two interfaces for the CUBE API server to route the traffic into the cluster network. So one of them is HTTP Connect. The other is GRPC. So for HTTP Connect, it's actually HTTP standard. So just like get, put, and post, it's connected to another HTTP method. So Connect comes with the perimeter IP and port. So if a web server received a connect request from the client, what it does is it does two things. One of them is it just connect to the IP and port as a proxy. And the second thing it does is to upgrade that connection to the TCP. So basically it's a TCP over HTTP proxy. And it's exactly what you want. And the good thing about that is it's, since it's a standard, so most likely you have native support for a lot of different languages. And we don't go that have native support for that. So that's the only thing we care. So that's good for us. And another thing is that it doesn't support UDP down. We're not really care too much about that. So that's basically what we use right now. And for GRPC, it's based on GRPC streaming API. So it's basically a TCP over GRPC. It supports UDP, but it doesn't require a specific client installed on the client side so that it translates your local connection, socket connection, into the GRPC-based implementation. So that's a little bit complex. But it's still there. If you want to use it, you can still use it. And then, of course, both of the interfaces are secured with MTRS. So that means your payload is secured, encrypted. And also, both parties of the client server are correctly authenticated with each other. Now you have that. So the question becomes how the product server know how to route the traffic into the cluster network. So one of the options that we do for SSH tunnel, basically, it just call into the cluster network. But we're not ready to make that assumption that the processor always knows have a routable connection to the cluster network. I know that most cases, it's true, but it's not always true. We don't want to make that assumption. So what we do is, spoiler, so we have this agent to proxy tunnel. If you notice that it's the wrong direction, it's a dialback from agent to the proxy. But if you think about that, it's exactly what Kublai does. What Kublai does is he registers an API server and say, hey, I'm a node. I can schedule workload for you. And it's a similar thing. And the agent basically dial back to the proxy server and say, hey, there's a network here. And I'm ready to accept the traffic from you. And that's how it works. And also, this is secured by MTRS. It's even more important because we don't know that between the proxy server and agent, it could be an untrustable network. It could be internet. And it must be secured. And we must authenticate as well. We don't want the traffic to be routed to a malicious site. Space. So of course, the agent is responsible for distributing a stream to different endpoints. And also responsible for opening and closing connection to the endpoints, like no port and webhook. So if you look at the full data flow and the dotted red arrows, so the request coming from the HTTP Connect or JRPC interface from the API server, it goes into the proxy server and over the agent to proxy tunnel eventually get to the agent and agent distributed the stream to the endpoints, no endpoints. Yeah, if you zoom in a little bit more, then you'll see the agent to proxy server tunnel. And it's basically also implemented with the JRPC five direction stream API. And yes, it's that. And of course, it's a full duplex connection. What it means that, of course, it's a full duplex. It's a TCP connection. It means that both parties of the both end of the connection can read and write at the same time. Otherwise, it's not TCP. So we have to have that. And also, we need to multiplex over the one tunnel. What it means is that if you have multiple connections in the tunnel, it must be supported. That means that if you have multiple connections over the same tunnel and read and write at the same time, it also works. So how we implement that is that we have this proto buffer defined data structure called packet, a packet that we have two types of packets. One of them is called data packet. It's basically the data you want to transfer over the wire. And also, the control packet is basically where you want to open the connection, close the connection, et cetera. And also, response of that control signal and some of the connection ID, et cetera. And with this implementation, we could have support for a reasonable connection. What it means that if the GRPC stream gets resetted, the connection dead, as long as the agent can connect back to the proxy server, all the connections will be alive. And with S2Tunnel, you can do that if a tunnel is dead and all the connections are reset. And of course, we can support more advanced features. Like if you have multiple agents dial back to the proxy server, all of a sudden, your proxy server have multiple choices, then it becomes a load balancer. And we can implement a different load balancer algorithm to share the load across different agents. And also, we can implement monitoring, auditing. You want to audit a specific user. And also, for example, you don't want to, you want to be fair to all the connections. You don't want one connection to drain all the traffic and all the other connections are not as fair. So of course, all of these are not implemented yet. The project is still in the early stage. So that's why we're here. And we ask your help. If you are new to the community or you're a veteran, you're welcome to join. We have slides that have a lot of links on it. The code is in GitHub, and you're welcome to contribute. And we're ready. I didn't even think I had. Is that better? Come on, come back. Come back. All right. So we've taken a look at this. And one of the distinctions I would like to make here is part of what you're seeing is part of the CAP and part of the definite implementation. And part of it is part of the reference implementation. So if you think back to what Yong Kun was just describing, everything he was describing is part of the reference implementation. And we'll get into it. We have working code. But it is a reference implementation. The reference is there for you to either use or to take and modify as you need it in your cloud. So how do I configure my Kube API server to then do these wonderful things I've been describing? So we have a new flag, the network proxy configuration file flag. And it takes a YAML file. And that YAML file is a connectivity service configuration object. It looks just like a standard resource. It isn't, however, a standard resource. And the reason is that we don't necessarily want to allow customers to modify this at runtime. So this is a object that gets loaded when the Kube API server comes up and is not modifiable, but is otherwise much like other resources we have. And in this example, we'll see that we have three connectivity services configured. We see one that says, I'm a cluster. I'm going to connect via HTTP Connect. I'm going to be connecting via HTTPS on port 8131. And here's all of my credential information. And on the master, I'm actually not going to worry. I'm just going to do direct connect to using TCPIP like normal. So the question would become, hey, what is this configuration that I just described to give us? And hey, if you say this works, where can I find the code? So what it gives us is actually this image we have here. So we see a Kube API server connecting to the cluster connectivity proxy. And we see we have two connectivity agents, one per node, presumably, connecting via gRPC, also to the connectivity proxy. But the Kube API server is actually doing direct reach out to both the SCD and things like the Webhook server. And any requests, as Jung couldn't describe earlier, that the API server makes are going to go through the proxy to the agent and then land in the appropriate place. So that's kind of what I just described. And let me go down just a little bit. So where could I find this code? Before I get into the details here, I will point out that all of these slides are available on the KubeCon site for this talk. So you can download this PDF. And you can find these slides there, this slide there, and get the links from that. So I've got three links up here. The first link is the PR, which I'm trying to merge that actually makes the Kube API server changes that I've described. The second link is the home of the reference implementation. So a lot of mine, and especially Jung couldn't work on getting the proxy server and the proxy agent working, all exists under that. And we'll get later on how you can help us make interesting changes to that code. And the last link is if you want to get much finer grained detail on exactly what is approved, we have the approved cap for this entire feature under that last link. All right. So what do the Kube API server code changes look like? So the first one is that we start out with this idea of a network context. And you want to be able to look, you set a network context, you ask for a lookup on that network context. And what you get is a context-aware dialer. And if we look at the network context, we're going to see that right now all it contains is a connectivity service name. That's presumably going to be either SCD master or cluster, very simple. So why did we worry about having a network context object to hold what amounts to one string or enum? The reason is that we have ideas on how we want to enhance this. I'll get into a lot of options on how this might get enhanced later. But an obvious thing might be that if I want to do a multi-tenant system, I might want to be able to indicate my tenants in the network context and have the lookup provide back the dialer that talks to that tenant's cluster. So that's just one example of how this might work. And then for the concrete usage, whether this is a log call or a webhook call, at the appropriate place, you would set up your contacts saying, hey, I need to talk to the cluster. Do the lookup. You get back your context dialer or an error, hopefully not. And then you're going to do a wrap transport to get back the transport that uses that dialer to actually forward the traffic. All right, so I sort of hinted at possible futures earlier, but there's quite a few possible futures and ways that we can really improve this feature. So the alternate communication, this is an interesting one, because this isn't even an in Kubernetes future. When I talked about extensibility, this is the extensibility future. So it's actually relatively easy if you wanted to go in and replace the GRPC that we have between the proxy server and the proxy agent with something like open VPN or strong swan or wire guard or whatever mechanism it is that you need to use to talk between your master and your cluster. So that's fine. I kind of hinted at the next one, but connectivity service configurations beyond the simple master cluster net CD. So I might want to do connectivity by label. If I know that I have a clusters in multiple zones, I may want to label the zone, and then instead of just going to cluster, I can send the traffic to the right zone. And that's very nice because then I don't get to that zone and then have to route a hop across the Pacific. So that's one very nice use that I can get out of labels to make everything work a little bit faster. I can do connectivity by service. This is the sort of thing I might use for doing multi-tenants. It can also just help me do things like make sure that not only do I land in the right zone, I could even make sure that my traffic lands on the right node. Yeah, as I said, there are several interesting multi-tenant use cases. It's also interesting because other things you might think about, I mean, if I have distributed masters and I know that I want to make a master call to one of the masters in another zone, I could do that by making very smart uses of my time. I can also allow better connections than HTTP Connect from Kube API server to connectivity server. So HTTP Connect had one very nice advantage right now. That advantage is that it is as close to a standard as we could come up with as an open standard for the communication between the Kube API server and the proxy server. GRPC, we tend to like it. It has very nice multiplexing behaviors, nicer than what you get out of HTTP Connect. On the other hand, I can prove to you that the client that we use for HTTP Connect is a standard because Curl will do every bit exactly the same thing. So we can look at exactly how important one is the other. We have a trade-off there. I know Amazon is very interested in this last one. We can allow the Kube API server to not be public. So today, if I have a remote distance between my cluster and my master and the cluster wants to talk to the master, it's got to talk to a public IP address to get to the master. And if it's a public IP address the Kube API server is talking on, that is a potential attack vector. What this can do is once I've set up this tunnel, then nodes and things running on nodes, instead of talking directly to the Kube API server, they can send their traffic through the agent, reversing everything we've shown you so far, have their traffic securely land on the control plane where it will go to the Kube API server. And we can have special logic in the agent to make sure that that traffic is only destined for the Kube API server. So this is another example of the sort of things that this feature buys us. All right. So I'm hoping there are a few folks here who've been convinced this is an interesting problem and that they'd like to contribute. So how can you get contribute? So Young Kun and I, being members of API Machinery Network and Cloud Provider, will of course be suggesting that a great six to work in our API Machinery Networking and Cloud Provider. There are home SIGs. We all do interesting work, and I highly recommend them, especially if you're interested in this sort of problem. I'm also going to say that the API server network proxy is a special project that Young Kun and I are the main maintainers of under Kubernetes SIGs. It's a small project. It's a much less intimidating code base than the main Kubernetes one. And you're going to get a lot more attention if you want to get started there. You're also welcome to help contribute on the main one, but I will tell you that that's a much smaller code base, and it's a smaller community, which means you'll get more attention, and we would love to have you there. Yeah, I know it's a very long day, and it's the last session. And none of you really want to be here. So yeah, I want to add that the contribution is not just limited to a full request. It could be a bug report, feature request, and some use cases, discussion, and it's all welcome. And yeah, that's it. So now would be the standard time for questions and answers. So any questions? Cool. Well, thank you all very much for coming, and I hope you found this useful. And we'll be here if you want to talk private. We'll be here.