 Hello, welcome to SIG Network Intro and Deep Dive. My name is Rich Schroener. I work for Sunder Networks. I've been a Kubernetes user for around three years. I'll start off with the introduction. And later we'll hear from Bowie Do, works at Google, about the Deep Dive and ongoing efforts within SIG Network. So as I mentioned, we'll start off with kind of an overview of some of the basics material to get you started, and then we'll transition into a deeper dive and look at the ongoing work. First off, what is SIG Network? This is a special interest group, which is a subset of the broader Kubernetes community focused on a particular area. In this case, it's networking. We're charged with pod networking, either between nodes or between pods. The service abstraction ingress and egress flows as well as the network policies and access control that a user may want to apply to those flows. Any of these things sound interesting. I encourage you to join us either on our Slack channel, SIG Network, the community mailer, or you can see us on GitHub under community SIG Network. This will be available at the end to grab the links. There's many APIs in the Kubernetes domain. Some I'd like to call out in particular, which SIGNet owns are service endpoint and endpoint slice. This is everything regarding either the service registration or publishing from the client side of the discovery. When you go up a layer from just the networking and start talking about the application, L7 HTTP routing, that would fall under the ingress API and feedback from the community as well as some challenges that have been seen in the initial 4v1 rollout. I haven't led to this gateway next generation. HTTP routing and service ingress. There's a lot of exciting work going on there and we'll hear about that from Bowie later on. Finally, we have network policy API, which is your application firewall. So the implementation of these APIs and components that kind of build up the responsibility that SIG Network has. First, start off with the Kubelet CNI. This is the container network interface. This is that low level network plumbing. How does this pod have its IP and how is it plugged to the network? Cube Proxy is responsible for the service layer. Implements that service API and we'll look at some more of the details here in just a moment. As far as the controllers, we have an endpoint slice controller. Empoint slices are a feature going GA hopefully in 120 as well as the service load balancer controller. It's constantly reconciling the state of the service objects and ensuring that in the cluster. Additionally, I mentioned the IPs. There's an IPAM controller and by default, you have a DNS name-based discovery with your default Kubernetes cluster. Started to begin to talk about the networking model. It's important to note that while the networking model dictates all pods, can reach all other pods across nodes and any location in the cluster. There is no special meaning necessarily for a pod's IP. It's just like anything else. Whether it be a VM or a container in the inside of a node, depending on your provider networking, you have many options. A few of them are listed here, like flat or overlays or if you're looking to scale and get that routing update. There are solutions such as BGP to accomplish that. Let's start with the problem at hand. I have a client and a bunch of servers. Which one should I connect to? Well, hopefully I didn't pick server two because after a long enough time period, everything will fail eventually. But maybe this is only temporary. If we try again, I'll pick maybe server one. How long can we continue on? We're just picking a random IP and hoping that server stays up forever. And there's a lot of minute issues that come up from clients that have to either reconnect. They might have to reconfigure the IP of the server or other challenges that we look to solve by abstracting this problem. So that is the service. We have one that's heard of the service object and it's logically placed in between the client and these backend servers again so that we can connect to the same service without having to worry about whether the servers in the backend are going up or down. So service, it exposes a group of pods, has a durable VIP. There are cases where you don't have to take this option if you have smarter clients that can handle DNS changes. But in most cases, clients like to have that more durable IP and that is the benefit that you get from using a service to connect to as opposed to direct pod connection. So Qproxy implements this using IP tables and IPvS but if we think about the flow of operations from a client, it goes to connect to a service. It's gonna have to resolve that and then eventually it'll reach its backend server. Let's kinda watch how that flow comes out in real life. Now, the DNS pod is actually a service itself but to avoid getting too wrapped up with our explanation, we'll ignore that fact for now and just imagine that we can resolve a service that we're concerned with, pod server in this case. When we query the DNS service, it returns the VIP that we were talking about. The client starts a TCP session or UDP depending on the protocol with this VIP and those Qproxy rules we were talking about translates this VIP to a actual pod IP. Now, when we actually meet the backend server, we won't have a destination VIP in the packet anymore but the Qproxy has a record that knows how to reverse this abstraction on the return trip back to the client. So now, if we go back to our initial example, a server two were to go down. I'm still connecting to the same IP address just as before. The worst thing that I have to do is have a client start a reconnection. So no reconfiguration of other IPs and all in all this tends to work out better than directly contacting the server. So the actual specification for this object has the standard metadata that you'll see across all Kubernetes objects and a selector. So the label selection mechanism is pretty popular across not just the networking area but Kubernetes in general. The service object leverages that same label selector and it indicates which pods are implementers or our servers of this service abstraction. You have to supply a port as well as a target port. The port being the front end TCP UDP depending protocol port that you would contact and the target port is the backend server. So actually having this layer of port abstraction turns out to make styles of development a lot easier. Not only do you get that benefit of the ability of the client IP address that you're connecting to or DNS service, additionally, you have the ability to remap protocol ports. The controller will see this new object that you've created and fill out some of the fields if you chose to not specify them as a user. Namely in the port section, we see that we actually have the protocol TCP now which is the default, but there are other protocols such as UDP that you can use as well. We're allocated a cluster IP and this is that ephemeral, I mean excuse me, this is that durable VIP that we discussed and that'll stay throughout the lifetime of this service even as servers selected by this app, my app selector go up and down or come into and out of existence. Next we'll hear from Bowie. Now let's talk about endpoints. A service is virtual, so how does the proxy know what pods to send traffic to? Like most things in Kubernetes, it's going to be a controller that helps facilitate this. The endpoints controller converts end pods into a smaller set of endpoints. And inside the endpoints resource, this represents a list of IPs that are behind a service. Now this is usually pods, but it's not always. You can manually populate an endpoint resource as well. Now recall that service had a port and target port fields. So an endpoints resource can also remap ports. You may have a port coming in that remaps to an actual port on the container. And generally endpoints are managed by the system, the controller we alluded to earlier, but as I said before, they can always be manually managed in some cases. Now let's look into what an endpoints controller does. So let's say we are given the service, name foo, selector app foo, ports 80 and a target port 9376. Let's say we have a bunch of pods that match the label selector for the service. So these are a bunch of pods that are floating around and they have app foo, some of them have app bar, some of the app quacks. The service will only select the pods that have app foo. Now what happens is that the endpoints controller will then take the pods that match the service and put them in the endpoints object. Now the endpoints object doesn't contain all the information about the pods. It only contains the endpoints information, which is basically the IP. Another way of doing service discovery is DNS. So DNS starts with a specification describing A records, quadruple A records, SRV, PTR records. And this generally runs as a service implemented using pods inside the cluster, but it doesn't have to. Anything that matches the DNS specification can serve as a Kubernetes DNS. Finally, the containers are configured by KubeLit to use Kube DNS or the provider of Kube DNS as their DNS provider. And part of this is actually to use Lipsy search paths to make it even easier to use the records that come with Kubernetes. Currently the default implementation is core DNS, which lots of people run as a service inside their cluster, but you could imagine any such implementation that matches that specification could also be a provider for DNS for Kubernetes. Now let's look into the details about how DNS and Kubernetes works. So here is sort of the most typical record that you will use. It's a DNS record for a service and you'll see that it's composed of four pieces. The first one is the name of your service. Then there's the namespace of your service lives in and then there's just SVC, which is a sigil or sort of like a constant there that tells us that this is a service name. And finally, there is the cluster's DNS zone. In this case, we have cluster.local, but this is actually not strictly required. You can configure your cluster DNS zone during setup to something else. Now let's talk about Kube Proxy. So Kube Proxy is the default implementation of services. But one thing to note is like most things in Kubernetes, it can be replaced. So Kube Proxy is a default implementation. It's part of the main repository. And it basically allows every pod and every node on the cluster to access the service virtual IPs to access the pods behind the service. What Kube Proxy does in its implementation is it uses the node that's a proxy for traffic from pods on that node to the back ends of the service. And there are actually many different ways to do this. We can use IP tables, IPvS, on Windows you can use the Windows kernel or there's even a user space option. In fact, that was the original implementation. On Linux, IP tables and IPvS are the best choice. They're in kernel and they're very high performance. The key to Kube Proxy is that it's transparent to the consumers. Anyone accessing the service VIP virtual IP won't actually have to be aware of the fact that there's this Kube Proxy in between. Now, what does Kube Proxy do on the control path? Well, it mainly watches services and end points. As part of this watch, it applies some filters. For example, it ignores headless services and then it links the end points which are the back ends, the pods that are selected with services, which are the front ends and basically by service, we mean the virtual IPs. It accumulates changes to both and then it updates the kernel state which is the node level proxy on the node itself. What does Kube Proxy do on the data path? Well, on the data path, it has to recognize that service traffic exists. For example, your pod is sending traffic to a particular destination virtual IP import. Then it has to apply some kind of round robin load balancing technique. Actually, it's random. It chooses a backend and also implements some of the service attributes such as client affinity on this path. Then finally, it uses network address translation to rewrite packets to go to a new destination. And of course, any kind of load balancing with network address translation, you have to undo the address translation on the response. Couple of common questions that come up. Why not just use DNS to do load balancing? Well, people have done this in the past but what we have found is that typically DNS clients are somewhat historically broken. I think one of the common offenders was Java way back in the day and they don't handle changes to DNS records very well. If you think about it, DNS typically you do a single lookup and then you kind of forget about the fact that that record could change out from under you. The next question would be, well, my clients already can do this kind of load balancing inside the client itself. Common examples are thick clients like the Redis client library or for example, GRPC, can I opt out? The answer is yes. So the way you opt out is you can have a headless service which basically gets a DNS name that returns a list of backends as A records but doesn't really need the virtual IP and hence has no virtual IP. Services in Kubernetes are also how you configure L4 load balancers. So Qproxy provides L4 load balancing within the cluster and between pods to cluster IP, virtual IPs of Kubernetes services. But you can also configure L4 load balancers that take traffic from clients that are outside of the cluster. So different LBs work in different ways which is too broad for this talk but basically Kubernetes integrates with most cloud providers to use their network load balancer, the external load balancer, internal load balancer and configure a cloud-based load balancer for your Kubernetes service. Let's talk about Ingress. So Ingress describes a HTTP proxy and routing rules and it's a very simple API. What it does is it matches host names and URL path. What an Ingress does is it describes a very basic HTTP router and it can target a service for each rule in the Ingress. Kubernetes defines this API but implementations are third party. So actually there's many, many, many implementations of the Ingress API out there. And we find that there is basically integration with almost every cloud and with every popular software LB. Here's an example of an Ingress configuration. On the left, the yellow, we have the Ingress with a host name and the number of paths. Now, since Ingress is an L7 concept, it typically stitches together multiple services to serve a given, for example, website. So in this case, we have a website that has a foo, a service and a bar service and they're mapped to these HTTP paths foo, slash foo and slash bar. And they point to a service, which is foo service in a service, which is bar service, which points to the actual back ends of the pods that serve that service. It's common questions that people ask about Ingress. One question, how is this different from service load balancer? So service load balancer basically is somewhat of an L4-ish concept, talking about virtual IPs, ports and protocol like TCP, UDP, now STTP. Whereas the Ingress talks about an L7 protocol and actually stitches together potentially multiple services into one L7 service. Next question is why isn't there a controller in the box? So what is interesting about at least the Ingress space is that there were, you know, HTTP proxying has been around forever basically and we didn't want to pick any winners. There were just so many implementations out there that were very, very mature. Now looking back, this might have been a mistake, but given the, just the number of implementations, I would say that the API is fairly successful even despite this. Network policy. Network policy is a way for Kubernetes users to describe how pods can talk to each other and how pods actually can't talk to each other. Basically it describes the allowed call graph for communications. For example, a back end can talk to a front end and a front end can talk to a back end and a back end can talk to a DB but the front end can never talk directly to a DB. If a front end never attempts to talk to a DB that probably is a compromise or a bug. So just like Ingress, implementations are mostly third party and they're often coupled tightly with the low level network driver. The reason for this is that a lot of the logic for accoling will also involve dealing with the packets that flow between the containers. Network policy is a very simple API. So it has very simple rules and has been designed focused on app owners rather than a cluster or network admins. So this is something that is currently evolving in the sake and we may need a related but different API for the cluster operators. Here's the example of how network policy works. We have a bunch of front ends. On the left, you have a bunch of back ends in the middle and we have a bunch of databases in the back. And we can write a network policy rule that allows only front ends to talk to back ends and back ends to talk to databases. But as you see here, those red lines, a front end talking to a database or front end talking to another front end are disallowed. And if the front end tries to send traffic in this way, the packets will just get dropped. Let's get into part two, which is a deep dive into new work that's going on in the SIG. Okay, so in part two of the deep dive, we're gonna talk about ongoing work in the SIG. Some of these things have reached GA stage. Some are in alpha and some are in beta. Topics that we wanna highlight today in this session are work on improving DNS scalability, improving endpoint API scalability, improving services, expressability, orthogonality, and being able to map services and to express a more complicated role-based delegation of responsibilities for services. And finally, IPv6 dual stack. Let's talk about node local DNS. So this has been a perennial topic, Kubernetes. I think there's been probably more than five talks at KubeCon about how to deal with DNS. But the truth be told, Kubernetes DNS resource cost is high. So first, we have expansions of DNS queries due to alias names. Alias names are great because you can create applications that look up names that are sort of independent of which namespace it's living in, which cluster it's living in, but implementing these aliases is pretty expensive because it requires this expansion. The other interesting thing is that we notice that Kubernetes also has high DNS requests because of application latency. So if you have a bunch of microservices, now instead of your VM having a single application, you'll actually pack it quite full with different applications and they all are doing the same amount of DNS. We also have noticed that there are many DNS heavy application libraries, such as Node.js. And finally, because the way DNS is typically implemented using Kube Proxy, the way that Kube Proxy uses contract, interacts badly with UDP. Contract wants to track connections. UDP has no sense of connections. So what's the solution? No local DNS. And this has gone GA in 1.18. As with most systems problems, there's the two things you go to, cache level of indirection, while the solution in this case is to run a cache on every node and add a level of indirection to the Kube DNS service. So we have to be careful about this because as soon as you talk about running something on every node, now you have increased the per node overhead and this can become easily dominate in large clusters. The other thing that we need to be aware of is that as a system critical service in a Damon set, we need to be really careful about high availability during upgrades and failures. Let's take a look in detail about how node local DNS works. So here we have the current way Kube DNS operates. Kube DNS in yellow on the right are a set of back ends that are part of a service, Kube DNS service that has a virtual IP. And that virtual IP is used by Kube Blitz to populate the upstream DNS server for the pods. Here's what happens when node local DNS is installed on the node. Node local DNS is a Damon set so it runs side by side with the pod on the node. And as part of running, it also installs a dummy interface and a set of no track rules. No track rules basically tell Kube Proxy that if you are sending traffic to the Kube DNS virtual IP, which is 10.0.0.10, actually to skip Kube Proxy and send the packets to the node local DNS, Damon. No local DNS will then take those packets and send them out directly to a replica of the Kube DNS service called Kube DNS upstream, which has a different IP address and that will reach the Kube DNS or core DNS service. Now, what's nice about this is that when no local DNS, let's say it runs into trouble or crashes in some way, there is an agent that's a sidecar of no local DNS that will remove the node track rule. In that case, after this crash, the node track rule will disappear. Your pod will then have its DNS tracker traffic not captured by the dummy interface. In fact, it will go through Kube Proxy and it will go back up to the upstream. So in this way, no local DNS is able to transparently insert itself and remove itself from your node without actually disrupting the DNS for your pod. And this is super important because many applications if DNS starts acting up, bad things can happen. Let's talk about endpoint slice. Endpoint slice is an interesting exercise in figuring out how to scale a Kubernetes object that has gotten very big. Now, in larger clusters, think 15K nodes and very large services can lead to scalability issues in the API. So as you remember, the endpoints object contains a list of all the endpoints that are a part of a service. For a service that's like five pods, not a big deal. But imagine that you have 5,000 nodes and you have enough endpoints to make up one megabyte. So maybe 1,000, 10,000 pods in the service. Then the amount of data that's going to be sent both written to the database and sent to all the watchers in the whole system, which is basically Kube Proxy in this case actually becomes quite enormous, especially when they get updated. So here we have a simple calculation, 5,000 nodes, endpoints are one megabyte. You'll basically transfer five gigabytes of data, which is basically a DVD. And then if you do a rolling update on the deployment of a 10,000 pod service and each one results in an endpoint's update, you may end up transferring up to 25 terabytes, which is a lot. And all it's doing is trying to tell every Kube Proxy what the list of endpoints are. So what's the endpoint slice solution? So here we have, there's an EP and endpoint. We have a single update to one of the entries in the endpoint and we end up sending the whole thing to every single Kube Proxy. Now with endpoint slice, what we have done is we have sliced up the endpoint. We have multiple slices, slice objects that represent a single endpoint. Here if given a single update to one of the slices, we only need to send the slice. Now clearly this is a trade-off between how many slices you have to deal with versus how small you can make that incremental update. And that's an interesting design question. So endpoint slice to implement it, we have a new controller, actually two new controllers and endpoint slice controller, which basically takes services in the service selector. And instead of creating an endpoints object actually creates slices, multiple objects from the service selector. We also have an endpoint slice mirroring controller which creates slices from a selectorless service. And other users can also create endpoint slices and they set a managed by to say that, hey, these slices belong to some other controller that's not the standard ones managed by Kubernetes. Now, one of the challenges of slicing is that you don't have this one-to-one correspondence between a service and for example, its endpoints object. Instead we have this label which is a service name that maps between a bunch of slices and the service that it represents. So I said before that an endpoint slice is a trade-off between how many slices you make, how small they are, and also how do you update them such that you don't end up in the same situation as before? You kind of want to do three things. You want to keep the number of slices low. You want to minimize the changes of slices per update. And you want to keep the amount of data sent low. So we've settled on basically a fairly simple algorithm for now seems to work okay, but really we need information in practice to see how this behaves. But the current algorithm does the following. First, it gets a list of all the endpoint slices corresponding to service. It runs through the service selector on the pause to figure out what the set of endpoints should be. Then it runs through all the slices in memory and it roofs stale endpoints in the existing slices. Next, it will fill in new endpoints into the free space leftover from the stale endpoints. And finally, it will create new slices only if there's no more room. What this is trying to do is basically try to reuse the existing slice objects without having to create new ones. Now the thing that's not currently happening here is that there's no active rebalancing. For example, if you had a bunch of slices and they basically had all the stale endpoints go down except for one of them, those slices will still be there. But the claim is that this is kind of pathological and really in practice you wouldn't get that kind of situation and instead we would want to avoid churning, deleting and adding slices. So what's the status of endpoint slice? So in V117, endpoint slice controller was beta. In V118, the slice controller was enabled by default but there was no queue proxy. In 119, everything is beta and everything is enabled by default. And finally, we're going to GA in 120. So endpoint slice is also interesting because since it's a core API and so fundamentally the operation of the system, it actually has taken these many releases to kind of get it right and make sure that if you upgrade or downgrade you won't actually end up in a situation where everything breaks. Let's talk about services across clusters. So as Kubernetes installations get bigger, multiple clusters is becoming the norm. And this is especially true on, for example, cloud providers where the cost of creating a cluster is to just spin up a bunch of VMs. Lots of reasons to have multiple clusters. HA, you could have multiple clusters in different regions. Blast radius of configuration changes. So if you do a sort of cluster scope configuration change you can only blow up a small cluster rather than a large cluster. Geography, latency and so forth. Many Kubernetes abstractions. In fact, most of them have sort of assumed to be cluster-centric. Services is one of these things. Services have always been a cluster-centric abstraction. And now in SIG network and actually SIG multi-cluster we're starting to look at how to basically extend the service abstraction and these sets of resources to work across clusters. So how does this work? Given a setup like this, how do clients in the front end talk the back ends? There are always ways existing right now such as talking through load balancer. And it feels like it should be easier than this. So what we're looking at with the multi-cluster service is something like this. That first you define a service, just a plain old Kubernetes service, but you also say that the service is exported. But what does that mean? So what it means is that the service is exported to some group. So we assume that there is some manner of grouping. And in the SIG multi-cluster, this grouping is called a cluster set. So it's a set of clusters that are sort of cooperating in the same administrative domain. We don't make very strong requirements on this grouping, but only that such a grouping exists. Now this leaves room for interesting implementations by not imposing such a strict requirement. But within that grouping, there has to be some sort of semantics. And one of the semantics that we are asserting in this group is that namespaces between different clusters mean the same thing. If you have a prod namespace in this group of clusters in the cluster set, that that prod namespace will sort of have the same ownership and admin model across the clusters. We can build controllers that operate across these clusters and sort of operate across these namespaces that are spanning cluster. What service export does is it triggers the multi-cluster service controller to publish this backend service when you have exported it. What this means is that it will create a service import. It also creates endpoint slices to represent the union of all endpoints across all exported services. In this case, it's just one that is exporting. Now clients in the front end namespace can talk to the backend namespace, even though it exists in a different cluster. Now with services comes DNS records. Multi-cluster service also defines DNS records as well. Just like before we have the name of your service, the namespace that lives in, and a multi-cluster DNS zone. Couple of interesting things you may think of is one, this is supercluster.local. This may be changed in name to clusterset.local. The other thing is, do cluster sets have their own domains? And these are things that we are currently working with, say multi-cluster to kind of figure out what are the use cases and where this should go. Kind of as a more advanced example, that the backend service actually might exist across multiple clusters. So you might have a service export and import that actually aggregates the service backends from both cluster A and cluster B. And if you access the cluster set IP or the supercluster IP, you will access backends from all of these. Services across clusters is mostly kept away right now, but people are very busy hammering out the implementation and the API and the names. And we're still working on some of the semantics. Now, one of the big challenges of Kubernetes or multi-cluster is that it's okay if the things that you're defining in the API don't conflict, but what if you have a bunch of services that are linked together by service export, but you put some config on one cluster that's conflicting with another cluster. Those things have to be resolved. Now let's talk about IPv6, IPv4 dual stack. IPv6 is something that's been rolling out for ages and Kubernetes is no exception. And some users will need to run network services and pods in both IPv4 and IPv6 at the same time. Currently, Kubernetes only supports one pod IP and this IP has to be either IPv4 or IPv6. It cannot be both. Some users need services that are in both IP families. So Kubernetes currently only supports one service IP. It has to be IPv4 or IPv6. So while this change is very small, it's actually quite fundamental. And I think one of the harder things that we do in computer science is to take something that was singular and make it plural. So one question also was, wasn't this done already? The answer is yes, but we found a couple of problems and this actually needed a major reboot. What does supporting dual stack mean for resources? In fact, it affects almost every resource that refers to IP addresses. So for pod in the status, we have a pod IP. Right now it has to be either IPv4 or IPv6. The way we sort of resolve this is to actually add a second field called pod IPs. That's the plural of pod IP. And it basically allows us to have more than one IP where one of them is v4 and one of them is v6. There's a bunch of API machinery to handle the fact that the singular field and the plural field have to be in sync. In this case, it's invalid if you submit a pod resource with a pod IP that's different than the one that appears in the list of pod IPs. The same change has to be done for node. Here the pod ciders are pluralized and API machinery validation guarantees that the pod cider and the pod ciders remain the same. Now let's talk about service. So here we have service and it has a cluster IP that also has to be pluralized. There's an interesting subtlety here. Which one do we give depending on the user's preference? Should we always give IPv4? Should we give an IPv6 address? Should one of them be the default? And that is kind of the crux of the reboot. It turns out that we can't make very strong assumptions about which family is returned in the cluster IP. We have various requirements. And it turns out all of these seem to be valid. The user says, I need a dual stack. I would like dual stack if it's available or I need dual stack. And all of these are things that users could request. So the solution is that it will default to single stack if users don't express a requirement. But if they need a specific requirement they can actually express it in their YAMLs. And this works for headless services, node ports and LBs. If the cloud provider supports basically dual stack LBs. And we're shooting for a second alpha in 120. It's like a very big PR, but it looks like it's on its way there. So this is great. Let's talk about services and the sort of next iteration and how we're thinking about the API. So looking at the current service API it's a resource that describes many things. It's a method of exposure, cluster IP, node port, load balancer. It's a way to group pods. So it's like a selector as we saw in the end points discussion. It also contains a bunch of attributes such as external traffic policy, session affinity and so forth. It turns out that sort of evolving, extending the resource has become quite hard to the fact that all of these fields interact with each other. And the other thing that makes this complicated is that this, as you see on the right hand side there's actually a hierarchy, almost the inheritance of different service types. One of the goals of re-looking at services is to evolve the L7 and L4 and kind of break up service into these more orthogonal features. Along the way we want to provide sort of a way to model different roles that are present in the cluster. One of the ideas is to decouple some of these service concepts and features along role axes. Now whenever you do a role-based design you have to first think about the personas that you want to kind of address. And we kind of split up the roles into the following sets. There's that of a cluster provider, an infrastructure provider. You know, this could be your cloud provider or this could be the person that runs your infrastructure as a service. There's a cluster operator or network operator who basically manages the entire cluster as a whole. And then there are the application developers who basically are given, for example, namespaces in which to do their thing but not control the entire cluster. Now the concepts we talked about in the previous slide we also have split up into different axes. So grouping and selection will remain the domain of the Kubernetes service. Routing and protocol specific attributes will be put into a protocol specific routing resource. And finally, exposure and access is basically the virtual IP, the endpoint that the clients will be contacting will be part of gateway. And finally at the top of this is gateway class. What gateway class does is it says that this particular gateway and route will be implemented by a particular type of provider. Very similar to how ingresses can have ingress class or storage can have a storage class. Let's go through each of these roles and kind of see, get a taste of like what the API looks like. So the infrastructure provider will basically create a bunch of gateway classes that then are backed by controllers that implement a gateway class. So these define a type of service access for the clusters. For example, you can give your consumers of the gateway classes a generic name such as internal proxy, internet LB and kind of abstract them from how these things are implemented. Now this, as I said before, this is simple similar to storage classes which abstract the implementation of mechanism from the consumer. The second persona that we're looking at is the cluster operator net ops. And this person we find is typically determining how services are accessed by the user. What port protocol addresses, TLS certificates, whether or not it's on the internet, whether or not it's internal, what resources, maybe it's expensive, gold, low bouncing or maybe a cheaper infrastructure. And really the cluster operator we feel will control the gateway resource. It's a keystone resource that's one to one with active configuration of some part of the infrastructure. Creating a gateway resource will spawn, for example, a software LB and may add a configuration stanza to an existing LB, may program the SDN. So the gateway will sort of describe how the whole service will be accessed and we also note that this gateway resource may be underspecified. So you can kind of use it to request a particular kind of gateway but the system and the implementation can kind of fill in the details. So here's a sort of a sketch of the gateway resource and you'll see that it's underspecified because all we're doing is asking for a port and a route selector to show you which routes are linked to this gateway but we don't really talk about, for example, which address it's gonna get and those will be dependent on and filled in by the gateway class. The final role that we're looking at when designing these sort of like orthogonal services is that of the application developer. So the application developer is going to come up with the set of routes that make up the, for example, in this case, the store app, which services is going to route to and kind of the protocol specific matching and filtering that's how they implement their application. One interesting thing is that we have done here is in modeling this, we have actually split out each of these routes into their own protocol specific resources. So we have an HTTP route, TCP route and so forth and this allows us to, one, make a resource that's specific to that protocol so it's much easier to understand and two, because they're actually separate resources that can all linked into gateway, it actually creates a bit of extensibility. For example, if your gateway class provider can handle like a new protocol, for example, let's say you do Redis load balancing, you can have a Redis route that can talk about Redis protocol specific details and actually add that to the system without having to say modify the entire standard or send it upstream. What happens to service? So service still exists and all the basic functionality since it's GA and part of the core will always remain but we're hoping to evolve it to a place where basically it's limited to grouping and selection. So it's basically mostly just talking about the set of backends that comprise the service. So if you want functionality still works but hopefully we won't have to add significantly to that surface area to add new interesting functionality on top of it. So here's an example kind of broken out. So we have a gateway class that's referred to by the gateway and the gateway class defines sort of how the implementation will be wiring up the rest of it. For example, internet LB that's implemented by a cloud provider that way the particular gateway class. Then we have gateway which talks about sort of protocol termination which ports how the client is gonna access this and then that will be attached to a number of routes that the application developer will be writing and then the application developer will also be managing the services attached to it. So the initial V1, alpha one, it will cover basic applications and data types, gateway class for inter-operation between different controllers, a gateway and route. So HTTP, TCP, HTTPS, servers to have secrets. And we actually have a whole bunch of implementers who are going to participate in this alpha and we're looking at both merging style implementers which is sort of multiple gateways that are hosted on a single proxy infrastructure. For example, you run Ingress Engine X that's like a merging style gateway that's running inside your cluster as well as a provisioning style or cloud-based gateway. So you have a gateway that when you create it it actually maps to a cloud load balancer. That is just some of the things that we're up to. So how do you get involved? There's issues.kates.io which will send you to a place to file bugs, clean up ideas and feature requests. And then you can also use this to find issues to help with. So large changes to Kubernetes are done through the enhancement process which is also known as CAPS, Kubernetes Enhancement Proposals. And enhancements are user-visible changes or even infrastructure changes of a sufficient size. So please submit enhancement proposals. The best way to kind of get your CAP moving is to show up at SIG network and discuss what you're proposing. Probably in that discussion it will spill onto the mailing list or maybe a shared document. And then after we have kind of consensus it will become a CAP. So if you want to get involved there's our community page. We have a Zoom meeting which is every other Thursday the Slack is hashtag SIG network and then the mailing list.