 So yeah, this is a protocol detection, a deep dive into how LinkerD achieves zero config. So my name is Kevin Limejuler. I am a software engineer at Boyant. Here are not too active on Twitter, but my handle's up there. You can reach out to me on GitHub or our LinkerD Slack. So I've been at Boyant and working on LinkerD for just a little over three years now. And it's been a very interesting journey. So for something so core to the CNCF, it's fascinating to see how the ecosystem has changed over the years. But there's also just been a lot of challenges. The things that we work on can be hard to solve sometimes, fairly cutting edge, and require a lot of design decisions. So today, I'm kind of here to talk about one of the things that I've really enjoyed designing over my time. So what's the story here? So up here earlier was my boss William Morgan. And he was talking about LinkerD and service meshes should be boring. You shouldn't have to think about them as a user, which I agree with, but also as an engineer that works on a service mesh. I find that once you really look under the hood, there can be a lot of really interesting things in there. And so today, when I'm talking about protocol detection and the things that we do to handle certain protocols, it's something that I have found to be really interesting as I work on. So I'm hoping that everyone here also finds it to be as interesting as me. And I'm always happy to talk about it at the end. Feel free to find me, ask questions. I'll also upload the slides afterwards. So I'll have some diagrams later. Lynn, I think we used a similar diagram tool. So if you want to take pictures of slide of diagrams, I'll have them uploaded later. So I'm going to assume during this talk some level of knowledge about how I think most service meshes work these days. But to just give a quick overview right now, this is specific to LinkerD, but can be pretty generalized. So we have a control plane and we have a data plane. So the data plane is going to be a pod and you're going to have your application container on there. And then with the injection model, you're going to have a proxy for LinkerD. That's the LinkerD proxy for other service meshes that tends to be the Envoy proxy. Every pod then talks to one single control plane. And for LinkerD, the components within that are the identity service where each proxy gets this identity for MTLS, policy, destination, which I'll be talking a little bit more about during this presentation, proxy injector for handling injection. Yeah, so that's kind of the basic architecture. And like I said, I'll get a little bit more into how we use the destination to handle protocol detection. And again, I'll try to stay pretty general where I can. I don't want this isn't trying to sell you specifically on LinkerD. It's more about protocol detection and other service meshes may do something similar. I'm sure everyone has the thing that makes it super interesting. So before I dive into protocol detection, I want to clarify a little bit about what zero config means in the title. So LinkerD is easy to install. It tries not to break existing applications. It strives to use Kubernetes primitives when possible and requires minimal configuration. So I'd like to focus on minimal configuration and how that affects existing applications. So every application is going to have some protocol that it uses, whether that's TCP, UDP, or something built off of one of those, like HTTP, SMTP, or MySQL. In order for LinkerD to not break existing applications, it must be able to be installed. And all these protocols continue working as they were before. And what I mean by that is when you install LinkerD, you don't need to go around and say service x is this protocol and service y is this protocol. It just keeps working. So how do we achieve this minimal configuration that I'm talking about? Protocol detection. So put simply, protocol detection is the ability for the proxy to peak the first few bytes of every client connection and determine which protocol that client is using. So let's first remember that all meshed application traffic inbound and outbound is going to travel through a proxy. And therefore, the proxy is going to see the first few bytes of each connection that it handles. And it uses these bytes to determine which protocol that client is using. So up here, we can see the prefaces for HTTP 1 and 2. Each protocol is going to have some preface that can be used by the proxy. And the proxy can confidently say, this is this certain protocol and make decisions based off of that. So why does protocol detection matter? So most features of a service mesh fall into one of three buckets, observability, reliability, and security. Understanding which protocol is being used is fundamental to each of these categories. So to dive a little bit more into that first, observability is one of the core things is knowing which protocol is being handled so that the proxy can parse that traffic coming through and provide additional details. If that traffic is HTTP, for example, then it can record the number of requests and responses and the response codes. Additionally, once it knows the traffic, request and responses, it can calculate latencies, volumes, error rates. If the proxy can't determine the protocol, we're really just limited to the number of bytes read and received by the proxy. And how that shows up is here's some screenshots from the LinkedIn dashboard, for example. I have a meshed curl pod and a meshed Nginx pod when I send a request from the curl pod to the Nginx pod, we can see the path, the latency, success rate. Again, pretty simple example, but this is the kind of things that we get when we know the protocol that we're handling. If the proxy was unable to determine the protocol, we wouldn't really get all this. So reliability, again, a lot of the features are gonna, such as reliability, rely on knowing the protocol. Continuing the example of assuming that we're working with HTTP, we can move from connection level load balancing to request level load balancing. We can configure retry policies and we can observe failures when they occur. And then finally security, protocol detection is a little less important here, the main thing is that we know if the application is already encrypting the traffic and the proxy can peek that and say, this is already TLS, so I'm not gonna worry about re-TLSing it, even if we're talking to a meshed server. So when does protocol detection fail? Everything that we've talked about so far is usually a protocol that the client is responsible for sending the initial traffic. And these protocols are known as client speaks first protocols and they initiate the connection and they also send the initial traffic. And the proxy sees what bytes the client sends and can determine the protocol before it establishes a connection to the server. It will make decisions like load balancing, if there's several endpoints and things like that. So now server speaks first protocols. These are protocols where the server is listening for a client to initiate the request, but unlike the previous examples like HTTP, it's the server that sends the first bytes on the connection. So it may, for example, be telling the client that the server is sending the first bytes for example be telling the client it's version or other important metadata. Above are some protocols that our server speaks first or where the client kind of keeps a connection warm but doesn't necessarily send initial traffic. And as you can see, these aren't really that uncommon. So this is something that the proxy should be able to handle all. So now that we understand when protocol detection fails, and the protocols that cause these failures, I haven't really explained why these failures actually happen. The server doesn't eventually respond because the proxy never makes a connection in a first place. So here's the scenario where there's a mesh client and a meshed, or in this case just a MySQL pod, and the client application container initiates a connection to the MySQL server which is first intercepted by the proxy. And before the proxy initiates its own connection to the MySQL server, it waits for traffic sent by the client so that it can determine which protocol is being used and make decisions like which load balancer to use if there's multiple MySQL endpoints. As we've covered though, it's the server that is responsible for sending the initial traffic in the MySQL case. So the proxy is now stuck waiting for a connection and it never makes the other connection to the MySQL server. So before I dive into the sluice that we now have, I'd like to first cover how we solved this before. So we could skip certain ports, skip proxying certain ports. How this worked is that there are skip inbound and outbound ports, and this is handled at the IP tables level. So every pod is going to have IP table rules. And if you configure a pod to skip port 3306 outbound, that means that when a application container sends a request to some destination with port 3306, IP tables is the first thing that gets that request, says we are not sending this to the proxy first. And so the application now just establishes a connection directly to that destination. And this means that the proxy never sees that. So this is good because we now are handling service week source protocols fine, but it's not good because this isn't going through the proxy and now none of this application is meshed. So if for like MTLS, you're not going to get that, you're not going to get observability, you're not going to get the reliability features of retries and load balancing and things like that. We could also stop trying to detect the protocol after a certain point. So we can, you know, this is handled with a protocol detection timeout. The proxy attempts to detect the protocol for some certain period of time by default in LinkerD, this is 10 seconds. And if it fails to detect the protocol, then it passes the connection through as TCP. The upside to this is that this traffic is now meshed. You get MTLS and all the observability features, but new connections experience a 10 second timeout, which is not ideal. This is a issue that has come up in, you know, if you've ever popped in our Slack, we help, this is like one of the log lines we look for in that we have an outbound connection, the client address that's going to some server, port 3306, and we are continuing after a certain timeout. And that timeout is the fact that we could not detect the protocol. We usually look for these log lines if users come in and say, hey, I'm experiencing a 10 second connection delay, what's going on here? So this leads us to opaque ports. So we've reviewed the two existing solutions of skip ports and protocol detection timeouts. Both of these were temporary solutions and they both have their downsides, but opaque ports encompass both of these by leveraging the control planes destination service and ALPN if both the client and the server are meshed. So here's where we're going to dive a little bit more into what exactly the destination pod is, or destination service is doing. So we can see a pretty simplified example. The client pods proxy has received an outbound connection request for 10.4201 port 3306. It asks the destination service about additional metadata for this destination. The information it expects back is the destination's TLS identity, a list of endpoints if there are multiple endpoints behind that destination, metric labels, and if this destination is an opaque destination. So the destination service takes this request and because it's on the same cluster, in this case as the destination, it can look directly at that MySQL pod and it can look at the annotations on it. It can, from those annotations, it can see that this has been marked as opaque. It can look at the pod spec of that MySQL server and see that it's injected. And it takes that information, bundles it up in a destination profile, and sends it back to the client proxy. And the client proxy is responsible for using that destination profile to make these protocol detection decisions. So this is the most text slide. So I will try to keep this brief, but basically what we're seeing here is I made two destination requests. I'm just running the destination service locally. And I made it for the same path, 10.42, 014, port 80. The difference between these two destination service requests is that I configured the destination, I think it was an Nginx pod in this case, to be opaque on port 80 for the second request. And so you can see that the responses are these structured destination profiles that I was explaining on the last slide, but the second destination request differs from the first with the red highlighting. We can see that there's now two new fields returned to the client proxy in this case. The first is this opaque protocol true, and the second is this protocol hint that says opaque transport with an inbound port of 41, 43. And these two fields are the crux for how we implemented opaque ports and I'll be explaining how the proxy actually used these fields to solve this problem. So how do we use opaque protocol field? Basically, we're kind of dived inside of the pod now and we have this application container and then the proxy container. And one of the first things that the proxy container looks at is this opaque protocol field. If it's false, then the proxy, when it's making the connection, it's going to wait to detect the protocol. If opaque protocol is set to true, then the proxy knows the client is not gonna be sending any information on this, so I'm gonna pass this through as TCP and make the connection to the server right away. Also keep in mind that this happens very early in the path that the connection takes because after it's just determined if it's gonna use TCP or detect the protocol, it still has to make decisions like if it's gonna encrypt the traffic or not. So before I explain the opaque transport field, I wanna explain real quick why we actually annotate the destination. So the opaque ports annotation, you're supposed to, we put it on the destination. And it's for the reason that on the left here, we have the client pod and if the proxy can't determine the protocol, it'll pass through that traffic because it's just sending this traffic to some destination, doesn't really care if it can't determine the protocol and then if say it waits 10 seconds, it doesn't get anything, it'll just start passing that traffic through, maybe then it gets the client hello for an MTLS connection. It doesn't really care if that gets to the wherever it's going later. But on the destination side of things, the proxy can't time out and then send this to the application because it may, if the proxy receives an MTLS connection and then says, oh, I can't determine the protocol, I'm just gonna start sending this all right into the application and then it gets a client hello for an MTLS connection. It's now feeding the application just a bunch of basically garbage traffic because it's all encrypted. So therefore the destination needs to know if its ports are gonna be opaque or not and the clients can just observe that. So now jumping into how opaque transport is used. So we've established that we should not detect the protocol for a connection using the opaque protocol field but now we need to ensure that the connection is TLS if the server is also meshed. And what I mean by this is that the proxy is making an outbound connection and it's determined that it should not detect the protocol but now we need to determine whether it should TLS the connection and has to be careful about doing this. If the server isn't meshed, then the proxy doesn't want to encrypt that traffic because the server isn't gonna know how to decrypt it. So it has to, the client has to also know the server has another Lincority proxy and that Lincority proxy will be able to decrypt it before it passes it into the server. So this is where opaque transport field comes into play. We've decided that opaque protocol is true so that we're gonna use TCP and now we look at opaque transport field. If there is no opaque transport field in the protocol hint then we'll pass just to the original port number which in this case is port 3306. If we do have an inbound port that we should use in this case it's 4143 and I'll explain why in a sec then we're going to pass our traffic to that port and that port tends to be the proxy on the other end because the proxy is gonna be running on a separate port than the application. And so it's pretty simple to get. Here is some simplified code from the control plane go and we basically the destination service when it's making this destination profile that I talked about before looks at the pod spec, loops through all the containers until it finds the Lincority proxy. It then loops through all of the environment variables in the proxy and it looks for this proxy inbound listen adder environment variable. Once it finds that then it takes that address and it says in the destination profile hey the proxy is listening on this port 4143. So when you establish this TLS connection send your traffic to there and then it will go to the application port 3306. And so that's when ALPN comes into play. So ALPN is application layer protocol negotiation and it's a TLS extension that allows an application to negotiate which protocol should be performed over a secure connection. The protocol that is performed over this secure connection in meshed scenarios is this transport L5D.io version one. And so what this is is it's this additional protocol that we wrap an MTLS connection with and the client and the server which are in this case are both Lincority proxies are responsible for knowing that we're dealing with this transport protocol that's wrapping an MTLS connection. If the client knows that the server speaks this transport L5D.io protocol then it knows that it's talking to port 4143 and the first thing it sends is this little header that says, hey, once we've established our connection in this header is the actual destination port 3306 in this example. So send this traffic to there once you've decrypted it. And this kind of explains that in that the client proxy says, hey, do you speak this? The server proxy says, yep, I'm good to go. Client proxy says, okay, here is this header port 3306. Once you start receiving my traffic and you've decrypted it, take that traffic and send it to this port in your pod. And so that's how we're able to have this traffic encrypted, but it's also how the server proxy knows where to send it on the pod once it's been decrypted. So that's how we use these fields and the destination service to achieve this. So here's kind of like the last major decision tree that we work through. And so sometimes configuration is required. The text looks a little small in here, but up at the top, the first thing that we determine is this protocol wrapped in TLS. If it is, then no configuration is ever gonna be required. If it's not, then we have to look at the opaque ports. If it is an opaque port and we are in the network that we're aware of, no configuration is required. And if the destination is not in this cluster's networks, then we have to enable profile searches for that destination so that we can see if it's opaque or not. You can mark destinations as opaque that aren't necessarily meshed or on that cluster. And if it's not an opaque port by default, then you have to add it if we're working with something in the cluster network. And if it's not annotated as opaque and it's not in the cluster networks, then you do have to fall back to skipping it so that it doesn't go through the proxy in the first place. I talked briefly about, and William was up here earlier talking about policy. It's kind of one of the newer ways that we've introduced recently to mark destinations as opaque. You can use servers and server policies to handle marking destinations as opaque if you don't want to worry about configuring through annotations. So yeah, policy is still pretty new but it will probably be the default way that we move to for marking things as opaque. So yeah, questions I think I'm a little low on time here. So yeah, if anyone has questions that they want to ask right now, feel free. I'm also happy to talk if you find me later about this. Again, I find this just to be very interesting. So if you have any more questions, feel free to find me after and ask. But yeah, seems like no questions right now. So that's it for me. So again, this has all been pretty specific to LinkerD. I work at Boyant and we are hiring. Yeah, thank you very much.