 Hi everyone, thank you for joining me today, and I'm excited to talk about TLS tracing encrypted microservices with eBPF So before we get started, I wanted to give a brief background about myself My name is Dom. I'm an SRE at Twitter working in observability across metrics and logging Prior to Twitter. I was a production engineer at Yelp where our team was responsible for keeping the website and mobile apps working and that's where I got a deep appreciation for observability tooling Let's go over our agenda for today We'll start off with a background and motivation for why Twitter was interested in this work From there, we'll do an overview of protocol tracing techniques Then we'll do a deep dive into the Twitter TLS tracing work we implemented We'll see a live demo that and Then we will generalize the criteria for how to support new TLS use cases Moving on to the background. So at Twitter scale, we rely heavily on right time aggregated metrics This is great for monitoring service health But it really limits your ability to slice and dice your data And so when you're investigating outliers or doing other real-time debugging, it becomes a real challenge Another thing is that our collection and ingestion for our Observability data has largely remained the same We saw that eBPF had promise with auto instrumentation and we saw the Hubble and Pixie projects doing exciting things And so we wanted to explore this further and see if the protocol tracing could help aid this real-time debugging use case Another important detail is that services at Twitter use MTLS by default This is fantastic for security, but renders many observability tools useless So whatever eBPF instrumentation we would use Supporting encrypted connections is a necessity. Otherwise it wouldn't provide any insight and Lastly, the tracing needs to be extendable for the protocols used at Twitter The majority of our microservices are built on top of this project that we open sourced called finagle Which is an RPC framework for communicating between services and While it supports protocols like my sequel Postgres thrift MUX is the most widely used protocol Which that's a patchy thrift over a MUX session layer protocol With that background out of the way, let's dive into how protocol tracing can be done We'll first discuss how it works for the plain text use case and then dig into Encrypted use cases after that So for tracing plain text traffic, we can attach a BPF program to the read and write syscalls through a K probe This curl command that you see here is dumping out the networking syscalls when we are curling google.com and the arguments that we see to the syscall Would be equivalent to the data that we would be able to access through the K probe So you can see in the send to syscall Five is the file descriptor and the second argument is the raw protocol data So from here what a tool like pixie does is it sets these probes on those syscalls It pulls out the socket file descriptor and the raw protocol data It sends it to the PEM, which is their user space daemon for doing more processing and From there it stitches together the request and response pairs and adds more metadata so that you get rich information about your traffic Let's move on to the encrypted case So we are unable to hook into things at the same layer for encrypted traffic because by the time it gets the syscall It's already encrypted So we need to access it Just before it's decrypted or after it has been decrypted and where this occurs is in the TLS library in This example here we can see the SSL right and SSL read functions Exist within the open SSL library So this is a common point of instrumentation It's worth mentioning that while open SSL is popular the there are many other TLS libraries And so for instance Golang has its own crypto implementation So while it it has a similar layer These SSL right and SSL read functions aren't necessarily It don't necessarily exist in that case It's worth mentioning that the popular BCC tools project has a tool that implements tracing in this way called SSL sniff So it sets you probes on these two functions and it's able to dump out the unencrypted traffic Pixie also takes a similar approach and so it Puts the probes on those two functions and from that point forward It sends the data to the user space process and the request and Request and response pairing occurs just as it did in the plain text case However, there's some additional challenges here in this diagram. You see that there are three requests going on Service B is making a HTTP call to service a while service C is making two HTTP calls to service a We need to be able to understand What data is associated with which connection and so while we can access the protocol data From those functions. That's just part of the story We need to have some type of connection identity which as I mentioned in the plain text use case is done through the socket file descriptor Unfortunately with the SSL right and SSL read functions The socket file descriptor isn't passed in as its own argument And so we need to work out how to determine that So you can see the prototype of the SSL right function The second argument is the plain text data that we can Use just as in the plain text case The first argument of interest is the SSL struct and so what you will see from the other Struct definitions here is that the SSL struct has an embedded RBIO member Which is of type BIO which is open SSL's basic input output abstraction Within that struct there is a num member and that is where the socket file descriptor is actually stored So in order to get this connection identity working We need our BPF program to walk this structure and pull out that num field So let's see how that works on the right hand side. You'll see a diagram of how this exists in memory The function will be given that SSL pointer which will go to the beginning of the SSL struct From there we need to determine two offsets We need the offset between the beginning of the SSL struct and the RBIO member We can then follow that pointer and then we need the additional offset of The BIO struct to the num member which contains the socket file descriptor one thing to mention here and this was Discussed earlier in the GRPC talk is that these offsets aren't guaranteed to be the same across open SSL versions And so what we assume is that for a given open SSL version those offsets will be stable The way pixies user space process determines this is through DL opening the open SSL library and calling the open SSL version num function before it attaches the Uprobe so that those so that the Offset calculation can be done when the functions triggered Now that we've understood the background and some tactics for doing this protocol tracing. Let's move on to Twitter's use case So we built out RTLS tracing on top of the pixie project The reason why we were interested in pixie is because it provides granular observability data And it's at the edge model scales for real-time debugging. I mentioned I work on the observability team and many of the tools that we provide are for service owners to understand their systems and The data that pixie provides is very applicable to those customers and What they really want to do is understand service-to-service interactions, which is data that protocol tracing provides and Finally Twitter's core instrumentation focuses on internal use cases What this means is that open source integration typically lags behind our internally developed projects So while we were hoping that pixie would give us better instrumentation for our finagle and our You know most widely used projects We were hoping it would raise the bar for the open source use cases that we didn't have enough time to fully build out integrations with All right now that we've seen the background Here's the familiar TLS tracing diagram so finagle uses a project called netty, which is a Java networking library and That library underneath the hood uses a project called netty TC native which provides its own shared library That fulfills the same role as the open SSL shared library so in the context of this we needed to trace live netty TC native and It's unlike open SSL Typically, it's statically linked with boring SSL Another note about netty is that it's pretty widely used across many Java projects So when we were embarking on this we were hoping that if we could solve this TLS tracing problem This wouldn't only solve it for Twitter's use case, but it would implement it for Cassandra for elastic search Java G RPC There's a large variety of use cases that all use this library And so I felt that there was a lot of potential here Let's talk about the challenges that we had to Overcome in order to get this tracing working So the first is that Pixies original open SSL version detection did not work for finagle Live netty TC native It does contain the open SSL version num symbol, but it's local which means it exists in the sim tab section of the ELF file rather than the dynamic symbol table and So Pixies call to DL sim would fail to find the function We can see that here in these two Obdump Commands, so the first one shows all of the dynamic symbols and greps for that symbol and we see it's not present Whereas the second command shows all of the local and dynamic symbols and we see that it does show up So this was one area. We needed to have a solution for The second is that The socket file descriptor was not populated in the SSL right and SSL read SSL struct So that was another area. We knew we were gonna have to solve this connection identity problem and And finally boring SSL was used rather than open SSL The most obvious difference here is that one of them is statically linked and the other is typically dynamically linked And we weren't really sure exactly what needed to be done here, but we Were pretty confident that there would be something to work out Starting from the first one Let's talk about how we determined the finagle TLS version So as I mentioned that open SSL version num function is in the ELF file So one way to solve this is that we can actually calculate its raw address and call the function with a raw function pointer The code you see on this slide is an abridged version of the code that exists within pixie that does this Called the raw function pointer manager So you can see the kind of high-level ideas is that we have an ELF parser We also DL open the library so it's accessible within the user space daemons virtual memory We then use the ELF parser to determine the symbol offset And we add that to where the library is loaded in virtual memory and that gives us the function pointer that we need So from there we were able to determine the TLS version and that problem was out of the way next up is addressing the missing socket file descriptor and We solved this by upstream upstreaming changes to netty and netty TC native to populate this on connection creation This pull request here is the one to netty TC native Which is the portion of netty that does the actual TLS and native code If you look at the top part of this Pull request you see that it's handling a bi oc set FD case and What it's doing there is it's making sure that the num field on that bi O struct is correctly populated The part of the pull request to the bottom is the change needed to Export this functionality to be accessible by the Java runtime So what it's doing is it's defining a new method called bi O set FD So that Java can call this in order to take advantage of this functionality The second piece to this was making this was actually using this in the netty project netty has a concept called a channel which is a one-to-one mapping between a connection and It also provides some hooks into connection life cycles And so we there is a thing called channel active which is called when a connection is created And so all we needed to do was call the function there and pass the channel in So that it was passed into the native code Problem two is then solves and Then the final piece the differences between boring SSL and open SSL I Mentioned that the boring SSL is statically linked And so what this means is you actually have to set you probes on the application binary rather than the shared library So if we take an example like Envoy you would actually put the BPF probe on the Envoy binary itself Rather than binary or Envoy's TLS shared library Finagle's case is a little different because Java is not native code It has a runtime so it is still put on the shared library But the shared library doesn't have dynamic linking Second thing is boring SSL is source compatible with open SSL But that doesn't guarantee that the offsets that we use to find the socket file descriptor are the same and What we found out after investigating this is that they are actually different and that boring SSL is written in C plus plus And so the layout in memory is very different from the open SSL C implementation And so the way we solve that is we have different version offsets For boring SSL and different ones for open SSL for each version that's supported All right with that I realized I've called this a live demo before but it is pre-recorded. Let's see it in action In the pixie docs there is this pixie CLI you can use To deploy to the cluster so I've went ahead and done a PX deploy In order to visualize this finagle traffic We'll be setting up two applications one of them will be generating plain text traffic verify that the network traffic Has plain text data We will then set up an encrypted example and use TCP up there as well to verify that the traffic is truly encrypted and Then we will visualize that in the pixie UI So pixie comes with a number of demos out of the box one of them being the PX finagle demo which generates thrift watch traffic At this time the finagle demo does not work with generating encrypted traffic So we will use that for the plain text case And I've went ahead and demo deployed it We can use the cube CTL CLI To verify the data running So we could see that the demo is already set up The encrypted case I've ran some containers on the cluster manually since as I mentioned the demo does not support it yet and With that let's dig into this So in this terminal window on the left-hand side I have the TCP dump output from the plain text case and the logs from the finagle demo We can see that the client is constantly generating this hello ding traffic Which is viewable in the TCP dump output because it's plain text on the right-hand side we have the encrypted demo and It is constantly receiving a string string response from the server And if we look at its TCP dump output, we can see that the string string text Does not show up which validates that this case is using end-to-end encrypted traffic With that let's take a look at the pixie UI and see how things look So the pixie Pixie comes with a number of scripts out of the box which allow you to visualize your cluster's data. There is a Upstreamed MUX data script that supports viewing this MUX protocol traffic I've went ahead and made some slight changes so that we can easily see the Encrypted case versus the plain text case, but otherwise it's the same as the upstream script So what we can see here is the MUX protocol is a session layer protocol So in addition to sending requests and responses with the T dispatch It also has a liveliness probe so we can see that trace here as well And then there is also this T init and RERR which is part of the connection setup So looking at these individual entries, we can see that there are two different sources One of them is PX finagle, which is the name of the the full name of the pod that the demo is running Pixie is able to understand Kubernetes metadata and so that case is annotated This local host case is actually our encrypted demo So we can see that it doesn't have that rich Kubernetes data to attach to it since I'm running this just on the cluster itself But we can see that both of them are generating traffic constantly So we can see that the plain text and encrypted cases are both working out of the box with Pixie All right Hopefully you get a good view of how all of that works end-to-end back to my presentation Oops, I knew this was gonna give me trouble. All right, so in conclusion, I wanted to Give an overview of the different dimensions of things you need to be aware of in order to support a TLS library So to begin with we'll be describing the differences between Nginx and Node.js and So for the SSL library, they both use open SSL They are dynamically linked The SSL write and SSL read symbols are Accessible and in the DynSim which means that we can set a Uprobe with the symbol For the version detection so that we know which offsets to apply We can use the we can use DLSim to call that open SSL version num function In Nginx case the socket file descriptor is populated which results in Pixie supporting that use case for Node.js the socket file descriptor is not populated and Pixie has additional Uprobes to capture that data which results in Node.js being supported Moving on to Netty. So for finagle, we know the library is boring SSL The linking is static We are the SSL write and SSL read symbols. We're able to attach it with the symbol itself In for the version detection, we are using the raw function pointers and The socket file descriptor is populated if you're using a TC native version greater than or equal to 2.0.5.3 on the other use cases of Netty All of the other details are the same We believe that it should be supported But it's yet to be tested and as I mentioned before this is something that I think would be really powerful to this one use case to support many popular projects and Finally Thinking about Envoy Envoy is another use case that uses boring SSL and one of the ones that the Pixie team was actively working on Its linking is obviously static The SSL write and SSL read symbols aren't present But we can do that raw Address calculation. So you have to set your Uprobe with the raw address The version detection since the last time I checked in on this, I believe it was to be determined and Envoy actually has APIs that hide the socket file descriptor So that's not used and so as indicated here. It's a work in progress and Just to note this is for Istio Istio also uses Envoy internally Looking at vanilla Envoy though The SSL write and SSL read symbols are actually stripped from the binary entirely Which means that we have no way of attaching the Uprobe which ultimately mean Means that Unfortunately Envoy can't be supported through this type of tracing so in recap To talk about the current status and the future work Finagle TLS tracing is one minor bug fix from being functional The rest of the functionality that we saw is already in Pixi's latest released and upstreamed The building blocks it introduced to support boring SSL will be shared by future integrations So I mentioned that Istio use case. It's piggybacking off of those changes and Then looking to the future I mentioned that Netty TLS tracing has the potential to support many other projects. This could include Java gRPC Elastic search Cassandra and I think it would be extremely powerful to have fully end-to-end tracing all the way through to your data store And so I'm excited to eventually Pursue what that would look like That's all I have. So thank you all for your time and I'm happy to take any questions you have Well, we've got a question over here. Maybe while we're asking questions, Mauricio could Presentation ready Dueling mics So through any of the work With any of the library or code maintainers. Was there ever any discussion about them modifying? Structures or emitting data or doing something to make it simpler for eBPF, you know to observe Sort of a meta question so as I dug into this problem, I was Once we Identified that we thought pixie was at a good first step into doing this so that we didn't have to build everything out ourselves We were working a lot with what already existed there And so as I mentioned in kind of the protocol tracing overview Pixies method is you know plucking that socket file descriptor out. So we were kind of just continuing down to see What all would needed to be done to? Unify that because we wanted the solution to work Ideally in the same manner to minimize like one-off use cases. I think now that we've like explored it fully like There could potentially be other things, you know like thinking about ways to get that data with other methods And I was just thinking because there's some Twitter specific not necessarily specific but lead projects in here But thank you Other question another one fantastic and then one here, okay Hey, cool work My question is have you experimented with the kernel mtls that would also allow you to see The encrypted traffic or have you any comments on the adoption of that? I Know some of that's possible, but I don't I haven't dug into it myself That's definitely something I think that's worth exploring and understanding more Hey, thanks for the talk. This is a little lower than I usually know, but I feel like I learned a lot so it seems like you're kind of at the Mercy I guess of like the the server the the networking Framework that you use right to be able to expose the things that you need to do the tracing So I guess is there almost like a wider call to arms for different frameworks to open up this Access up so that we can have this kind of tracing throughout That's exactly what Ori was talking about in his talk earlier today about, you know, this is using you probes and Fortunately for this use case SSL right and SSL read are more of a defined contract because boring SSL provides source-compatible Support for it, so it's less unstable in that sense, but I do think that doubling down on like Implementing USDT's in popular libraries is the right way to go to get stable interfaces for Getting the data out of this so find Ori is the answer What was that? Hi for Java based applications with the cheat compiler Sometimes the code that is generated is native code For example a you call a function one time or and then many times so it generates a native code So when you do the you probes or even on the SSL or some other like and Java as its own SSL version So how do you cope with the cheat compiler? So I mentioned before I don't have my slides in the Nettie has kind of a sub project inside of it. That's the net ETC native part That's all native code. So the jit does not have any control over that. I Wouldn't consider myself a Java runtime expert, but I believe it's it's separate in that sense I'm asking when you have the jit compiler working and it's not out of jitter So how do you cope with it or is there is no solution you have to go higher levels? For other type of instrumentation. Yeah So I believe there is a project to add you can do USDT's in Java code. I When I initially worked on this I thought we were gonna have to explore that but then I was relieved that there was native code that I could Attach to so that's something I would like to explore more, but I barely scratched the surface We still have more questions. This is great Do you have do you use this instrumentation and Only for debugging when you have a problem or you can have it all the time running We are not using it today. We're currently understanding Where our first investment will be we aren't running a Very small number of our use cases are on Kubernetes. So because pixie is a Kubernetes Observability tool. We're understanding how we can run it outside of Kubernetes and extract the data from it And we are anticipating that we want it on all the time My question was do you have any do you foresee or have any performance impact when running this instrumentation? So there is there should be a slight performance impact because the you probe will cause a context switch when that function is called From the issues that I've seen at Twitter of us not having good capabilities for Slicing and dicing or investigating real-time issues. I think that that's gonna be worth it, but it's not been measured yet