 Hello everyone, thank you for joining our talk on Envoy Vassim filters for supporting Yahoo headers. My name is Mirimai Dhoome and I'm a Principal Software Engineer at Verizon Media in the core infrastructure team. Hi everyone, my name is Michael Chepak and I'm a Software Engineer here at Verizon Media as well. So today at Verizon Media, formally Yahoo, we have over 30 Kubernetes clusters across multiple regions. Each of these has an stereo service mesh deployed on it. We have two parallel ingress layers fronting our Kubernetes clusters. One is powered by Apache Traffic Server serving 2 million peak requests per second and the other by Envoy which serves 230k requests per second. By the beginning of 2021, our goal is to migrate all the traffic from our ATS layer to Envoy. Today our stereo service mesh is running at a scale of 6,700 application pods all bundled with the Envoy Sidecar proxy. Some of the key features of Envoy that we are utilizing today are as follows. We make use of its multiple protocol support such as for HTTP, HTTPS, GRPC, TCP including the SNI pass through mode. We allow our users to take advantage of the HTTP request manipulation options that it provides. In terms of security, a majority of our applications delegate the mutual TLS authentication and authorization to the Envoy Sidecar proxy. And in terms of observability, we make use of the tracing attributes and the Prometheus metrics that are generated by Envoy. So now I'd like to talk a little bit about the request flow that we see through Envoy proxy. So as you can see in this diagram, if a request originates from outside of the Kubernetes cluster, it must pass through our Envoy ingress layer. And the Envoy layer is then responsible for sending the request to the target pod of the appropriate application. In addition, if you have two services, say A and B, which need to communicate with each other and both of them are residing on our Kubernetes cluster, they are able to communicate directly with one another using the Envoy Sidecar so that the request will directly go from a pod of service A to a pod of service B and not have to go through our ingress layer. Another point that I would like to mention here is that the configuration that is provided to all of these Envoy proxies is provided by our Istio service mesh control plane component called IstioD. So like I mentioned previously, Envoy is a rich featured powerful L3, L4 and L7 proxy and we utilize a number of the features that are available today. However, there are some custom features that we would like to add and extend Envoy. Envoy provides a number of extension points, such as through custom HTTP filters, network filters, access loggers, among others. In order to implement these extensions, there are a couple of options. One option is to write a C++ native filter using the extensive API that Envoy exposes. In order to do this, a developer would take the Envoy source code, add their own custom filter to it and then compile it together to form a custom Envoy binary. Of course, if this use case is generic enough, they do have the option of merging it with the upstream Envoy main source code. The second option that we have is to write a Lua based filter. Envoy provides a Lua API so that developers can write their custom Lua script using it. This API is intentionally small so that the Lua scripts are simple and safe to use. For more complex use cases, developers are encouraged to use one of the other approaches. The newest option available to extend Envoy is to write a WebAssembly based filter. I will talk more about this one in the upcoming slides. So what is WebAssembly? The official definition states that it is a safe, portable, low-level code format designed for efficient execution and compact representation. We chose WebAssembly as the mode of extension for Envoy for a couple of reasons. Firstly, the binary that gets created has a small size and also this VASEM modules are loaded very fast which reduces the time required for startup. VASEM also has support for multiple languages. So if you have any dependency on existing libraries, you can continue to write your VASEM filter in the same language and then together you can compile them to form a VASEM module. These VASEM modules run in a memory safe and sandboxed environment where the host environment is responsible for the memory allocation for the VASEM module and also to define what are the functions that are available to it. This binary format can be pretty printed in a textual format so that it is easy to debug and to optimize. Another advantage that we saw is that the VASEM filter that we write does not have to be recompiled with the Envoy source code. So we can continue to consume the Envoy open source versions and binaries and we can keep in sync with any new releases on new security fixes and once we have fetched the latest versions, we can deploy our VASEM filters on top of it. So now that I've spoken a little bit about WebAssembly in general, I would like to talk more about how Envoy in particular implements WebAssembly. So Envoy embeds today a V8 VASEM VM. So if you take a look at this diagram, you can see that when the request streams come in, they are serviced by one of the Envoy worker threads. Each of these worker threads embeds a thread local VASEM VM. So if you write a VASEM filter that is required to perform some per request-based manipulation, that filter will be executed in one of these thread local VASEM VMs. In addition, you may also have a use case where you want to perform certain non-request specific functionality. For example, you may want to aggregate the metrics across multiple requests. For such use cases, you can write a singleton filter. This filter then gets executed in the base VASEM VM, which is embedded as a part of the main Envoy thread. In general, there is a number of API that are available for the VASEM module. You make use of the stats and logs API in case they want to take the metrics and logs that are generated by the worker threads and then expose them to an external sync. VASEM modules can make use of HTTP or GRPC API in order to make external HTTP or GRPC calls. The timer API can be used in case there is some functionality that needs to be performed at regular intervals. There is also provision for a shared data, which is basically a key value pair store and a message queue. So for any kind of operations that require the worker threads to store certain data, which can then later on maybe be consumed by your singleton filter, these two are useful. So the API that I described in the preceding slide is defined by a proxy agnostic ABI standard, which is called proxy VASEM. So by proxy agnostic, I mean that if once you write a VASEM module, it can be executed in a number of different proxies such as Envoy, Apache Traffic Server, HA proxy, as long as these proxies implement the proxy VASEM standard. So proxy VASEM defines a number of event-driven streaming APIs, which govern the interaction between Envoy and the VASEM module that is being executed. These interactions are in the form of functions and callbacks that are wrapped by a context object. So as a developer, if you want to write a VASEM filter, there are two context objects that you need to keep in mind. The first one is the root context. So the root context is created at bootstrap and has a lifetime which is equal to the VM in which it gets created. There are a number of methods that you can override in the root context like onStart or onConfigure, and it can be used to perform certain initial setup operations that may be required for your VASEM filters. Additionally, as the requests come in, there is a new stream context that gets created for every request and similarly a new stream context object. So this stream context object has a lifetime which is equal to the request stream in which it gets created. Again, there are a number of methods that can be overridden here. So for example, if your VASEM filter needs to make certain request header manipulation, it would override the onRequestHeaders function. In case it wants to append certain data to the response body, it would override the onResponseBody function. Similarly, as new requests come in, there will be a new stream context created for each one of them and the stream context object also will be created independently. So this describes a little bit about WebAssembly in general and also some specifics about how Envoy has gone about implementing VASEM. We will now take a closer look at our specific use case of Yahoo headers. So what are Yahoo headers? So Yahoo headers can be defined as a set of custom headers that are used to securely identify a request, its remote source and the path that the request takes through the network. So there are a couple of headers that we were considering. The first one was RID or Request Identifier. This is generated based on the Unix timestamp and a randomly generated number. This is similar to the XRequest ID that is provided by Envoy. The primary purpose for this is that it aids in debugging if you want to trace how the request has flown through your network. An example for RID is shown here on this slide. The next set of headers that we were concerned with were RA, RP and RS which stand for Remote Address, Remote Port and Remote Signature respectively. So RA and RP are used for the address and port of where this request actually originated from. RS in turn is generated by making an MD5 signature of the RA and RP values with a secret and then base 64 encoding it. The primary purpose for RS is to make sure that the RA and RP values do not get tampered with as this request moves through the network. Sample RA, RP and RS headers are shown here on the screen. So in phase one of our filter development, we focused on these set of headers. In phase two, we have a couple of other headers that we wish to implement. The first one is the path header. So the path header is used to track a request as it goes through multiple hops in the network. So it represents an ordered list of hops. So this is similar to the X forwarded for header, but it includes some additional attributes. Some of the per hop attributes that are added are the address and port of where the request was sent from, the address and port of where the request landed and whether that hop was SSL enabled or not. In addition, we also add a signature which is similar to the RS signature that we defined previously. Again, this is used to ensure the integrity of this header. In the example shown here, you can see that there were two hops that took place. The information or the attributes for every hop are separated by semicolons and each hop itself is separated by a comma. In the example shown here, the first hop has SSL enabled as is indicated by the s equal to one flag while the second hop does not have it. The next header that we are concerned with is the via header. So there is a default implementation that Envoy provides. However, it relies on a static value. In our use case, we want to have a dynamic value for the via header where we can specify attributes such as the protocol, the protocol version and the specific proxy through which this request was passed. So if you take a look at the example over here, you can see that there is the protocol and the protocol version. There is the actual ingress pod through which this particular request was sent and then that's followed by a UUID. So this is a unique identifier of the specific proxy that it passed through and this is followed by the proxy and the proxy version. Since we use Istio, it says Envoy Istio and the Istio version that we use. The UUID can be used to detect multi hop cycles. Apart from this via can also be used for general tracing of the request and also to identify the request and response protocols. So this summarizes what Yahoo headers are. And now I would like to pass it to my co-presenter Michael who will talk more about our design goals, challenges and the architecture that we came up with. Thank you. Okay, thank you, Mirin Mai. So now let's talk a little bit about some of the design goals and challenges that we face throughout our implementation. So the primary goal that we had is we wanted to implement support for Yahoo headers while reusing the existing core Yahoo C++ libraries. So we actually have a Yahoo connection plugin for Apache traffic server which does have the RS, essentially header validation and generation flows in place. So the main idea here is we wanted to reuse that functionality so we did not need to reimplement it and also to stay up to date in case there are any changes there and also if there are any security fixes as well. Okay, so now let's talk a little bit about some of the challenges that we ran into. So the existing RID generation depends on read access to dev random to generate a unique seed to actually be able to generate unique RID headers for user requests. So the problem inside of our WebAssembly filter is that we do not have access to follow reads here. Okay, another challenge that we ran into is the RS secret is stored in the Yahoo key management service. So here we ran into a couple of problems as well. So the first one being is the existing client for the key management service relies on the curl library which requires access to the underlying network socket which we do not have inside of our WebAssembly filter. Also, we need to make multiple calls to this key management service to get the RS secret. So every time we're waiting for a response, we would essentially be blocking to wait for that response, which is essentially a busy way. This blocks the envoy threads actually from executing and also from coming up. Another issue is that we cannot take the RS secret and write a file into the envoy container with it because once again, as part of our WebAssembly filter, we do not have access to follow reads. Okay, so let's take a look at the architecture that we came up with to get around some of these challenges. So here we have an ingress node, which is running a demon set pod with the ingress utils server, which has a local host server inside of it. When this is coming up, it's going to make a request to the key management server, get the RS secret and put it inside of its own memory. So because we have this local host server, basically the VM filters themselves do not need to make your request to the key management server as well. So here we have the ingress pod with the envoy container. When it is coming up, it has a singleton VM, which is running as part of the main envoy thread, as previously described. And this will go ahead and call this local host service, get the RS secret and then write it to the shared data, which will then be available to be read by other VMs inside of the envoy container. Okay, so now when we have the envoy worker threads which are coming up, initially what will happen is they will use their start time as a seed for the RID so that we can have unique RIDs for the user request. And then they will go ahead and read the RS secret from the shared data and load it into its own memory. Okay, so now let's go into the secret fetch flow itself. So when the envoy main thread is being initialized, it will call into our WebAssembly module, which has a root context defined. Here we have two functions in place. So the first one is the on start, which will go ahead and make an HTTP call to the ingress utility server for the secret. And then we have the on HTTP call response, which will go ahead and get invoked when the ingress utility server response actually comes back. And here is where we actually write the secret to the shared data. Okay, now let's talk about the user request flow itself. So here we have the envoy worker thread. On the right we have our WebAssembly filter, which has a root context defined with an on start function. So here inside of the on start, we read the secret from the shared data, populate an in memory variable with the RS secret that we can use then to generate and validate the RS header, which is coming in. Now when the user requests are coming, the stream filter chain will call into our filter where a context object will be created. So here inside of the constructor, if the in memory variable with the RS secret is not set, we will actually go ahead and also read the secret from the shared data. This is used as a backup just in case the ingress utility server took a little bit longer to respond. And for the main functionality, we have the on request headers implemented. So here is where we generate the RID and also validate and generate the R8, RP and RS. And this will occur for every user request which is coming in. Okay, so now let me show you the envoy configuration. So we are running on boy one dot 13 dot X with is still one dot five dot X inside of the bootstrap configuration. We have the ingress utility server cluster defined and also the singleton filter in the filter chain configuration. We have the Yahoo headers filter defined. So here this is the bootstrap configuration. As you can see here, we have a static cluster resource defined for the ingress utility server pointing to local host. This will be used by the actual singleton VM to make a request into to fetch the RS secret. Down here we have our singleton filter definition with the singleton Boolean set to true. Now inside of the filter chain, we basically have our Yahoo header configuration. Now let's go into demo to see how this functionality looks like. So when a user request is actually coming in, we will check if these three headers are actually defined. If they are, we will validate the RS signature. If the signature is valid, we will go ahead and continue the HTTP request. If it is invalid, we will generate an RARP and RS from the incoming client connection. We will get its address and port and generate an RS and then continue the HTTP request. Now if the headers are not set, what we will do is we will generate new ones and then continue the HTTP request. Okay, so now for the RID flow, it's a little bit simpler. So here if the RID is set, we will just continue the HTTP request. If it is not set, then we will generate a new one. So now let me show you a demo of how this functionality looks like. So I have an echo server, which is behind an Istio ingress pod. So here I am not passing in any headers. So what you will see is that basically a new set of headers will be defined here created. So the RID, RS, RP and RA are brand new, which were generated from my incoming client information. Now if I go ahead and I pass in a set of headers inside of my curl, since these are valid, they will be persisted throughout the connection life cycle. So here as you can see, these headers persisted because the validation was successful. Now if I slightly modified the signature making it invalid, what you will see is the validation will fail and then we will generate new ones from the incoming client connection. So as you can see here, basically these three are new, the RS, RP and RA from my incoming client connection. Okay, so let's talk about some other functionality that we have. So the proxy with some stats API allows you to have custom metrics. So here we define essentially metrics for our filter to see how it's behaving and also to increment in case there are any failure scenarios so that we can alert upon them as well. So as you can see here, if you hit the Envoy Slash Stats API, you can access the metrics with your setting. Okay, now let's talk a little bit about some of the performance metrics. So here we wanted to run a load test to see how our filter behaved under load. So we have one Istio Ingress Pod running with 24 Envoy worker threads, meaning 24 CPU with our filter. We run 4K RPS for 10 minutes and we have Vegetal Load Test Ponds generating the client load with an Echo Server upstream. So as you can see here, we have multiple scenarios which we are testing, which is similar to the demo. Essentially the generation flow if there are no headers coming in, the validation flow if headers are passed in and the validation and generation flow where the incoming headers were invalid and then regenerating new ones. So here I pointed out the worst case scenario where essentially when the incoming headers are invalid and regenerate new ones. So the P99 here increased by half a millisecond and the CPU increased by 3%. And for the memory and RSS it increased by 0.66% here. So this is a little bit to be expected because you are copying the request into and out of the VM. Okay, so now let's talk a little bit about some of our key learnings. So the first one is we have that the WebAssembly eases the developer experience. It allows you to implement the filter in your programming language of choice that you're comfortable with and then you can compile down into the WebAssembly filter. Also, the Envoy deployment itself is separate from the WebAssembly filter. So this means that the teams can work independently to implement their logic. Okay, so now let's talk a little bit about the Wasm VM Sandbox. So first of all, it ensures reliability. So if there are any issues with your filter and it's crashing, it will not bring down the Envoy process itself. Also, there's the security aspect here as well. So essentially the VM filter cannot access anything on the Envoy host as it has not have any access to it. Only what is defined through the proxy Wasm APIs. Also, there are a couple of restrictions in place. So like I mentioned previously, you do not have access to follow reads. You cannot do busy ways. You don't have access to the network socket and also the threading and exception support is not there, but those two will be there sometime in the future. And also you do get a little bit of a performance overhead, which I mentioned previously. Also, one thing that I want to mention is that compiling with external libraries is also not straightforward. So what we had to do is we needed to compile down with the core YakuC++ libraries and also open SSL for MD5. So due to this, we had to essentially compile them down into what's known as LLVM big code, which then can be used to import and compile down with your WebAssembly module. So that was a little bit of a learning curve and wasn't straightforward. So that is something you should also be aware of. And finally, the Envoy was sent is still an alpha stage. So it is in the process of being upstream to Envoy, so not all of the functionality is in place. Also proxy was sent APIs are still a work in progress. So there are some things that they do not have yet. For example, SDS support. So one idea is we could have had our RS secret be inside of the SDS and then we wouldn't have needed the ingress util server. But unfortunately, since this is not there yet, we didn't do that, but it will be coming sometime in the future. Okay, so we had a couple of acknowledgments. So we wanted to thank the Envoy was sent community and also specifically John Pleviac from there, the Istio team and also from the core infrastructure team, Suresh, Baskram and Catherine as well. Okay, so thank you so much for coming to our talk and listening to our experience for implementing Yaku headers using the WebAssembly filter. If you have any questions, then please let us know and we'd be happy to answer them. Thank you so much. Hello, can you hear us? Hello there. Can you hear us? I believe so. So we can take the first question, which was regarding the performance. So we did see some amount of performance improvement in general in terms of the throughput per box, but it does require a little bit of tuning because we've been using Apache traffic server for a while now at Yahoo. So a lot of those have already been tuned. So with Envoy that has been a bit of a learning curve. We're learning how to tune some of the aspects, like especially if there are large payloads, we've had to tune some of the settings on Envoy. So there has been some performance improvements, but there's also been a learning curve for us to get the maximum performance out of Envoy. Yeah. And then yeah, so in terms of the SSL libraries that we're using to actually generate the secret. So we're using the open SSL MD5 library to generate the RS secret itself, basically with the base 64 encoding. And in terms of generating the secret that is used for the MD5, that's actually part of the actual key management service. So I don't think I can share too many details about that. And then another thing that I wanted just to clarify because I know in one of the last slides, I mentioned that basically the WebAssembly functionality is not fully merged yet into Envoy. So actually as of a few days ago, it did get merged into the actual Envoy upstream code base. So the Envoy was some repo was basically archived and then everything was upstream to Envoy so that functionality should be there now. So just wanted to clarify that as well. Okay, let's see. Yeah. I think the last one was regarding the language. So we actually chose C++ as the language because we had a lot of existing Yahoo libraries that we wanted to compile with. So that's the reason why we chose that. Yeah. And yeah, I mean, in terms of compiling with OpenSSL, yeah, it was definitely a little bit painful. Basically, we had to compile the OpenSSL down into LLVM big code first that we could then use to actually compile down with our WebAssembly filter. So that was definitely of a learning curve for us basically figuring out how to do that compilation first before we could use it. Yeah. And also to clarify, we didn't have to compile it with Envoy itself. We only had to compile it with like our filter. So that definitely simplified things. Yeah. That's like the biggest advantage that we see with VASM is that we don't have to recompile with Envoy source code itself. We can just, you know, build our filter independent of it. Exactly. Yeah. Definitely a large benefit. Cool. Yeah. Please let us know if you have any other questions in the chat. Yeah. And then also if you have any questions, then feel free to reach out to us as well. So basically our handles are Mduma and Mchev back on the Envoy Slack channel and also on GitHub. So feel free to reach out to us as well with any questions you have. Thank you. Cool. Thank you. The next session is starting. Thank you so much. Thank you. See you. Bye. Bye.