 Hey everyone, my name is Matt Klein, Envoy Maintainer. I've got Alyssa Wilk here, also Envoy Maintainer. We don't have any slides or anything like that. I can give a little short project update if people are interested, but mostly just gonna do an open Q&A. So if there's any questions, we can cover anything that people would like to cover and we'll get the mic passed around. So I guess it's honestly, it's really up to all of you. Do you want to just jump straight to open Q&A or would you like to hear a short update? Small room, so you can just call out. Okay, let's see, story. Well, I mean, it's been a long time since I've been at KubeCon, so that's the first part of the story. It's been three years. So it's fun to see all the changes and I think Envoy has matured a lot in the last few years. The project is very widely used at this point. And as I mentioned at EnvoyCon, I think has become kind of boring, which is a good thing. So I think the project now is, there's less new shiny feature development and we're spending a lot more time just on the nuts and bolts in terms of security. Fuzzing, software supply chain, really just making sure that the project is stable for all the people who are using it. Some of the major things over the last year or so are investments in Qwik and H2B3, that's now generally available. So please check that out. I already mentioned security and software supply chain. There's other talks and people at this conference talking about Envoy Gateway. So that's our new Ingress project. That's not about Envoy proxy specifically, but that's a Ingress controller project built on top of Envoy that we hope will make it simpler for people to onboard onto using Envoy for Kubernetes Ingress cases. A lot of work has been happening with Envoy mobile. So Lyft is now using that as it's mobile library. Google is heavily invested. We have other folks from different places. Also who are starting to use it. So that's been fantastic. And I'm sure I'm forgetting things, but it's a perpetual motion of PRs and features across all the different protocols that Envoy supports, whether that be Redis or Dubbo or various things. And I guess the final thing that I would mention is we continue to put a lot of work into the build system. So CI release, it's been a long standing request that we ship things other than Docker images. So I think finally in the next couple of months we're gonna have packages for different operating systems. So RPMs and Debs and all of those things. So hopefully that will make folks happy. And then a lot of work continues on documentation and examples Envoy for better or worse is a very complex piece of software. And it's hard to make every constituency happy in terms of the documentation and the examples and all of those things. So trying to make that better where we can to help people out. I think that about covers the general state of things. And over the next period of time I think work will just continue on all of these axes. So I don't think there's any major new shiny thing happening, people are still working on WebAssembly. People, we see a continuous number of PRs coming in on various security filters, whether that be OAuth or Jot or external authorization or external processing or all of those things. So those are also things that are popular. I don't know anything, Alyssa, that you would like to add? No, okay. So that's really all I had. And we're happy to answer any questions that anyone would have. So please ask us anything, happy to explain concepts or if there's any problems that people want to talk through, happy to do that. So happy to pass the microphone around if there's anyone that wants to start out. Thanks. Is somebody who's new to the project in the space and just trying to start to get engaged a little bit? Is there, I know you went over kind of what's going on, but do you guys have a roadmap or areas that you'd like to see investment in? How can we engage with the community and bring forward hopefully resources to help? Sure, thank you, great question. We get that question pretty often is do we have a roadmap? And the answer is no. And let me explain why. I think Envoy is a bit of an oddity in this ecosystem and the fact that we're not backed by a vendor. It's like we really are a truly community led project. And it's very widely used of course among lots of vendors and cloud providers and lots of large internet companies and all of those things. But the features and the bug fixes and all of the things that we have coming in, they're really based on the community and what they want to work on. So like we don't have a product manager. And I can give you a sense at any given time just by watching the project in terms of what things people are working on. But there's no roadmap. Like I can't tell you what we're gonna do over the next couple of years. I would say it's much more like Linux in that respect. A lot of work happens. None of it is, I would say particularly, like no individual thing is particularly noteworthy but in aggregate, it keeps the ship running. So to answer your question in terms of how people can get engaged, what I would say is I think there's a misperception that in order to help with Envoy, you have to be like a networking guru or like a C++ guru or something along those lines. We actually need the most work and the most help with doing release management, back ports, helping with our CI system. A lot of that is like Bazel and Python and Bash scripts and all of those things like typical DevOps type stuff. We always want to improve our documentation as I was mentioning previously. I've always found it funny with Envoy that on any given week or day, I will hear from people that Envoy has the best documentation of any open source project that I've ever seen and then that Envoy is the most awful piece of software that is the most poorly documented thing that I've ever seen. So I'm just saying it's like you can't make everyone happy, right? And we do the best that we can and I think we care about documentation but what I always encourage people is as you're ramping up on things, if you were confused by the documentation and then you had to figure something out, please fix the documentation. Like it's a pretty low friction way to get involved in open source and I would say from the maintainers, we appreciate that very much, right? And that can range from small fixes, typos to clarifications to more cross-linking to now we have a maintainer named Ryan who's done a, I mean, he doesn't do C++ or like networking, like literally all he does is like work on examples and documentation and build tooling and he's done like amazing work. And now we have, I mean, I think we have 30 sandboxes if Docker compose that illustrate different features and all those kinds of things. So, I mean like making a new sandbox that demonstrates some feature that you struggle to make work is super useful. So those are the ways that I would say to get involved and they typically don't require that much effort. Of course, if there's features that are missing and you or your team want to help add them we'll work with you to help design them and give you tips and all of those things. I think the community is very welcoming whether it be on GitHub or Slack or anything. So, hopefully that answers your question but I would say there's plenty of ways to get involved that aren't super time consuming. Did you want to add anything, Liza? Yeah, I'd say that the one other thing is if you don't have a particular agenda something that your company needs to be done but you want to get involved in the project if you look at the open issues there's a lot tagged with help wanted and some that are specifically tagged with beginner that are things we thought would be kind of easy hello world starter projects but actually like if we tagged it it's worth doing, like we would love to help there. Yeah, and I mean even other things like we have fuzzers that run continuously on the open source fuzzing engine and there's a lot of false positives on there because writing fuzzers is hard that's a whole separate conversation and honestly we need people to come in and just fix bugs. I mean it's like there's tons of ways to contribute so I would say peek through some of the issues and always I try to make myself very accessible so you can always email me or contact me on Slack or whatever, I'm actually happy to meet with you and your organization to talk about how you can contribute so it's any number of ways. You mentioned about mobile, can you say something and what are the challenges, what features, what's not the roadmap but where do you expect the outcome to be? So Envoy Mobile was a project start a few years ago and I mean essentially at a very high level it's we have all of this code that runs on mobile clients, lots of apps that run on iOS and Android and when you think about it, a lot of us at this conference we are server engineers and historically I think there's a lot of silo between a quote server engineer and a quote client engineer that I personally think is counterproductive because we're trying to develop a product and at the end of the day the product that people use is the thing on the phone or on the webpage and a lot of times that doesn't work for a variety of reasons so you can think that the server is returning a good success rate but it's broken on the client for any number of reasons so most of the problems that you face on the client from a networking perspective are the same on server you're doing low balancing and observability and a bunch of other things so for a long time Google has had a networking library called Chronet so it's part of the Chrome networking stack and that's something that they've used and other companies have used for a long time and a few years ago with so much investment in Envoy we thought that we could run Envoy on the mobile client on iOS and Android and get many of the same benefits of Envoy in terms of observability and all of the other features and so that's work that's been ongoing for quite some time and part of that again is a feature set it's also the fact that there's so much effort in people hours that are going into working on Envoy like hundreds and thousands of people hours and years that have gone into working on Envoy at this point that having a single code base that is open source there's a lot of benefits in terms of understanding what that actually looks like but I'm gonna let Alyssa answer more because she actually works on this full time now Yeah so again as Matt said kind of getting the modern protocols is key for a networking library so if you have any kind of late sensitivity or quality of experience tied to revenue doing that bundling gets you kind of the latest and greatest of HTTP2 and HTTP3 which in our experience just the quality of experience improvement is huge the other thing we're really excited about is being able to have Envoy end-to-end as Matt said these are probability but again my background was launching these modern internet protocols and doing HTTP2 end-to-end with Chrome and our Arbus Book Proxy and then HTTP3 end-to-end and being able to rapid iterate on be it your new compression algorithm or new business logic or have wasm and iterate on wasm and then kind of cut your build at a certain point I think is really really powerful and so again we're focused on mobile right now but also again Envoy's a client stack and go be on mobile so if you have any sort of remote devices internet of things you can kind of fetch wasm via XDS and install your local application and do that all with the Envoy client stack so we're mobile focused today but I think again having the observability the plugability and the modern networking stack of Envoy end-to-end is really powerful so for roadmap again we do have a roadmap for Envoy mobile because we've been pretty tightly coordinated with Lyft on what we've been doing so and there is a talk that'll probably be online that JP from Lyft gave kind of talking through what they've done but they're fully deployed with two carve outs, one's VPNs which is getting turned up this quarter and then the other one is proxy support which they're doing experiments in Android and then we're rolling out iOS this quarter so as of the end of Q4 this should be like fully drop in replacement for that networking stack and then we're now pivoting to focusing on binary size because to roll out we want it to be as small as possible and then there's a bunch of other latency enhancements in terms of like kind of DNS, HBS records that get you like a little tiny bit extra HB3 doing a little better job with HB prefetching so we've got a list and we've been pretty transparent about what we're working on when so again if you're curious you're welcome to drop by we've got a weekly community call on Zoom which is on the Envoy calendar so feel free to drop in or hit us up on Slack if you have any other questions. Next question. You mentioned there's a part of the, not roadmap but the plant work is CLCA compliance. It's a software supply chain for software artifacts and what's the effort there? What does it take to make Envoy compliant and which level do you target? Yeah, so software supply chain. I mean obviously this is a big industry problem right now and I think it's great that it's getting a lot more awareness. Envoy in some ways ironically because it is C++ has I think less software supply chain issues than some other ecosystems again ironically because it's so much more difficult to actually integrate dependencies than in other languages so yay. So I think we have less problems in some ways but we have the same problems in the sense that Envoy even though we've done again not me but we have great people that have spent a lot of time developing tooling in CI to list out all of our dependencies and check them with JAWS and I mean it's like a lot of work has gone into this and I don't know the current number but Envoy maybe you know but it's like I mean it's easily 50 or 60 transitive dependencies that Envoy depends on you know and that's obviously across the entire code base in terms of like tests and different extensions and whatever but I'm sure the core code that everyone uses probably depends on like 10 or 20 different dependencies and those are and those are different libraries and the problem that we have is that from the Envoy side we take security really seriously I mean we have to Envoy is run a lot of places and people that run it you know want it to be a secure piece of software and if we take the security really seriously but the dependencies don't take it very seriously it kind of defeats the entire purpose right I mean that's the problem. So there are certain dependencies that we have that are more critical than others so for example the HTTP parsers are very critical from a security perspective and historically Envoy you know used the HTTP parser from Joian which like is deprecated and then we've used NG HTTP too where the maintainer of that library like refuses to work with us on any security process so I mean it's not safe from a supply chain perspective so that's one area where you know Google has been investing a tremendous amount of time and basically getting us off of those libraries so it's honestly it's subjective right and it's like we have a scorecard that we apply to all the dependencies that runs in an automated way so it looks for things like how many you know PRs, head approvals or code reviews or like do they have a security release process and all of those things but it's a really imperfect process because you're kind of like you're weighing do you rewrite something which has its own risks or do you depend on this library where you could get zero date because they have no security process so it's a very difficult problem which I don't think we have in it there's no like black or white answer to this so it's like what I would say is that this is another area of contribution where the tooling around this to help visibility to identify low performing dependencies and potentially replace them is important and or some dependencies they themselves this is open source they need maintainers right so I mean actually identifying some of the dependencies you know that also need maintainers and code reviews and all of those kinds of things you know that's an area of like general contribution and that's kind of what I was saying before is that none of these things you know it's like the tech press it's not gonna write about any of this stuff this is like just the boring nuts and bolts of a highly used piece of software but it's very critical so yeah. Did you have anything to add? What's that? If we were to add some new protocols on top of Envoy how could we work with the contributors and your team because on our own we may not be able to do all the work we need better understanding. Sorry so I guess is the question that you need help doing the development or you want it to be upstreamed and you wanna know what that process would look like? Yeah so I guess this is another general topic that I don't even know the current number but Envoy is like a million lines of code now I mean which is like incredible to me and that's including you know all the tests and the extensions and everything else I mean it's a pretty large code base and that obviously causes issues for us in terms of CI costs and speed and a bunch of other things so the reason that I'm saying this is that there's a difficult tension between Envoy is a powerful and extensible piece of software and we want it to be used in as many cases as possible but at the same time the more code that we bring upstream there's burden on the maintainers to keep it building and stuff like that because unlike Envoy, unlike a project like Linux which does not build all the software on a regular basis that's in the tree we do currently build and test all the software that's in the tree so every software that's added is tested and like verified which I think is a good thing but it has cost so the reason that I'm bringing that up is that depending on the protocol if it's like a one-off thing for your company I don't know what we're talking about right but if it's a thing that no one's gonna use other than you or your company we're less likely to want to bring it upstream so what I would say first is probably open an issue and have the conversation around what that looks like we do also have something called Contrib at this point so there's two portions of the Envoy code base there's like the main code base and then there's Contrib just for like less vetted contributions and we put that there to help get more contributions in but they have less CI costs and they have less vetting like they're not tested as much and they're not put into the main Docker images they're put into a secondary set of images so I would say start by opening the issue and having that conversation the other side of this issue is Envoy today still does not do dynamic loading of extensions this has been an issue that's been open for seven years there are pros and cons to static compilation versus dynamic loading the project is not opposed to dynamic loading but again there are pros and cons so I'm just trying to give you like the general picture of what we're looking at right now so I think we are open to other protocols but there's costs so it probably depends on how widely used that protocol is and whether it should live in tree or some separate extension obviously Lua is an option for some extensibility WebAssembly is an option for some extensibility I believe soon in the next few months that it will be a go extension model also where it's not tiny go it's not WebAssembly it's like full go that will run via Seago there have been companies actually that have for years done this they've run go extensions within Envoy but none of them have been upstreamed so there is a company in China I think Financial who wants to upstream that that would be another avenue for example of like getting some of that extensibility in there but it probably depends on the use case Yeah, any other questions? This is maybe a little really inside baseball but I have yet to figure out what virtual listeners let us do that we could not do previously could you unpack that a bit? Is that maybe too much? No, no, no virtual listeners it's like a very new thing just landed it was internal listeners is that it? Oh, fantastic. All right, so well the internal listener is used for some use case that Envoy want to talk or Envoy internally want to talk to the same Envoy instance to save the cost of networking costs it's really for some use cases that you just want to plug the Envoy upstream to Envoy like listener like those those can happen within Envoy instead of going through all the local network local host network stuff and that saves you some cost in terms of all the network stack providing faster and you can plumbing those like all the listeners side filters into another like upstream that use in some use case like you'd want to do the tunneling stuff that helping that performance. Addis, do you want to add something? So it's not that there's a lot extra that you can do but the canonical example for internal listeners is imagine you have HB traffic that's encapsulated in Kinect so if you're not familiar with Kinect you have your kind of Kinect Google.com and then inside of that the payload of that is another HB request and you want to process that so the way to do that in Envoy is you literally do three passes in Envoy you do one to strip off the Kinect headers and then you'd like forward it to loop back and then you'd do another one to process that internal payload and then you might have to do one more pass to re-encapsulate, right? And so as Lisanne said it's a big performance optimization to not have to go through loop back but the other addition, yeah and again there's shortcuts you can do if you're like a power user to not do three full passes but you know if you're doing HB2 connect like you really can't bypass that like you have to like de-mux and then de-capsulate and then forward it through another pass. The other nice thing about doing the internal listener is you can actually then not just pass that payload but also pass metadata about the original request about like the original destination IP or you know if you did like PZF sniffing or anything that you've done kind of inspection of that original connection you can pass that as metadata you know essentially for that and have it go all the way through to the end so that can be helpful kind of additional data. You would, yeah you could have maybe attached it through a bunch of hacks but it's just way easier to just literally pass the data structure through so that's like the real win I think. Any more questions? Thanks. So security related question. And curious when you load a certificate into Envoy and you've got a private key associated with that obviously how is that being protected or stored or just in general I'm curious to understand the thoughts around that and where you might be going if anywhere. By default the private keys are stored in RAM so I mean it's pretty typical for most similar systems. Envoy does already today support offload engines for signing so it supports hardware security modules and those types of things. I actually don't know off top of my head what is entry, do you know Leeson? Is there anything actually like entry for HSM or is it all people that have done external integrations? Okay, so it is possible today without any core code changes to use an HSM but we don't ship in open source any HSM implementations but there are people that are using Envoy with HSMs today. Yeah. I was gonna ask about debugging issues with Envoy I found it a bit painful at times having to go to debug logs or trace event. Like I sent an email today to the users list we have at the scale like connectivity issues to upstream when we're having a lot of connections but then the log doesn't say anything else just cannot connect and there's issues like this that I don't know if like increasing some level some logs to a higher level would be helpful for people to not having to, it involves changing the log levels looking at the logs it's hard I don't know if just something other people have an issue with. Yeah, all the other people take this also. I guess my quick statement is a hard problem to solve because logging is actually very expensive. So for a proxy instance that may be doing like thousands of requests per second logging will quickly become your bottleneck and a real anti-pattern or failure in some of these software systems is that if you start logging a lot during failures you'll blow it up even more. So the logging levels in Envoy are very purposeful actually to make sure that it never logs at high volume like really ever basically that has potential debugging issues and there's actually been a lot of work or thought that's been put in over time in terms of how to either make things a bit more obvious in terms of back off logging like only logging every so often or more information in logs or being able to turn on logs temporarily on a per request basis. I think without knowing the specific issue it's such a complicated system that it's hard to say any one thing about what we could do but it can always be improved that's for sure. I know you have a lot of thoughts on this. Yeah, I think as Matt said earlier a lot of the time we get improvements as the community wants it. So when Google joined we basically took everything that we used for debugging GFE and ported it over to Envoy. And so that was a lot of the response details for why is an individual request failing, adding more annotations and these all go into access logs not the standard error logs. Adding more verbose logging, adding the error logs so when something unexpected happens that automatically does a power to back off one log once but don't ever will. So I think again like as we've encountered issues we inevitably sometimes will have to do like our own custom builds with a little bit of extra sprinkled information and then we try to upstream that. So I think in general a lot of people in the Envoy community will reach out for help with debug and we try to help them. I know that when I help people debug I always end up upstreaming what I use to debug their situation or sometimes I'll add some way to dynamically trim things on and off. So again I haven't actually checked my email recently enough to seeing your particular issue but that is another way people can give back as you kind of struggle through this is making the next person struggle a little bit less and sorry if we don't have the exact knobs for you today. Probably have time for one or two more questions. Any anything else? So I was going to suggest to the question about adding new protocol support that there's something called Iraqi that has like a meta protocol do you know what I'm talking about? No, okay. Well anyway, the reason I wanted to ask is if you had an opinion on it is these people have added like a filter that allows to do meta protocol definitions so you don't have to write new filters and I just thought I'd get your take on it but if you haven't seen it don't. Yeah, I think I know what you're talking about. I don't think it's fully baked yet. So right, it's like it's, I think what we told them is they're supposed to rewrite Dubbo on top of it and then when that happens maybe more baked and I think that's happening but it's not, yeah. I mean, I think the only thing that I would say there is that there's still going to be some C++ code that's required less for sure but there's still the burden or undocumented maintenance and all of that but yes, I agree that that would make it easier. So I think we can do one more question and then we'll just be around if people want to ask anything. Any other, anything else? Oh, okay, last question. Yeah, in lack of other questions what's your favorite bug you've had to fix in Envoy? I don't even know. I, that's a tough one. I would actually have to think about that. It's been many years so I'm not sure that I could think of any one particular bug. I don't know, is there a favorite bug of yours, Alyssa? What, what, viola? No, I don't know. Alyssa has infinite buffering as her favorite thing. I call that more of a feature but I don't know. I'm not, I'm not, I'm not quite sure. Like I wish I could give you a good answer but I don't, I don't really have one. I don't know. Leeson, do you have a favorite bug? No, sorry. All right. Okay, well thank you everyone and we'll be around if people have any other questions. Thank you.