 Do you have other topics? This is not really like high priority. No, I just wanted to bring up that. Yeah, that's fine. Let's just gather if there's any other items that people want to talk about. Okay. So yeah, I think we have a long sending like all the issues switch C++ standard library to lib C++. So I think Matt, your question is basically how much lib C++ is used, right? Yeah, I mean, just because, again, my information might be way out of date, but I remember last time that I had looked into this, it didn't, it seemed like, you know, people were starting to move to Clang, but most people that were using Clang were still using the GCC standard library. So I just wasn't sure how much production usage lib C++. Actually, my understanding again, it also may be out of date, is that on Mac OS, lib C++ obviously is like the standard library and then there's a heavy use from Linux. It was much less mature. Right. Yeah. Well, I don't have much data about this. Probably hardware, you might have better info, but like, I think Chrome have switched to lib C++ one and a half year ago, is something like last year, January is timing. And also I think the Android use the lib C++ for the Android and DK stuff. About server side, I don't have, really have much idea from there. I know some BSD, like open BSD probably use that by default, fish, that's so far I know. Okay. Yeah, I mean, it would make me feel a little more comfortable. Like if we knew that there was a little more server side use, I guess my question is, do we think that we're gonna be changing things such that once we switch, we won't be able to switch back? Or like, will a user still be able to compile against GCC standard library? I think we will still keep the compile compatibility and for the official image, we might be able to do like gradual change, like we published two image at the same time during some period. Yeah. One side job that verifies that we maintain lib standard C++ is that the idea? Yeah. Okay. This is probably a good time to, there's an issue that I open, which I will link to. But one of the things that I do wanna start thinking about is whether we want to have multiple images, multiple, you know, like, so basically we'd have a, you know, kitchen sink image with every extension and then we would have like a normal or a light version or something that has extensions that we feel that, you know, people would reasonably use. There's just, there's been some concern that, you know, particularly with providers that are trying to support, you know, quote, unquote stock envoy that as the number of extensions gets larger and larger and larger, the code surface gets larger and larger and larger. Yeah. We have that exact same consideration and then we filter out extensions. I mean, you know, anyone who's a suit, I think anyone who's like reasonably sort of as competent and you're working with envoy should be able to build an image, right? Which, like, I think you're right, but the problem here, and this is actually partly coming from Google, but partly coming from other providers is for products like Traffic Director or AppMash or other situations, there is actually a desire to be able to bring a quote, stock envoy, right? Like take the public docker image and just have it run. And- Is that our responsibility or is that the responsibility of someone like, I could say, Tetris who's doing, you know, binary distribution? I don't, to be clear, I don't have any answers here. Like this is a open discussion. I was just suggesting that if we're starting to take the stance, which I think is reasonable for now, that extensions are like drivers and we're gonna allow drivers in from all kinds of different providers, some of which people won't care about. Do we continue to build and ship everything in our public sanctioned image? And it just, it occurred to me that if we're gonna start doing multiple images, potentially for like a lib C++ and like a GCC standard library, is this an opportunity to start thinking about multiple SKUs of public images? That's all. Okay, so my personal take, and this is, I'm not speaking for Google, I'm speaking for my area as an envoy maintainer is, you know, either I feel we're in the business of doing binary distributions and we should do a good job with that. And that involves folding and qualifying and testing on multiple platforms, all this kind of stuff. Or we should just like, we should never actually even have a Docker image involved with a binary and it's enough to make this the de facto and just full distribution. It's like, go speak to an envoy binary distributor who would do the supports and qualifications for your platform. So that's one extreme position. I wonder what others think. I definitely hear you. I don't think that's gonna fly. I mean, there's already plenty of rage that, you know, we don't chip binaries in every possible thing. And I think this get envoy thing from Tatchrate certainly helps. We like, we don't, we don't, we don't need to harp on this. And again, I'm curious to hear what other folks think also, but it just seems like eventually we need to figure out what to do with our official images in terms of whether we continue to ship every possible extension. So here's a question. So like CNCF, they'd actually fund things which are, you know, very useful to making open source work for the community. And there aren't things that, you know, necessarily, you know, the core developers work on, like for example, CI is an example. With binary distributions for that category, can they pay contractors or someone to actually do this for us? And you can manage that part of our work. I don't know. That's a great question. And I mean, that would probably be a conversation between at this point, Tatchrate and CNCF because that's what get envoy basically is. Yeah. So I don't know. What do you think, Lisan? Yeah, I think that's reasonable, like for multiple image at this point. But I think for this like specific issue about libc++, I think we will do, we will have to do some kind of like dual image for a while for that because it's breaking the only, I think the dynamic open tracing stuff. That's for sure. So dynamic tracing does a dynamic load of a library compiled outside of our build system. Is that why that? Right, right. That's what dynamic OOT does. And I think because the ABI compatibility we have here at it. Yeah. For sure it breaks the existing users of that. Have you reached out to Ryan Burns offline? I see that issue, I don't know if you are not contacting him other than that. Okay. I think the switching is not only for the open tracing issue, we have multiple motivation to do that switch. Okay, maybe just on that open tracing issue, do you wanna just either ping me on Slack or send me email and I can add Ryan Burns and we can just see if we can get that memory leak fixed. We should just do that either way. I see it on that issue. Okay. Well, I'm just saying he might not be checking GitHub but it might work better over email. Yeah, sure, but I think Josh mentioned that the memory is more libc++, more memory efficient. Sorry. Josh? Yeah. Was there something I needed to add? You're right, I have an example. I think you will mention me that libc++ is more memory efficient in terms of the standard string implementation. Yeah, it just inlines small strings into the base of print of the object. So anything 22 characters or less does not require like any separate pointer allocation. It's very efficient. I thought that was a standard string optimization these days. The standard lib doesn't do that? libc++, I don't think does that. Really? Yeah, I don't think so. libc++ does that. That's very surprising. I thought that was like a string optimization that everyone does now. Okay. Well, yeah, I mean, again, I don't have any objection to the libc++ change, I would just love to learn a little more about what the production usage looks like. Cool. I mean, this is the default in Google now, although I don't know how long, I'm pretty sure that's true and I don't know how long that's been the case. So you're using internally for all of your server software and Linux. You're using libc++? Yeah. Okay. Well, I'm pretty sure. Yeah, I think it is. There was announcements about it recently and that you have the option when building a binary to back out. Yeah, opt out of that. I mean, if you're using it everywhere internally, that's good enough for me. So... Yeah, I feel like it's getting a fair amount of mileage. Okay. So, I mean, if other people, I have no objection, so if other people have any thoughts, we can raise them, but that sounds fine to me. I mean, the part of this that I really like is that with this, we now have basically, as we track with a particular Clang version, we know exactly what software we're running on every distribution, which is quite nice. Yeah, but yeah, for the... Right, I think that's one of the major motivations for this as well. Okay. Cool. Does anyone have any more stuff on that? Anything that makes everything go smoother and makes memory smaller is good for me. Yeah, okay. I was gonna briefly talk about the regex issue since it was already open publicly and there's some things that I wanted to chat about. Any objection to talking about that? No, let's talk about it. So for people out there, there's an issue open, I can link the issue into the doc where some fairly simplistic regex causes Envoy to Oom with large input. And it basically has to do with the fact that the standard C++ regex implementation, it has some kind of recursive memory hogging implementation. So this kicked off a long discussion internally within the maintainers and the security team as to whether this should, we should count this as a CV and do it in private. And since the issue was already opened there's already warnings in the docs about regex is being very risky. We decided to do basically like a public security issue. So what I am working on right now is I'm basically going through particularly the routing API and looking at everywhere that we use regex and I'm introducing like a new regex message structure that'll be a little more flexible and have more controls and be more clear in terms of what dialect is using. And then I'm looking at the Google v2 library which is designed for untrusted input and at least as the documentation says is supposed to execute in linear time and safe amounts of memory no matter what the input string length is. So anyway, so I've been looking at that and I've been looking at introducing some new abstractions into Envoy to hide the actual regex engine and implementation for most of the code so that if later we decide to do something different than re2 we can swap it out. The one thing that I was looking at which I was I aming Josh about is that I was trying to see if I could replace most of the uses of standard regex with re2. And I think this has already been brought up but the re2 API is a lot simpler than what standard regex supplies. And there are some pretty complicated use cases in like the stats tag extraction code that I don't think re2 is probably ever going to be able to do. So my current assessment is that we're probably never gonna be able to get rid of standard regex at least in certain parts of the code where we have trusted input and trusted regexes but I'll do what I can for the untrusted part. So that's the current situation. I'm working on that right now. Yeah. Does that make sense to you, Josh? Yeah, that sounds good. Like I was sitting and I had mentioned this all I am. I was working on this like well, I think well over a year ago but I do remember that I had a goal of getting rid of regexes from that tag extraction code entirely. Yeah. And I had gotten about 95% of the way there but the technique that I was using couldn't handle a couple of the forms and so I would have to have like two strategies and it didn't seem and it seemed like with a bunch of other hacks, I got like two second startup time on 10,000 clusters which I thought was kind of good enough so I just looked. Yeah. So my question there is can we get rid of it at least from the API for stats? Like the bits that appear in bootstrap and so on where we specify the tag extraction code and just keep it for the internal uses. Yeah, that might be the case. I would rather do that as a follow up just because I don't know what people are doing there and again like if people are modeling their regexes off of the ones that are checked into the code like I almost started crying last night like these regexes are, I mean they're like you need like a PhD and like a fellowship in regex to understand these regexes. Yeah, I don't understand. We had some very smart people working on writing them and that's what happened. But like could we at least for stats like just add the again seek regexes an option there and then we'll figure out how to deal with the actual turn down and deprecation which has been happening. Yeah, I think that's totally reasonable. I think I would rather do that as a separate change just because it's not really security critical because all of the strings there are effectively trusted. So, although even that's not necessarily true just in the sense that I mean I suppose people could be generating stats that are somehow based on user input but if they're doing that, that's insane. So... How are you generating like the cluster? I've lost complete track of how you do the forward proxy behavior but originally we're talking about dynamically generating clusters based, how are you, are you still doing that? Are you naming them based on the inputs or? No, so that's not there. There's no dynamic generation of clusters. So there's no regex there. But I mean in looking through the route API, I mean there's a non-trivial number of cases in which we allow regex and they're all busted basically. So, but those I think it's straight forward to fix all them it's just that we're gonna have to end up deprecating a bunch of fields and I actually have another question for the group which is because re2 will not support all of the regexes that standard regex dialect supports. I actually am questioning whether we should deprecate the old field or just have some kind of warning and stat that basically warns people that they're using an unsafe regex and then they can have a config to like turn off the warning or if we should fully deprecate it. I'm just concerned that basically like we're gonna deprecate functionality that someone is actually using somewhere and then with the re2 engine they're not gonna be able to implement it. So, before that I have some basic question about that since we don't do any exception catching on the regex matching pass. I thought the C++ standard regs would raise the exception when runtime some issue have on the complexity than the stack. So, would that simply do that solve some of the case that we have the crashing today? I don't think so. You're good? It's an oom now. Yeah, I think the standard regs have the error stack that there was enough, not enough memory to perform a match or some error code like that. Since we don't catch any exception there today. The case that was failing though was actually I don't know how much we know and how much was the conjecture. I think just running out of stack space. Right. Yeah, there's explicitly C++ standard regs constant error code says error stack was enough memory to perform. What I can do today is I can take the case that was reported in the issue and I'll run it locally and just see what actually happens. I don't think it changes the fact that we still want to offer because it's much safer. I mean, but if that can cut by exception, I don't know if that case will be doing the catch anyway. Yeah, but I guess what do people feel about should we like on the public API perspective, should we fully deprecate the config that effectively uses standard regs or should we still allow it but warn or like have that be part of our linter? Like I could see both arguments. I mean, my gut tells me to deprecate and still someone like, you know, complains very loudly just because it's unsafe, but I'm not sure if people have thoughts there. My opinion is deprecate the old one but I have no idea what the community will say. Yeah. So I need more. Well, given our current deprecation cycle, I think my plan unless people strongly object is I'm gonna deprecate the old way and we can wait and see if people complain. And then if they complain, we can figure out what to do. Like we're not removing anything and basically from the V2 API with the V3. Yeah, that's fine. There it comes. And we don't hear any complaints in the timeframe of the V2 API or after we've deprecated and feature flag turned down this and people are, you know. Well, it's more, it's more though given all of the work that Alyssa has done with the stats and the warnings and stuff like that, there are people now that are trying to run like deprecation warning clean. So, you know, if they start seeing a deprecation, they're probably gonna notice and then they're gonna possibly complain about it. So I think that's the case in which hopefully they would tell us if they absolutely cannot move to the V2 dialect. Yeah, that's the one. If they don't, they have had a bunch of lines of needles to... Yeah. Yeah, the... The only concern I have, again, if we do this, it'll be like the six month cycle where we have to build a release and wait and then build... Yeah, that's fine. I mean, I'm doing the config in such a way where just based on some of the names of the fields, it's not like the names that I would necessarily like. Like I have to use safe underscore regex, but that's life. We can clean that up in V3. Is there a path where you, when you see a regex in the API, you attempt to compile it with RE2 and emit a warning if it won't accept it and then do the deprecation in how it is. So we would basically change the meaning of the fields. Yeah, we've discussed this before. I think this is feasible. Actually at one point I thought, what I'd like to do is make a PR into Envoy that will look at each regex and decide on a regex by regex basis if it's compilable in RE2 and make the decision that way. But fortunately I didn't go down that route. I could take a look at that. I'm also concerned just about subtle behavior differences. I feel like it's better to be explicit about what we're doing, but I could look at that. And also like, I mean, currently what guarantees are we making to our users? Are we saying, no matter what a regex if you put in there it's safe or if you happen to write one which might work in RE2 and it'll probably be safe, but otherwise it won't be safe. Like I guess it's like a very complex model for the safety of regular expressions. Whereas if you have a field which is safe, well, I mean, it says safe, it must be safe, right? Yeah, 30 and also I think that it's, it's not impossible to like look at a regex and decide that it either is directly compilable in RE2 or you could convert it to one. But there probably is a class of regex patterns which would compile in both compilers and do something different. Right, and that's my big concern. And the way that I'm looking at the messages right now at least on the API config level, and we can obviously cover this during code review as I'm actually trying to be pretty explicit and future proof about which engine we're using so that we don't wind up in this situation in the future so that people will basically know that they're using the RE2 engine. And then if magic regex engine foo comes in two years, we can add that and not break people basically. That's fair. Yeah. So as like I said when I saw that as a site idea for that is for the route matching part, shall we do something like this is like a little bit separate from the regex issue? Shall we consider something as some of the URI templating based route match? Since what I have seen so far is that the people use the regex for that purpose for a lot of cases. Isn't that basically another regex syntax slash engine? No, the URI templating is what you specify for the GRPC templating and what the format of the variable bindings in the open API spec that you do the like the curly braces with variable names. Yeah. I understand that it's not a regex syntax, but are we really saying we want to have essentially a pattern match option and you can plug in, you can specify through some enum whether it's RE2 or PCRE or URL templates, then you supply that pattern. I mean, it's something that I think is worth thinking about. I think it has to be a different issue. So we should. Right, right. I mean, it's a different issue, but like to, I mean, to reduce the radius usage. I agree. That's probably something we can do because that is a widely deployed pattern already. We saw open space stuff and zero PC transport in bindings. And you know, the thing that I like about that and I've experienced this now at multiple quote, web scale companies is, you know, as much as you try to push back on not having regex routes at the edge, they creep in, but my experience is that they're typically a very, very, very small dialect of regex and it's possible that we could do something simpler that would cover almost every case. Oh, yeah, I don't know. Do you wanna open an issue on that? Yeah, sure. Okay. All right. I guess, yeah. So I'll just keep going on that PR and I'm mostly out Thursday and Friday. I'll see if I can get an initial version out by tomorrow with all the other code reviews that's happening. I'm not sure that's gonna happen, but I'll try. Okay. I wanna add two more quick points that we're gonna just interject. One is that we discussed, actually, we discussed this internally yesterday. We like the RE2 option as the mainline thing that we think most people should do, but it's worth pointing out that PCRE mostly officially accepts the same language as stdregex and it's probably better. I don't know a ton about it, so it's an interesting thing to possibly, it should be in the table as this is not for the future. The other one is that a user can inject new stats into Envoy via a request header or a response header. I forget which one. There should be codes that- Right, but those cases all have to be turned on like for security reasons. Like none of those cases are on by default and if that's the case, then we need to fix that. Okay, we'll talk offline about that. Yeah, like I'm fairly positive that all of those cases should be edge sanitized. If you know of a case that's not, we should talk. Oh, might not be seeing all of it, but we can tell it. Okay, real quick in the one minute that we have, there's the KubeCon intro and deep dive. I sent an email about this for maintainers. I've gotten a couple emails back from Snow and Steppen. Obviously everyone that's there can come. It's mostly that we get four free maintainer tickets basically, so if people want a free KubeCon ticket and you happen to be there, just let me know and we can figure out who gets them versus who might be getting a ticket for some other reason basically. So you don't have to answer that now but I don't know what people's plans are in terms of coming to EnvoyCon or KubeCon. Yeah, I haven't figured that out because of the GCP Next in London that week. I don't know whether I will go to that live. Yep, okay. I am getting kicked out of this room, I gotta drop off. So, bye talk to you later, bye. See ya.