 Let's see. Oh, A14. OK. Oh, wow. We have lots of agenda. OK. Should we, I guess we have a packed agenda. Should we get started? Yeah, let's do it. Is Chris actually here? Good to see him. We only have a couple of people. I guess he probably just wants another plug for EnvoyCon. The CFPs are due in like a week, two weeks or something like that. So people can submit. That would be awesome. I'm happy to help with CFPs if anyone wants it who's listening in on the recording or I'm sure other people would too. So great. You want to talk about fuzzing, Harvey? Yeah, just general heads up that a lot of puzzles have been landed on. I kind of feel that we now include coverage of most of the interesting parts of the data plane. I would like folks who are aware of things which are not covered and are interested in having fuzzers written or there are other parts which are not data plane related whether we can get a high payoff from fuzzing. Please do reach out to me and I'm happy to consider adding such fuzzers and at least getting on the roadmap for fuzzing so we know where we can best focus our efforts there going forward. That's just a quick advertisement there. Yes, then on to Congress' GCC. Wait, sorry. Just on the fuzzing thing, just for our offline conversation, just for everyone's benefit, I mean, I think you've done an awesome job on getting all of this working, but I think we're all on the same page that it's worth it at this point just to burn down the existing bugs, basically. Is that the plan, effectively? Somewhere between 30 and 40 of these bugs now. Well, some of them are related to fuzzers themselves or the integration or the integration test flakes and that kind of thing. So hopefully they all just provide a general improvement just to the ability as we address them. There's been some interesting ones just pop up in the last few days. It's one of these header parts that have come up. So we can look into them. I think they're exercising very unusual situations, like very large timestamps and epochs and things like that. We should generally just fix these things. So yeah, that's kind of the plan is to actually burn those down and also just understand buzzer efficacy. There's a lot of metrics we can look at to improve things like how there's stability metrics, there's coverage metrics and there's an indication rate metrics which can be used to guide how you want to tune a buzzer to actually do a better job and actually make some use of some of those. Do you want to briefly just talk about that meeting that we had about the net specter stuff? Like I don't think there's anything secret about it. I mean, it's somewhat related. Yeah, Google, we'll be looking at tooling around what the detection specter gadgets coming in the coming weeks. And we have folks at Google who are interested in this and working on this area. This isn't that's particularly important given the recent net specter paper where people demonstrated remote exploitation of specter by packet timing essentially. And that's definitely kind of interesting. Cool. Sweet. Anyone have any thoughts or comments on that stuff? Yeah, we're just generally in security hardening. I mean, that's an area where we're generally interested in these days at Google. Yeah, I think we would like to basically take on most things that are reasonable in that space and at least get them prioritized and you know, go off the low hanging and figure spent about the first sort of efforts. I mean, no, one thing we can definitely do is get clarity back online. That's something which is in the background. Yeah, I mean like we should definitely do that. I think again, from our offline conversations, you and I are on the same page that co-variety is generally a sea of false positives which is a waste of time. But I see no reason not to turn it back on since I think it's pretty trivial to do so. Even if we're going to give that, you know, report a glance every few months, I think that's better than not having it. Yep. Yeah, there are some other things we can look at also in the area of hardening. Like Clang has stack, smash. Yeah. Which we don't turn on today. I think it's some performance talk. Cast is also like a safe individual libraries and things like that, like they're likely to use instead of defaults. You know, whether we want to turn all of these on in the default hot belt or not, maybe one, but it may even make sense to have a hardened version of on-boy work for folks who are, you know, using this, for example, in-edge applications versus internal applications where traffic is largely trusted. And, you know, over time, that may make sense. There's also other things that I think could be interesting to explore, for example, cutting out of on-boy things which aren't necessary. So examples include, like, do you really need Jason and Yaml support for some folks to do things more optional? Could you, you know, make the codex optional or, you know, that kind of thing? Do you need H1 and H2 support? Most of the actually interesting ones, I think, are places where we can reduce dependencies and third-party software and data plane because, you know, when you're looking, when you're doing a security analysis of on-boy, on-boy is just one part of the picture. In terms of trust, I actually think we have a very healthy relationship in on-boy. And I think most of the contributors, most contributions are systematically reviewed by people who, I think most of the people in our community would trust. Other projects I don't have the same understanding of. I know some of these projects, most of our dependencies are either rule-originated or using many other places. But beyond that, actually understanding the nature of the communities and how easy it would be for experts to sneak into them, I don't have as good a feel for. Yep. Yeah, I mean, this is all, I think, super valuable work. So I'm very happy that it's happening. Is it feasible for someone to go and add fuzzing of, say, like extensions, or is it something that requires access to whatever the systems that are actually running the tests, or is it just a matter of knowing how to do it, or is there- That's actually very easy to do. And there's a lot of good examples. I actually plan on writing up a medium post of fuzzing in the coming few weeks, and that will sort of explain the basics and how to add these. So if you own an extension and want to add fuzzing, that will be the place to maybe take it to start from. But it's actually, it's really, it can be as simple as, like, you know, if you, in particular, if you're finding a fuzzing puts in protoform, or you've got datamole in protoform, it can be as simple as like 50 lines of code or fuzzer. I mean, I added a fuzzer for the route lookup table the other day, which in about an hour and a half, and that now exercises all the route lookup and path matching stuff. And that was a pretty big win for a very good little set of efforts. And so I would generally actually encourage people who write extensions to do that. I mean, and also people who write non-trivial data structure, so a good example might be like the LC try. That would be a good thing to fuzz. Yeah. Okay. Yeah, having a blog post and some getting started documentation, I think will be awesome. I've been really impressed with how powerful the whole framework is. It's pretty incredible what you can do. Yeah, I would have some documentation on that already, but it could definitely improve for making more like a tutorial than, hey, here are the basic steps for getting the build to happen and that kind of thing. Yeah, and like I think just some discussion since you've done so much of the, like so much different tests now, I think even just some best practices because it seems like when you build these fuzz tests using proto, like that's when you can do really incredible things, but even just some guidelines from your experience on the way to think about it, like it's unclear to me how you're generating some of the corpus. So even just like some general discussion of how you've approached some of the tests, I think would be really useful from a learning perspective. Yeah, that's definitely fine. So I'll get that out there and hopefully we can get other folks in the community also building fuzzers and- Yeah, that would be fantastic. Yeah, just read that effort. Cool, so yeah, I think you would have heard me- Well, that's a good, I think that's a good segue into talking about compiler stuff because you had talked about turning on stack, you know, stack guards and stuff like that. So yeah, so I think Peoder opened a PR just to change the default compiler to Clang. I think this is something that I've been thinking about for a while in terms of do we wanna switch over to Clang? It seems like that's where the industry is going. So I'm like, I'm for it. I just think it might need a little more thinking and diligence than just making the change. So I guess I'm curious what people think, but the three things, you know, given magical resources that we might not have that I've been thinking about is, I think we should move from Clang five to Clang six. So that includes Clang six since it's now released Clang six format. So essentially upgrading Clang. I'll give it to Clang eight, like should we be going to Clang seven or is there? Clang seven is the current development version as far as I know. So Clang six is the last like release version, I think. It's what I saw Clang eight the other day when I was looking at some documentation. Then maybe they just released Clang seven. I'm not sure. But I know that in the Ubuntu repos, Clang six is definitely the one that they have and then it's mainline. I'm guessing what you've saw is they're probably about to release Clang seven and that's why it's switched to Clang eight probably. I see that that probably makes sense. Yeah. But like, I mean, but we should, since we're I guess clearly very far behind, we should at least switch to Clang six, particularly because now that they build those Debian packages for Ubuntu, like it's super easy to actually upgrade. So I think we should do that. The other thing that I, so then, so I guess they're switched to Clang six or Clang seven. There's whether we want to switch the default builds, there's potentially do want to revisit some of the compile flags, like turn on stack guards and other things. So I would like to revisit using FLTO. So that's the full program optimization. What would you try in the FLTO on? So you don't, so there's two separate things. So FLTO is just whole program optimization without any training. And then there's the, there's the PGO, the profile guided optimization. So I'm just proposing turning on FLTO or at least experimenting with it. The reason just for historical context, the reason that Envoy has not historically used FLTO is that at least as of about three years ago, GCC has historically had problems with debug symbols when FLTO is on. So like the code might be faster, but then if you look at a core dump, like symbols and call stacks are all messed up. I'm guessing that's been fixed now, but I honestly haven't looked at this in several years. So, you know, it would be worth just looking into like, what does FLTO do for like a standard performance run at least for some real world use cases? Does a core dump look correct? Like can you debug it basically? So those are the three things that I've been thinking about. Are there things that other folks have been thinking about? I would just put out that the motivation for Pira's work is that Istio have had some serious performance. So what's come up is some people using Istio and they configure like a thousand, order a thousand listeners. And this is, you know, obviously with the original destination routing stuff, I've seen like incredibly slow load times, like minutes. And they're deciding that on void, what Istio and by implication on void isn't scaling practically. And so Pira and Lizanne and so on are engaged right now in a bunch of work on trying to get to the roots of this. And one of the things did end up being an LC try and it was sort of solved by switching to Clang. But yeah, that's just some background context. So in terms of the actual things to actually consider turning on, yeah, I think these are all interesting. Well, one thing we really would benefit from and this is a lot of work. So I don't think this is gonna happen overnight and maybe not at all. It's a way to systematically make these decisions by having an open source performance framework for Envoy. I agree. Like these are the tests we run. We saw this regression of 5%, we'll improve them of 10%. I agree. That's like the rational thing to do. Building that out is a lot of work. Yeah, and that's something I would love to work on and maybe sometime in the future I will do that. Actually, ironically, getting physical machine resources to do that is actually not a problem because CNCF has a test cluster. So like we have computers and a network that we could use to actually do such a thing. Obviously, building the thing with like how you deploy and like actually monitor and do all those things. That's like a multi-month effort. Yeah, stable low variance benchmark represents what the community cares about. Like this is a lot of work. Right. So the best that we can do currently to be honest is I think when we make some of these changes, like at Lyft, we can do some basic performance entity checks. It seems like the folks over at Pinterest generally do per fork, like they can test some things. Like once Google is using it more in production, like you'll be able to run some benchmarks. Yeah, we have teams building internal benchmarks which we should actually fill in this role. Yeah. Yeah, that's kind of like after the fact, right? Yeah. Yeah. So I guess what I would suggest is it seems like, so there's two things. So there's the LC try thing and it seems like that was fixed by Brian Payne's PR. So it's like, I don't know that the client fix is even needed. The other high level comment that I would say which is probably obvious to people is that with regard to Istio and people using thousands of listeners, I would say that sometimes some of the use cases I would not consider reasonable. So like Envoy is not being used in a reasonable way. So there's performance fixes that we should investigate but that doesn't mean that like every benchmark that people say is slow means that we have to change the world. Like that's my only point. I agree. I mean, looking at like the specifics there, it seems that there are multiple ways of fixing that problem. They basically provision like all the services to all. Exactly. I've told them this many times, like they cannot do what they're doing but they do it anyway. So I'm just making the point that I'm not opposed to performance fixes. We should absolutely do them. I just think that we have to look at the big picture which is that like we, you know, in helping people use Envoy, we have to also help them use it in a reasonable way. That's my only point. Yep. Yes. That's fair. So like I don't, I feel like with Peotr's PR, I just don't feel like enough diligence has been done. So I would still propose that we close it and open another tracking issue on like, upgrade clang, try to get some input on people testing clang in some production environment to look for like obvious regressions. And then maybe at some point, I or someone else can do some investigation on FLTO and determine whether we want to turn that on or not. Those are, and then I think in there too, we can talk about do we want to turn on the stack guards, maybe look at the Perf Delta for that. Like there's a couple other things. I see no reason personally, like I feel like the Docker image that we export from the project should be a security hardened as possible at the expense of Perf. And then if, you know, we can have guides that say, you know, if you like want to have maximum Perf, you could turn off this stack compile flag or something. But, but, but again, like that's something that we can, we can chat about. Yeah, no, I kind of agree. I mean, that will also put us in a better position if we need to like deal with the security responses and like that, like by having more things out there, which are my defaults safe, yeah. Yeah. All right. So do we then, so I guess what would the, so do people agree that we should potentially close his PR and like open a tracking issue or do we want to continue to discuss in the context of that PR? But I think close, it's bigger issue than just that, all right. All right. Okay, sounds good. Anyone else have any comments on the compiler issues? Actually, we're now using Clang 5, sorry, what? Upstream, we're now using Clang 5. Yeah, baked in like we, we download from the Debian repo and we're using Clang 5 now. So I think just like switching to Clang 6 is a total no-brainer. And I think, I think Peoder has a PR for that somewhere. So we should just do that, like that, that one is just easy. So let's just do that. And then I think whether we switch the release docker image to Clang, I think that's an open discussion. And then obviously these other compile flags like FLTO and stack guards and whatever, I feel like that's a, that's a separate conversation. I was just Googling around. I think Xcode is actually based on Clang 4. Really? How, like, don't they- They stopped a few versions of Xcode ago. They stopped telling you what version of Clang their compiler was based off of. So- Did they fork it and like, they're forked now? Or really? I know, I know someone at Apple that we can ask about that. So I will, I will find out. I mean, it's probably fine. I think it may be a problem someday when you wanna switch to C++. Some newer year may eventually be a problem. And I think the other stuff is just flags that you're not gonna, maybe not gonna be able to turn on in that Mac build, which is okay. It would probably be easier and cheaper if I just went around and gave everyone an Ubuntu ThinkPad and I just dropped this whole OSX situation and then wouldn't have this problem anymore, would we? There we go. Just throw it out there, I don't think- No, that's a good point. I have no idea what the implications are there. Okay, looks like next is packed header format. Who pooped that on there? Well, I put it on there. Like, sure, I don't think it's dialed in, but essentially there's a discussion of like, hey, let's do a packed header format. And then he's like, I just wanna make progress. I don't feel like I don't know if anyone has something into what that format should be, but it'd be nice to decide one so he can come make more progress. I mean, it seems like for better or worse that we've already decided on things, like for the XFCC header we're doing, I think semicolon's unlimited. So, I mean, like all things being equal, I would say we should just try to be consistent, but I haven't thought about it that much. I probably wanna bring Pioter in on this discussion because there's an RFC for like- Oh yeah, that's right. Yeah. No, but I think we would actually like, we're maybe missing out because he actually has some, like he attends IETF and speaks to folks who are designing headers and stuff. And apparently there's an RFC on how to, you know, create these sort of like list-based headers and in a way which is, you know, standardized and hopefully will be adopted more widely for other standard headers going forward. And this has come up in other discussions, like with the Envoy error procreation. I don't know how it was propagation and I'm kind of, yes, I was originally very sort of pro and gung-ho about using JSON and proto and stuff like that, but I've been persuaded that the right thing to do is actually just basically do whatever's in this RFC. Yeah, I mean, if there's some RFC that sounds good, let's definitely look at that. I do think we have to consider some existing places like XFCC where we are doing this. And I mean, it's fine. Like we can just call that header legacy or something and we're gonna do this new version or we can have an option that does old style and then the standard or something. Yeah, okay. I guess can we loop Pewter into that issue? I think for Mike Shor's case, yeah, it's problematic because what he wants to do is going to require changes in Lyft's client apps. And obviously once we roll that out, that's a giant pain to change. So it would be nice to figure something out that's a little more concrete. I don't think what he's working on is so burning urgent that we can't spend like a week or two trying to come to some consensus here. So should we tag Pewter in that issue and try to get a link to that RFC? I just dug up the RFC from a prior thread with you. So I think I got it. Structure headers for HTTP, yeah. Okay, all right. Okay, I'll post that in the bug as well. Okay. Or just those two and people can weigh in, please do weigh in. All right, sweet. Last issue looks like controlling time and integration tests. What's that? Yeah, this is mine. This is just, I've noticed while looking around the code and also we've been burned recently with some test flakes that a lot of code is pretty much just using real time in the context of integration tests. And I was surprised by that and I thought maybe there should be some way to pass down like a time source from the server or something that in the context of an integration test would be simulated. And thought that might be a bunch of refactors, which I wanted to just get a sense is that going to be something that we wanna see or is that something where we just want integration tests to use real time and just make the timeouts long enough so flakes happen less often? No, you and I are on the same page. Like I hate statics, I hate real time. Like I would love to purge it all. I think it's just the code has grown organically to be like it is. Yeah, so if we start generating some refactors, as long as it's kind of in that direction that'll make sense. Yeah, I mean, while we're on this topic, one thing that I would love to, we're kind of out of time, but like one thing that I'd love to figure out, maybe we can do it over email or something is, I feel like with the integration tests in particular, we need to have some, like we should have some guide around like how to test it before submitting it just because it has flakes. But anyway, we can talk about that next time. I think it's out, so yeah, we have to go. Okay, that's good. All right, anyone else have anything? No? Okay, all right, bye.