 I'm not sure which other maintainers are joining. I see we have Snow and folks in Cambridge. There's Matt, great. Hey. These are items to the agenda that you'd like to discuss. I have a couple of API fronts, but I'm sure there's more. OK, well, let's get going. And folks can add stuff as we go. So one thing I'd like to just bring up is something, an issue that Google's faced recently, as we actually added in a PGV annotation, which actually, you realize later, may be interpreted as a breaking change. So one of the things we've become pretty strict about in the APIs is not breaking APIs at the proto level or semantically. That is, like, removing features, which have no way of being re-expressed. And I think that's great. One area we haven't been particularly strict upon historically has been adding in additional PGV annotations, which, because they're annotations, they were never really considered to be part of the core proto API. But in reality, they actually are, because we start rejecting configurations when we fail these PGV checks. So I guess I would just like to ask if we're just generally comfortable, particularly the API shepherds who are here, which I think is just mad at myself for being much more careful about pushing back on PGV changes and making sure of these. The only safe thing to do is to make a PGV change weaker, I believe, within a major version. And adding in a strict PGV check is actually a breaking change unless there was already an implied contract. And I think that's where things become a bit murkier. And you can see in this particular case that there really was an implied contract that you, for example, you didn't put null characters in the middle of a header name. That would be bad. Envoy rejected that anyway, it would crash. But if you're talking about like full RFC compliance for header names, values, and being very strict about the character stats there, that didn't, I don't think that was actually necessarily an implied contract, but that's debatable. So they say any sort of RFC compliance proxy should enforce these things. I think that makes sense. One thing that I wonder if we should track from a feature perspective is if we could add some type of validation hook to PGV, it would allow us, for example, to do something like change one of the validations, but put it behind a runtime hook and then actually go through a deprecation period. And I mean, it would be some work. Like I'm not saying that it's easy, but it seems doable. And that would allow us to potentially do things like this header change, which are probably a good thing to do, but we probably can't do by that definition until we cut V4. Yeah. I mean, the question is for other users of the APIs, is that an acceptable thing to do as well? Like, we moved to the major versioning and the clock cycle around the APIs to provide folks with these stability guarantees. And now we're saying, well, the APIs allowed this yesterday, but in three months' time, they stop allowing this that might make other API users unhappy. I think that's probably true. I think there's a bigger question here, though, which is that we don't guarantee that PGV is supported in every language in which this API might actually be used. So I guess it's like, do we even have an official policy on, like do we assume that the PGV validations are a critical portion of the API? Like I think the answer is yes, but that's something that we should probably document. So are these PGV validations not actually exercised in every single embed build? They are in Envoy, but we don't know, for example, that the people that are building control planes, like we don't know that they're using those validations potentially in their tests or something like that, right? Because they could be building a control plane in any language that Protobuf supports. Right, and I'm not even sure what the status of GRPC is. I mean, GRPC is the other big consumer of the APIs today. Yeah, and since their core is written in C, I highly doubt that they're doing any validation. But not sure. Well, they have that three languages implementing the APIs, Go, Java, and C. I guess my personal feeling is that I think the answer is, yes, like we should probably assume that PGV is a core portion of the API and we have to treat it with the same deprecation stuff that we do elsewhere. It does seem though that there should be some judgment here, which is like in the case of this header one. I mean, this seems like the kind of change that we probably should be able to make. And I'm just suggesting that there may be a technical gray area where at least in Envoy, if we could also annotate a change with like a runtime feature flag that would allow us to go through the normal deprecation period. Well, I feel like the thing is like PGV annotations are a convenience for developers. You can always implement some that's in a PGV annotation in code in Envoy rights. And we could do the rejection there. So we could always have Envoy just using normal mechanisms adding to the bootstrap or something like that. Hey, I'm gonna be really strict about enforcing header RFC values and just do that there versus doing it in the PGV annotation, right? Sure, yep. Having runtime flags would definitely help a lot. One of the main problems right now is that like we have no way to turn this off. Yep. The question is also a problem. Yeah. And again, like just like quickly brainstorming the technical solution, I could imagine that you could add a flag to the PGV annotation, which is something like feature flag name. And then we would just have a hook where we could plumb that into Envoy's runtime system. Am I actually making it more difficult across languages especially? So this flag's nice to be specific to the core Envoy implementation not something that like other users can actually rely on. Yeah. Right, yeah. I mean, I don't like from a library perspective, I don't know how PGV could offer something that was general and let's PGV itself had some type of API where you have to register your feature flag system or something like that. Yeah. I think the great way of doing this would be to deprecate the field and replace it. Yes, yeah. That would be the cleanest way of doing this. It would be the most pain to the most users. And again, that's why it's like, I feel like there's always some judgment here, which is we're trying to balance pain to users versus doing the right thing. And it's like, this feels like a case in which we could probably just do it. But, you know, it might break someone. So it feels like we need to at least have some type of feature flag control. So the reason I'm looking into this was actually related to like the validation of the path rewrite and host rewrite things, which are actually a lot more likely to actually have configuration that is actually incorrect and why it should be rejected. And someone's actually depends on what the validation ends up being. And someone's actually would prefer to be fairly strict and do I actually avoid users like carrying themselves by actually making a rewrite that why should we black hold the requests? It's all of them. I think the more complex this validation becomes, the more likely it should just existing code in Envoy and it shouldn't just be like, it's pretty, we don't know, PGV isn't particularly expressive today, for example. Like if we had like full cell, you know, expressivity and that kind of thing, that might be different. I think that the validation of this match should be as simple as a muster with a slash. And that would actually like break a single fix. The past muster with a slash. That's actually like the extensive validation might actually be, but like that might still be breaking things. You could also imagine replicating, like if we had a solution where we deprecated string header name and change it over to a wrapped message that was like valid string header name, you could imagine saying something like when we validate our protobufs that we like re-implement the PGV check-in code because we know what type it is. That seems like a lot of API turn. It is, it is. And the other benefit. Go ahead. I was just gonna say, the one benefit from a PGV type annotation is that at least theoretically, people with control planes or systems that validate config, they could run these things in advance. Agreed. And that is kind of a nice thing for people that are building systems to not have to send it all the way to Envoy just to figure out that it gets rejected. No, no, I agree. I mean, you basically have a contract that are explicit. I mean, the concern over API change, I don't think it's as big a one because guess what? We have a lot of API turn anyway built into the new major version thing. Every year you're throwing away all your old API code and running new stuff. Yeah. That can be, that can be pleasant for anyone. Yeah, but that's, we can discuss this offline and turn here, but that was basically the community agreed approach to providing stable APIs for Envoy and other clients, such as GRPC. Now that was a Google request in one way or actually resourcing. Sure. But it's, yes, it's not pleasant for anyone, but it turns out that there actually aren't better stories in the proto or other than never deprecate a feature or basically break wire compatibility based on a calendar cycle, which again is not suitable for long-lived projects. Just to get like a closure on this, is it okay if I like the next steps that I make would be to like file a public issue and make a PR that restricts those validations down to the ones that we already have implemented in code and do that churn of marking the field as deprecated? So the code replaces characters as opposed to filling out. The asserts? No, I mean, there's actually some, your field also like changed some of the- Those are only for tests. Oh, never mind. Yeah, those changes were for tests. Okay. Yeah, nothing in that PR changed prod code. It was just API change in test code. Yeah. The prod code would just like in top mode would just like do random things. Yeah. Anyway, does that make sense? Yeah, that makes sense for me. I think that's fine. Yeah. Are you gonna add another like well-known name to PGV? You're just gonna go straight and just do make it explicit in the onboard code? Like that's what I'm trying to figure out. Like it's like adding another field to PGV unless like I can get you to monitor that PR like it always takes me like six times as long as I need to just to get people to review it. Yeah. But I'm willing to do like a on boy valid slash r slash n slash and like no character check just so that we don't have to rewrite it. I mean, this check after it's in some of the different places that's probably what's the name. Yeah. Sorry. If I can just like bug people. Yeah. If you would prefer to do it within PGV but you feel like you're blocked there because of code reviews, let me fix that. So I mean, let's talk about that offline. Like we can get you code reviews faster or we'll just honestly make you a maintainer of PGV. So let's just do the right thing. Like let's not do it in PGV because you're frustrated about code review speed. I'll turn that up then quickly. If I can get reviews on that then that's the easiest solution. Yeah. So if you would prefer to do it in PGV let's just do it there. And then if you could ping me if you feel like you're blocked we will figure out the right way of getting you unblocked. Sounds good. Thank you. Cool. What else is there? Anyone tried any of the V3 API stuff and have anything to report? I kind of doubt that anyone is using it yet. And to be honest, it's like a chicken and egg problem because I think until we block people from adding things to V2 like people probably aren't going to move to V3. So I think we probably just have to move forward with the blocking. Okay. So is everyone okay with the current plan of record that at the end of Q1 tooling allowing we will stop allowing additions of any new fields to V2 APIs and the new features need to be go straight into V3 or V4 alpha APIs? I think this first transition is probably going to be a little painful but I don't see any way around it because I don't think we're going to flush out the issues until we get people to move. Okay. Well, I think that's the plan then and yeah. Hurry. I just have like one piece of feedback on the API tooling and I don't know if this is just my own problem or everyone else's but typically when I'm making dev changes now in Envoy I make changes to the V3 API but the proto format script will only push V2 to V3 not V3 to V2. Yeah. I mean basically what we would do is again for until end of quarter probably you would make those changes to V2 and then at the end of quarter you would we will flip this around so that V2 is effectively frozen and no one can modify it and then V3 gets to be the canonical source of truth and that will propagate forward to V4 alpha. Yeah. Like from a optimal tooling perspective I think where we need to go from a North Star perspective is you should only have to edit one proto. Like everything else should be automatically generated. So it's like all of the shadow protos like all of the next version protos like to me that's a build implementation detail. So if we can actually get to a point where you run Bazel builds and it just like does the right thing I think that would be a much better developer experience. Like obviously we're not there now but I think that should be our goal. We need to eliminate the explicit shadows that Steph has actually done as part of our technical debt burn down. But yeah, I mean right now you are only supposed to modify a single file it's just confusing what that file is. Well there's that but there's also I think it's more confusing because people don't generally like it used to be where you edited a proto you do Bazel builds and you would automatically get all of the changes. Now you have to edit the proto you have to run the proto formatter then you have to do Bazel build, right? So it's just like what I'm suggesting the dev flow should be from an end goal perspective is that you edit the proto you type Bazel build and it just does the right thing. Kind of skeptical that can actually work because you need to do some non-hermetic things there but we can discuss like I think we probably need to ease out and maybe some other Bazel experts in the room because I know in the Bazel run you can do some pretty non-hermetic things but that's different than Bazel build. I think it's fine again if it doesn't work that way today but I would hope that we would agree that like that would be our optimal state, right? Like it would be nice if the whole thing was hermetic and like I just make a proto change and it just does the right thing. How do you rewrite the source tree? I mean, you need, there's going to be multiple versions living in perpetuity of each of these APIs in the source tree. You need to do it somehow. I feel it's going to at least, like always at least one step in which a file gets copied back out of the Bazel cache or something like that to make this work. I honestly don't have the answers but I just feel like that would be where we'd want to get or at the very least when you type Bazel build there would be some type of error that says like, no really, like there's a change but you have to type this other command or something. It's just like the flow today is quite confusing. The other thing I've got is definitely plausible. Yeah. Yeah, I think that's the recommended way of doing like checking in generated files. You add a check to the build to verify that it is what it's like, what is the effect to it? The other thing that I found is that doing git push now takes quite a while because of the git hooks. So even if I'm not changing any APIs I still have to go get a coffee while I wait for all of the... You should do dash, dash, no verify. That's what I do. But then I slam by the formatter because I didn't... Yeah, so what I do, like this is horrible but like I do the hook until it starts running Bazel then I control C and then I do no verify. Yeah, I do that for a while too. I feel for someone who's sufficiently irked by this should just basically figure out how... I mean, it's not just with the proto stuff that we discussed the other day. I think as we brought up it was the Python stuff as well. We're running checks which don't need to be run basically during fixed format or check format. And we should just basically be able to compute everything that's going to be pushed and only if there's a file that's modified with the proto suffix or a Python suffix that you run the respective checkers on. And that's a project for someone who really wants to make this fast. I don't know if it's enough of a pain point for someone please go ahead and do that. I can create an issue to track. Yeah, I also think like we're like format checking like the generated files during the get hook which presumably as long as we're... If it's getting checked in CI, it's probably not useful for me to check every time I like in the get hooks. But yeah, it's still doesn't work that we get done after like streamlined this. What I would recommend doing is just opening tech debt issues for some of these things. There are people out there that like fixing these types of things. And it's like having these issues opened it'll make it more likely that we can, you know get people out within the community who might want to work on them. Yeah, and like I said, very quick work around could just always just create fast fixed format which doesn't do this stuff. It's, yeah, I can just make that the pre-submit hook. I actually almost want to do that just to avoid this bug that is hitting all of us for the rewriting the build files. Hey, I have an opening. I know, I know, I know, I know, I know. It's just driving me insane. It's very weird what's going on there. There's either some path issue where it's actually not seeing the file or some reg X issue which I'm yet to figure out. Yeah. Okay, do we have anything else on the agenda? Good ones. Twice. Okay. Okay. We'll see you in two minutes. Bye everyone. Yep. Bye.