 Oh, hey, I'm not the only lurker, but it's excellent. Yes. So you can do it. Do we have any agenda items? I'm not sure what they will be. I'll win my vision proposal. Oh, OK, cool. I don't know if it's worth. Do we have anyone who hasn't already talked about it at length? Yeah. I mean, covering your 30-page documents in this meeting might be difficult. Hey, the right eye talk, I can go through a page of it. That's true. It's quite possible. Yeah, yeah. I mean, we'll get it at least if you do that. Not it. It's you. I haven't read the full doc. You shouldn't have read it. I just saw it was placed on pages and I'm like, so that's what we're getting a summary, right? Something like that. You should read on the right audio books. There is something to be said about making a document long enough that no one will read it and they'll just say yes. So that is a strategy. Yeah, I mean, basically I feel like unless there's any strong pushback on the versioning proposal, it's just we'll start rolling it with it. I mean, right now I'm working on the next TDSPR. And once that's done, yeah, I'll start actually working on some of the machinery and things that we need to actually make this all for. Are there, I guess my question is, are there any remaining open issues? Because I know we had some concerns about either the timelines of things. And then I know there are some concerns around how people set deprecated, which is part of the TDS proposal. So I'm having a hard time wrapping my head around. Are we good or are there still open issues? We have a list of open questions in the doc and they're not very interesting. I mean, some of them are like, when are we actually going to really deprecate V2 as it's done today? I think what we'll do is we'll go to come, yeah, we'll essentially work towards having V3 stable APIs ready based on the existing V2 with some cleanup by the end of Q2. And then we will also have V4 alpha. So we'll have three APIs at the end of Q2, V2, V3, and V4 alpha. OK, yeah, I guess my sense is that it's going to become clear once you build the tooling. Like there's probably, I mean, there's some, I guess, conceptually, this all makes sense to me, but I feel like until we can play with the developer flow, it's not quite as clear what the implications are. Yeah, I agree. So that's why I think you want to set up a tool, a playground. So we have two separate things we need to do by the end of Q2. The first is get the V3 APIs ready. And I think a lot of that's going to be pretty mechanical. Is there any place we can clean up stuff, like using a unified header map format and all these places where we know we've replicated things? And it'll largely just be a refactoring. I don't imagine any significant real change. We'll clean up one-offs and all this kind of stuff. And then there will be the actual tool and just support all of that. And that's probably the most the work will be. Yeah. The thing I have the largest concerns about, and again, we can handle them, but we need to make sure we do, for when I say VMD carving, is the pushback will get specifically from Google Cloud, like a guarantee that we're going to. And so what we need to do is that the moment we cut something, go to them and say, you need to do your comms now, you need to act that you've done your comms, the clock is ticking and on point. Yeah. Yeah. Three-year crop rather than two years. I mean, basically, when we push stuff into v3 at the end of this quarter, nothing that's been basically deprecated will survive into that v3 API. Yeah. So that's what calls us to have to go to them with this. Yeah. No, no, no. But my point is that if we think backwards a year before we remove anything, a year and a quarter before we remove anything, we have to get those teams actively. Like, you have a key zero internal dear comms before the end of the quarter because your policy requires a year warning to customers, and you've got a year and a quarter. So by end of quarter, you have a year and a quarter. Yeah, one of the things that I was thinking about is I almost wonder, we don't use the Envoy announce list very much or very consistently. And I guess one of the things that I'm wondering, and again, this is me just brainstorming out loud, is should we have something similar to the security distributors list for deprecation clock tick, right? So that people that care, they're going to get a cadence of announcements on a particular email list for distributors, which basically says, this is when the clock starts. This is when it's going to be going to be done. And to be honest, I don't see it as any different than when a bondu does long-term stable releases, and they say, it lives until ex-date, and here's where we're doing patches, and then we're no longer supporting it. These calendars are written down, and they're well understood. So I'm wondering if we just need to get more prescriptive about actually having a calendar that people can look at. Yeah, I think that is absolutely a worthwhile thing, and we should do it. I think that the odds of that solving the problem I'm concerned about are very low. Yeah, probably not. We will internally filter. So I think, again, it behooves us as the ambassadors between Envoy and DTP, to say, hey, there's this calendar. Here's your P0. But it's not even just the cloud providers. I mean, we have to be able to have that cadence and say, look, it's totally worthwhile. Right, I'm just saying, even here at Lyft, much to our company, one customer, we have the same problem, and so does everyone else, where no one wants to deal with the deprecations until the code is gone, right? Yeah, well, I think the fate of ID fault will take care of that, and we can make the old things fate of ID fault are quarter before. I actually think that that mechanical aspect is going to be mostly addressed, because I feel like having your binary not start up is a much better signal than getting an email. I think you're right. Absolutely. But I mean, there is absolutely no harm in doing better communication, and given that it's a quarter of cadence, it's not like a whole pun. It's a one-off script, and then maybe the script. Yeah. But again, the actually vibrations for people running their own envoys aren't I'm no longer concerned with, because fate of ID fault. The communication of cadence, I think, is just fly out worthwhile. And then my concern is, it's Harvey and I being in the middle of PCI design, where that they don't end up in conflict. Yep. OK, so getting back to the proposal, I guess we've had a few people join us. So kind of curious from the folks who've joined, haven't already read the proposal or have read it, what you would be interested in having me cover here. And just for context, this is the API versioning and the stable envoy API versioning policy, which is linked in the minutes. As you say, if no one raises questions, I mean, this meeting is being recorded. So it might be worth it just to give a five-minute summary. Sure, so there have been some concerns that the V2 APIs have not had the stability properties that are necessary for cloud operators or folks who are adopting Envoy's APIs outside of the Envoy code base. The most prominent example of this is GRPCLB, who committed to replacing that proprietary client because of load balancing control plane protocol with that of Envoy's. They're using XDS essentially. And so you can kind of see how we got here. We moved from V1 APIs, which are very sort of REST-formed JSON schema-based to something which was proto-based, which in theory had the promise of real backwards compatibility. But we were playing a bit loose and fast, even though we had an official policy for breaking changes. And we were not, you know, caretically and randomly turning things down. We were not maintaining wire compatibility over the long term. And there's a bunch of best practices over how to do this that have grown up, at places like Google, and offering like cloud APIs, which it was felt that it would be a good idea for us to better embrace if we want to. First of all, provide those who are offering services based on Envoy the stability that they need to be able to make product guarantees to customers. And second, for folks who are outside of Envoy to be able to actually rely on these APIs and where their release and deprecation cycles are completely decoupled from those of Envoy. So in response to that, I essentially have a proposal, which I've shared. There's both the long form, which is like 20-odd pages in a short form, a version of it, which is like a couple of pages. And they basically just describe how we can take what we have today with the V2 APIs and turn this into an sort of an ongoing evolution of the APIs. And the number of, you know, we were essentially going from a world of V2 XDS APIs to V3 LDS, V4 Bootstrap, V6, you know, leaf header map helper, V7 data formats. So basically a family of APIs, which really in reality, the Envoy APIs were all along, each with their own independent sort of major versions and release cadence. They basically will follow a single clock, but beyond that, they will evolve somewhat independently. The main thing that we can, like a really big win that we get from the Envoy communities, we get a clearance for our technical data in the APIs, be able to make our breaking changes without the fear of actually having any existing users affected. Only users of the new API versions that we make these breaking changes in the Alpha APIs are affected by changes. And we're essentially in this position where existing APIs will have no breaking changes. And so that's pretty great. The doc contains a lot of technicalities over how we're gonna deal with things like tooling and minor versions and version discovery and feature discovery and quirks around how to deal with clients, which may not support the full feature sets. I pretty don't want to go into too much detail here. I'm more interested in sort of hearing from folks that they have questions about any of these specific items. The goal is to maintain a release cycle for Envoy, which will still be quarterly. And just as it is today, and we're on top of that, we will also do API major version releases and spin outs once per year. The initial target for that is end of Q2. So yeah, I'll open up to any questions in this part. I think I have some concern mainly about like how much turn we will have when we improve, when we bumped like a version of the API. Like we did like in the exhaustive from V2 alpha to V2 and there's a lot turn to having like two version of API registered in server side and then Envoy side in control plane side and data plane side. And that go back and forth a couple times. Do we have like two or change to support that in both side? So on the Envoy side, the plan is definitely to make that as painless as possible by writing lots of tooling to automate things like, if you need to copy photos across with only a single field difference and build all the code to ingest that, that will be largely the plan is to automate that away with reflection tooling and so on. Now you are right that any management server at least during role arts and so on is gonna need to probably support two major versions realistically at least. And that is I understand and go is very verbose and cumbersome. I personally don't have plans to build out, go control plane or management server sort of tooling for that. I feel that problems should be solved. I think once we establish some good patterns for doing this in C++ for Envoy, it should be relatively easy to say, well Envoy works that way and they managed to reduce that code churn by using reflection and code generation. Let's just do the same and go and that should be a relatively straightforward exercise. You say that, but nothing is straightforward and go. Yeah, that's my impression too, so. Yeah, I mean, I personally, I didn't think, I can commit to doing a lot of the work or either myself or my team for stuff which is necessary to enable the Envoy side of this and for the data plane API photos. But the actual, although it's go work, I mean, frankly, like I'm having the unqualified to do the go work, I'm not a go developer. This is, I'm at home. It's a reasonable point though that most of our consumers for these photos are go. So I would say that we need to be cognizant of the pain that we might cause people. So maybe like as we start doing some technical discovery, we can loop in some folks and just try to get a feel for what it would mean on the go side. And to be honest, it might be enough to loop in the GRPC people because they have a vested interest in making it work well for go. So they actually, this might be the forcing function that we need that they can try to make some of this a little less painful. I agree. And also just keep in mind that the timelines over how this plays out are a bit different. So we need to really get the Envoy side of things, the client side of things happening and then an API repository like this quarter if we want to hit our original goals. But then we have like a whole year after that where V2 is still a fully supported API and we'll have V3 as well. And so that's like an entire year in which we can work on the server side of this to start supporting roll over and roll outs. So yeah, definitely I don't think we're gonna touch anything in the management side by the end of Q2. And then we'll probably use some combination of yeah, we'll reach out to folks, we'll hear from the community what that pain points are and ideally prioritize stuff. I mean, obviously we all have a vested interest in making the APIs work. And so if we are fine at the end of let's say in a year and a quarter that it's just so difficult for people to do rollouts that it's completely blocked for that that would be a failure situation. So we need to avoid that's happening by, basically just watching what goes on as people do these rollouts then probably refocusing our efforts and perhaps even on the control plane side as needed. Does that help, Lizard? Yeah. What else on the agenda? Well, just to let folks know that we're actually forming a working group for the university data plane API which has just had a CNCF technical committee approval. So we will be kicking this off shortly. I let me just link to the actual. Can we mix it? Sorry. What's your plan for general community? I mean, the plan is that there will be a few members from the Envoy community there. The idea is that we keep this committee fairly focused on link that we keep it reasonably small focus but we will be relaying information back to Envoy community and provide opportunities for other parties who have legitimate need to be members of this community to join. Like it's a generally basically it's, you know, it's I guess a semi-public forum is the way I would describe it. I think that there's gonna be a learning curve where I don't think we know how engaged people will wanna be. So I think we'll need to see. But I think right now we have commitments from all of the major clouds and effectively from the low balancer side, Envoy and GRPC. So I think initially it's still going to be Envoy focus but even there, there's value in making sure that the API's evolve in a way that meet the needs of all of the cloud providers. I think the goal is to invite, you know, would be to invite other load balancers like whether that be HAProxy, Nginx, Linkerdee, whatever. I don't know that they're gonna be interested but I think that's the goal. Yeah. And also I think you have things for like a client side, like mobile and even hardware load balancers. Yep. That's probably looking a bit further out. Yeah. So those are the main topics. There's someone wanna give us, I think it probably is worth raising and just have this in a chat about is what's going on with stability of CI. Yeah, I was actually going to raise that as our next topic of discussion also. Do you wanna give an update, Liza? Oh wait, for this. Just, I mean just in general our current CI status, like what's going on? What are some of the things that are in progress? So the CI stability. So I have a couple of the evaluation PR goes out which trying to speed up CI for a couple items. One is the RBE, which I expect to do the full building test within around 10 minutes for the whole CI. And the one is the coverage state speed up to switch from GCOVR to LVM coverage. And I think we can trigger the GCC to counter-search for the release build as well that will speed up the build much as well. Yeah, and then I guess the other thing that's ongoing is we're evaluating the Azure Pipelines, right? Yeah, Azure Pipeline separation is ongoing that's mainly from CNCF have a deal with Microsoft. And it works out well as of now for the Mac builds. So I'm kind of planning to switch from a circle for Mac probably this week or next week. I think one of the things that we need to do which maybe you and I, Liza, can take offline is I know that Yte got a new job which probably means he doesn't have as much time to work on repo kitty. And I do want to know specifically what APIs does Azure Pipelines have and do they have the APIs that we need to just move away from circle? Because if that's the case, I think per our... So I guess for people out there that don't know there's a permission issue effectively where we need to have a permission model that potentially allows trusted people to get faster builds and then there's a slow path for random PRs. So like if you're in the org or you're trusted in some way we could send your bill through the RBE and it would be very fast. And then if you come in via untrusted you would go through the slow path or something like that. But circle doesn't have any such permission model today. The good thing I think we can evaluate is Azure Pipeline have the GitHub comment support like slash AZP build by org members and that can trigger some permission. We can look into that as well. Okay, yeah, let's take that offline and I'm wondering actually if it would make sense to do like a short one page spec on what we're trying to do because I think optimally it would be really nice if this could be built into repo kitty, right? Because then it would all be like a unified system and if they have the APIs it shouldn't be that hard. Yeah. Okay, so maybe you and I can chat offline and let's because I think what I wanna do is that if it is too busy to work on the repo kitty changes I think we can have CNCF find us another contractor to work with today to do the changes. Sure. Cool. Sounds good. Okay, anything else? I think we're gonna start planning for EnvoyCon 2019 which will be co-located with KubeCon and San Diego in early November. So if you're interested in helping on the program committee let us know offline. Add a quick follow up on this one. How's the remote build executor working out? Have you found that to be useful? I saw your PR opened. So the RBE PR currently is using the Google's managed service to run the build. So basically it spawns out like 100 to 200 workers remotely and submit the artifacts needed by each worker and the data basil defined protocol between the basil and the server. I can send you a link for the RBE basil documentation later. I'm familiar with it. I was just wondering how it's been working off the Envoy code base. When I tried it six, eight months ago it was pretty sketchy. I had like a lot of problems with the general's and whatnot. Yeah, the plus plus two chain configuration is kind of a bit hard for RBE since we need to generate, pre-generate the cross tool files for the Docker image you're using for RBE. And I think six or three months ago we still have the repository rules that build the dependencies in shell script and a lot of stuff that doesn't work well with RBE. We already deprecated and removed those repository rules. We have external CCC make stuff and that helped a lot for the migration to RBE. Yeah, that makes sense now. Thanks. Thanks, Tammy. Bye. Bye. Bye.