 Hey folks, apologies. Somehow the last meeting didn't get stopped properly and so we got delayed on this one. We someone willing to share the meeting minutes from this meeting. So I had trouble joining the meeting that said, another meeting is in progress. Yeah, I had the same issue. And so I'm not quite sure what went on there. I hope the left to see what their advice is for how to avoid that in the future. Cool. Oh, thank you Ashley for adding the meeting stuff if folks to go ahead and please add themselves to the attending list. That's very helpful. Cool. Should we get started. All the we're going to be good to want to share the meeting notes, but in the meantime, let's go ahead and get started. So welcome to the next number of services mesh working group. If you haven't added yourself yet, please add yourself to the attendee list. We have this particular call which occurs every Tuesday at 8am. We also have a call every other week. At 3am Pacific time. We also have. We also participate in the CNC of telecom user group. Which occurs every Monday at 8am and every every first Monday at 8am. Every third Monday at 3am Pacific. We also participate in the CNC of Sydney network, which has been rebooted and. Is occurs every first and third Thursday of every month at 11am Pacific. This particular group is, is a little bit interesting now because they, they now spend time vetting things for the CNCF. So if you're looking to get more involved with the CNCF, that's a fantastic way to do so. We also, we also have a few events coming up. So on March 18th, there was a go San Francisco meetup, which I will be talking about cloud native zero trust. And March 30th through April 2nd is cube con and cloud native con Europe. At the RAI in Amsterdam. Schedules are up and SM con is going to be on March 30th. They collocated with QCon. The schedule is now up. So if you would like to see what we will be talking about at NSM fun. Please go take a look. We have open networking and Ed some in North America. We're waiting for the schedules to be announced. That'll be April 20 or 21st. We have cube con and cloud native con China. The CFPs have already closed and the schedule will be announced in May. We have ONES in Europe. The CFPs close on June 7th. And that'll be an Antwerp in September 29th. In November 17th through 20th is the cube con, cloud native con North America, Boston. The CFPs will open in April 22nd. So the day after ONES and they will close in June 12th. Announcements will be in September 14th. And I don't think there's other events that we know about. So we have, again, just a reminder, we have a project page. The link is listed on the, on the meeting notes. And are there any other announcements that we want to do before we move on? I'm assuming not. So in that scenario, social media community team. Ashley, you have the floor. Hi everybody. So last week as far as social media went was pretty exciting. We reached our next milestone of officially having 700 followers on Twitter. So that was exciting to see. We gained 17 followers in the last week. We followed 15 in additional counts and tweeted and retweeted 26 posts. So as I mentioned that surpassed the 700 followers. As far as other things that were tweeted about and SMCon related, we got some registration reminders out there. The schedule announcement went out last night. And there are already a few people retweeting their, their schedules, their sessions. I'm sorry. So that's exciting to see as well. Outside of that, some general core reminders, video recaps, some CNCF news as far as promoting some events coming up. OSS in China later in the year. Some CNCF and LF events just again, letting everyone know that childcare is provided at those events. So just trying to get some additional attendees there for some parents. And other events that were promoted this last week was the introduction to network service mesh. Meetup that happened in Austin this week. So that was promoted a few times, a tweet that went out thanking the attendees as well as some retweets from others that had got some news out there that had attended that event. Some other general retweets, telecom user group meeting notes, CNCF blog and some VMware open source tweets. And then as far as LinkedIn, a bit of a slower week. We only gained two additional followers, but at least we are still growing there. So hopefully this next week we'll continue to follow. And I guess at this point we'll mainly focus on pushing sponsorship for NSMCon as we are a little over a month out and also promote the registration mostly outside of that. The meetup that's coming up in San Francisco later on in March. And then other sessions related to NSMCon and KubeCon. Sounds excellent. So really quick note to the folks who are speakers at NSMCon, you will find that your talk is actually individually linkable from the events page. So if you want to go and promote your talk, you can literally have a link that takes people directly to your talk on the events page. So it's very good stuff. Thank you very much, Ashley. This is awesome. It's super good to see us go over 700 followers. Oh, one of the things I didn't want to actually raise. So there was a really good blog post, Frederick. Do you have the link to it that came out of China about network service mesh that we may want to promote? Oh, that's a great idea. Let me go grab it. Yeah, let me see if I can find the link myself. It's been translated to English, which is super helpful because I otherwise would not know how amazing it is. Let me go ahead to the agenda. Yeah, it's an simplified Chinese and English. So I've put the link into the chat if you want to grab it from there, but it's super well done. Apparently there's some very thoughtful folks looking at what we're doing in China and writing really good things about it. Great. Good to know. I'll get that out there then. Thank you. Cool. Awesome. I'll also remind folks that you should feel free to add yourself to the agenda at any point. I added a few things, but I know there's more going on than what I've added. Back to you, Frederick. To be lose Frederick. Yeah, sorry. I was on mute. Well, and so. So with that, we have the. We have the main agenda that's coming up. So. And you want to talk about what's going on this weekend. I can talk about some things that I've been sort of heads down. So I'm not following as much what's going generally. So if other folks have things that encourage them to add them to the list as well in the agenda. So the sort of things that sort of got my attention were. The, uh, the command forwarder VP agent repo has opened. And there's an initial code drop PR. I'm still trying to figure out why the CI is not running on that repo. So that, that hasn't been merged yet. Cause I don't want to merge code that hasn't gone through the CI, but the, the initial code drop is super simple. It's like a 50 line main file effectively. Not a Docker file. Um, everything else is being pulled through from the SDKs. Um, so that's landed. Um, and then there's some improvements to mechanism testing. And that resulted in a bunch of generic things. So basically, uh, there's a link here to examples of the mechanism testing mechanisms are interesting because, uh, we ended up writing a bunch of them. Right. You're going to have, you know, we've got like four or five different kinds of mechanisms. And then you're going to rewrite those mechanism chain elements for each forwarder that supports them. So they're a thing that people are going to write a lot. Um, and so I, I put together a generic test suite to check that network chain elements for, um, Test suite for mechanisms to the right thing. Cause their contract is fairly simple, but straightforward, which is they should set the mechanism preference. Um, before they call the next chain element. And then when the next chain element returns, they should do whatever they're supposed to do, um, to, for whatever they're doing for the mechanism, but they've got to wait for the return. Cause that's where they get the complete connection. That's on the client side, then on the server side, that's where they get the complete connection. So they, they just do whatever they're going to do. Um, and in doing that, um, ended up building up some very generic machinery. So for example, um, There's a whole set of network chain elements that do simple checking. Right. So there's one to check to see if your chain returns an error. Um, because as it turns out, um, Sometimes when you test a chain, it gets wrapped up in other pieces. Um, and which is the case for mechanisms. And so you want to make sure you crisply understand that you've returned an error. Um, you know, check to see if the client propagates to your PC options. One of the things that's different between clients and servers is the extra, uh, variadic, uh, GRPC options, um, thing. And because it's very attic, meaning that if it's not there, it's fine. Um, I literally forget every single time. And so this is a simple thing that lets you write tests that simply check for that. Um, Checking the contents of context, um, after and on return from a chain element is important because it actually matters. Um, Particularly in the case of mechanisms, when you have actually, um, set up whatever things are in the context. Um, and then just generic things to check other aspects of the system, just to make sure, for example, that. If you have a network service requests after the chain, it's important to make sure that you have a chain element. Um, the key here is that, um, We wind up. If you want to check to make sure that a request contains something, you've got to have a chain element to follow to catch that. Um, likewise, if you care about things being in the return connection at certain points in the chain, you're testing. Um, You know, basically a chain element that'll inject an error or inject your GRPC options. Um, And then, um, I'm going to talk about some basic stuff here. Um, that generally speaking, and I think you made some points about this, Andre and the previous call, you want to try and keep your testing as simple as possible. If you've just got a chain element that. Um, Does a thing and returns the thing. Um, then you probably want to be checking just that and not wrapping it up in additional complexity. But sometimes it matters. Um, When in the chain of things, this happens. So for example, in the case of the mechanism stuff. Yeah. Yeah. More interesting is if we have more complex chain elements. Uh, with integration to a VP agent or to some library for Kernel, We also need some integration tests for this kind of chain elements. No, totally. Um, You know, I that that's that's one of the nice things about the way that VP agent SDK is put together is the only one that actually touches VP agent is the commit. Um, everything else is just setting up the config for that. Um, And then one of the other things that's kind of interesting. That's coming down to the VP Asian 3.01 is, um, The GRPC guys recently realized that it's hard to test if you pass a struct into everything. So the GRPC client connection, which is a struct has been replaced in most places with the GRPC client connection interface. Um, Ingenerated code, which is an interface, which means you can mock it. Um, So we can actually mock that out with unit tests as well. Um, Once we've made that transition. So there's a lot of good stuff going there. Um, But the baseline point is let's keep the testing as simple as we can. And it happens that this chain element stuff makes it super easy to test a lot of these things. Um, Any questions, comments, et cetera on that. Yeah. As a person who's reviewing a lot of this stuff, Um, Definitely, uh, I think we, it would help if we can get some examples for people playing out. There could be examples in the code just to show. Show off some of the, uh, more interesting areas. Um, I know there's a, there's a lot on the agenda to do though. So I know that that may not happen immediately, but just something we can add later on. Well, there's been something bouncing around the back of my head as I've been doing this. So my head went, I felt like my knees actually poked me and said, maybe we should have a style, a style guide. And I think what he was meaning by that was less like indent four spaces, but more, if you're doing something sort of like this, then this is sort of the pattern that's been established and this is why the pattern is. Um, And that might be super interesting and helpful. So, um, See who added the scene of conformance with NSM. I can go if you want or can wait. Yeah, is there anything else you want to talk about on that ad or? No, I think that's about it. I was basically providing some pointers to folks. But I think Andre is underlying point of keep your tests as simple as you can possibly keep them is a good one. But I think we may also want to consider in the case of things that are going to be done a lot like mechanisms writing a few test suites here and there. I don't want to get too crazy with that. If you're just doing a one off, it's probably not worth writing a test suite. But if you're doing something that's going to be things like that are going to be done a lot of places, then it may be worth it. Cool. Okay, what's in the you have a fork. Okay, so there is a new initiative on the in the CNCF, CNF conformance initiative. Basically, it's kind of an outgrowth of what the test there was, but basically checking CNF cloud native network functions. So within this community, it's really network services. And we're needing to have some examples for the conformance suite that come out of the box. There are some trivial examples just that that we're using now. But one of the things that we need to develop is a good story of a journey from a maybe a DNF world or just the non conforming or non compliant CNF into one that's more compliant and pretty, pretty sure that NSM would be a good, a good fit for as many different pieces, but the declarative portion. So just from our experience from taking DNFs from lots of hard coded subnet IP addresses and all that stuff. Is there maybe we can grab some low hanging fruit from maybe some examples that NSM has, where someone might be hard coding some things and when the addition of NSM makes it easier to have things be in a more declarative fashion. So, I'm trying to put that out there. If you guys can point me to something or if you want to help get that into the test bed. I'm sorry, the performance would that be great. Yeah, I think the thing that comes out to me is in this I think comes from sort of standard 12 factor out of thinking, which is an application should learn things from its environment. And things like IP addresses are literally things that you learn from your environment and NSM. You know, so basically things should learn from their environments and the system should be dynamic because dynamic things are resilient static things tend to be really, really unresilient. And then the other one that comes to mind is a scoping argument, which is if I statically part of why static things are not resilient is if I statically configure a bunch of stuff in a bunch of places. The scope over which I have to get that right is the global scope. And the global scope is really fragile. It's taking your fucked. Whereas, if you can localize things to a much smaller scope, then the scope over which you have to get them right is very, very limited. And so you're much less much more likely to get it right. Does that make sense. So, yeah, if there's anything where where you guys have examples from from maybe some type of network service that people were working on before Network Service Mesh and it's open source, and then after Network Service Mesh that story and the code. You can use as a sample. If we could, if you can point me in that direction, and it would be, you know, another outlet for Network Service Mesh. It's part of sample code for the performance. Don't have to answer now that, you know, something to think about. Definitely, we can try finding something so that's not very many examples before, but we can see about trying to find something that's that meets every requirement. Well, I mean, even the, what is it the Sarah, you know, example, I don't know if we're still using that but the we make it to where it's easier to just declare what you want for the security context, whereas before people were trying to maybe car code ask for specific ports to be open and all that stuff. Yeah, because I, my, from my recollection events time period, there was a lot of of static like static configuration. A lot of a lot of work towards like trainers to to specify things directly. I recall doing demos of this type of stuff back around 2016 2017 ish. I don't know if I have any available anymore, but I'll take a look. We could use something similar like it was an evpn example that I used to that I used to to use. And it was a real pain to try to get this thing set up and working properly because of all of the manual things have to be configured in order to try to get the to Kubernetes cluster networks to to work with each other. And so I'll see if I can that's amazing. Yeah. Worst case scenario, I could try reproducing. I could try reproducing it but it's, I'll see what I can find. Okay, great. Thanks. Anything else we want to discuss on this particular on this particular topic or I mean that's that's all I had if you guys want to, I mean, the, the code base is out there as far as what the different tests that we want, and it's still very much open. So the declarative part is definitely, I mean, there's so many parts that and if I'm going to be part of, as far as best practices, I know this community has been like certain opinions on what would be best practice what is cloud native and the thing that has always jumped out to me is the declarative portion. I think that you should have a strong voice on on that type of thing. So, um, I mean the one thing I would actually suggest being very careful about with just just talking about declarative is you can declare all kinds of things that are stupid. I can, for example, construct a system where I have to declare all the fine minutiae of every little detail of the infrastructure that I'm in what I'm working in. And it's still declarative, but it's also not a terribly smart approach to the problem. So I think declarative is one piece of it. But I think also like, I, it has a sense of immutable infrastructure. I think it's either IPAM to be part of the infrastructure, right? You can learn about it. You can ask for things from it, but you don't get to dictate it from the outside. Does that make sense? Okay. Yeah, sure. So, go ahead. No, I mean, I don't agree. Okay, go ahead. So, one of the things that actually is being discussed within the CNF conformance was the fact that these small bits of functionality that we call CNFs are supposed to not store state, not have state at all. Meaning that things like, you know, IPAM can and I believe should be managed outside. I mean, from the example that I'm trying to construct with NSM and actually I'm working on something you know, which I hope I'll be able to announce in the weeks to follow. But from the example that I look at from the practical way of applying NSM, I see a lot of problems exactly regarding to IPAM. We have seen this also with Ericsson, with the bridge and then load balancer domain. I mean, I think that the way that we approach IPAM today is not the best suited to whatever we try to do. That's what I think. Oh, so in what sense? Because I think you and I would agree that basically something has to manage the IPAM. What I was getting at is an IPAM story that involves putting an environment variable into your pod, assigning it its IPAM in the world. And therefore that's statically defined, which means all your pod specs everywhere have to come into proper agreement. That was what I was railing against. Does that make sense? Yes, but to me, like no one in the world should be writing pods their own. I mean, there should be, you know, top level orchestration that actually distributes these IP addresses and figures out for you. Like having the inline IPAM, you know, all the DHCP and stuff. I mean, you don't get the top level picture. It's very hard to deduct it from the running infrastructure. That's my argument here. Okay. No, I mean, this is part of why Network Service Mesh, you know, things like the source and desktop IP coming back in for Network Service Mesh, those are purely optional. If you want to manage in a different way, that's part of why those parameters are optional. Number one, because you may, you know, number one, because you may not actually want them, right? They're definitely network services for which you wouldn't want IPAM at all. But number two, you may get them from someplace else. Yeah, but even if you get them from somewhere else, maybe you want to, like, introduce them from the service. Otherwise, your clients need to be privileged. Otherwise, your clients need to be what? I'm sorry? Yeah, no, I mean, this is actually a good point. Somebody has to go set those, you know, and that somebody has to be privileged. Network Service Mesh handles that because the forwarder, which has to be privileged to insert things like kernel interfaces anyway, goes and does that. I mean, you're talking about user space CNS and I would expect most CNS to be user space in some fashion, and to be using some kind of a user space interface because kernel interfaces are very slow. But if you're living purely in user space, then you don't need to be privileged to do those things. Does that make sense? It does to me. I mean, just as much as we expect on the enterprise side, almost all enterprise applications are going to ask for a kernel interface, because they're written on currently stuff, but they're also not going to be super smart about their own IPAM. On the CNF side of the house, we would expect almost everything on CNF to be using something faster than a kernel interface to move packets around, whether that's SRIOV or MIF or V host or whatever. Does that match what you're expecting to see? You may be, it's me. I agree with you. Okay, good. I mean, just because I have a notion of the world doesn't mean my notion of the world is right, by the way, which is why I ask questions like, does that make sense? Also, I have a bit of like a problem with this, if I can discuss this now, because usually NSM mostly is virtual wires and only cares like point-to-point connections. So in a way, an IP address is completely useless. You could be on-link routing instead if you want to do that, if you see what I mean. Well, it's not so much the NSM itself is point-to-point connections, but we presume that there is someone on the other side who may be providing a network service that's point-to-multipoint, and that's the someone who knows what the proper IPAM is there. Does that make sense? Yeah, that makes sense. But what I mean with this is that in this protocol between the service and the client, sometimes I can see a need for more data than just the source and the desktop IP. I would like to put a big address as well, to have the form of a virtual IP. Like when we did this load balancer. Oh, okay. I'm missing the possibility of transferring more data over this protocol. And this data NSM doesn't really care about, but it's important to the client. So thank you for bringing this up. This actually brings up an important point that we should probably go just literally just go fix right now, which is like literally as soon as this call ends. So when we started out with, so what I think you're saying is that there are other things you might want to put in the connection context. Is that correct? Yeah. Okay. And when we started out with the connection context, it literally started out as just a map, a string string map. And we did this and what we came to realize over time was that this was a little bit loose given that much of what we were passing was relatively simple and structured and so you got things like IP context, Ethernet context, DNS context. And we did that we ripped out the string map, this, the map string map, the map string string. The truth of the matter, the truth of the matter is, you still need some kind of a generic context that gives you that map support, because even if you do eventually get to a place where you converged on all the things someone might conveniently want to pass there. Number one, you want people to be able to innovate without having to come and get something upstream. Right. So you, giving your example if you wanted to provide a bit, right. And I don't know the full depth of what you want to do but just on the face of it that sounds like an eminently reasonable thing to want. I don't know whether or when or if that would ever make its way up to being like a bit context that might not make any sense, or it might, I don't know. But what I do know is there ought to be in the connection context some sort of a map that you can use for that kind of thing. Does that make sense. Yeah, I added them up in my hackery but I never like I never. I just go push a patch up that adds that to the API, because it's something that I keep forgetting to go do, but it just comes up every few months and it should be done. So please feel free to go push that on. If you want but it's, I mean it's kind of, I just put the map into the struct of the context struct and then. Uh huh. And then you have to re-generate the DRPC but you know that. But I mean that that's a perfectly valid thing to want to go out to the API. Right so and you know if you go push that PR. I'd love to get that because you're absolutely right we're never going to be able to see all the things people are going to want here. Right. And all we can do is codify the things that we think are going to be super common. Like the IP context and the internet context and the DNS context. And then allow a safety valve so people can actually get their work done, which is what you've done in your hack and we'd love to see a PR for. So that and you know I yeah that that would be super super helpful. Thank you so much I do appreciate it and thank you for bringing it up. Right. If people don't bring these things up. Number one, even if they're things that like I said that's been in the back of my mind every, you know, for months now but it never quite got done. Or sometimes you bring up things that haven't occurred to people. And eventually I'm super curious to see what you're doing with that the parameter. Interesting as well. But I also put like a hack into the forwarding plane to be able to generate like extra interfaces with a bit. But that I don't want to push up because I don't think you want to have it. I certainly like to at least understand what you're doing there just had a curiosity. Maybe some kind of plug in concept to the data plane to be able to like catch stuff in the map. So that's certainly definitely an interesting discussion. You know that that's definitely interesting discussion and this gets to be. So I mean, and one of the things that we're getting more flexible with the forwarders is, you know, there's no reason you can't bring your own for that supports your own stuff. For example, I mean, generally speaking the one piece of advice I would offer you is that it logically speaking. Anything point to point is probably best thought of as a network service. Now implementation wise, where you put that network service from the purpose from the purposes of optimization. That may vary right, but the logical and the implementation are useful to separate because when you when you have clean cognitive breakdown of what you're doing and where then the where you put it to get implementation optimizations is unlikely to screw you up. But I'm sure you've seen this to get it again because we're both a networking and have been forever, where you muddy the conceptual end of it, and suddenly you've got like 15,000 features in one place where they're driving you knots. And when you can't pull them apart, right, so just just for the sake of argument, it sounds like you're using the script to do a point to multi point thing and you've hacked the point to multi point thing into your forwarder. Right. Yeah, what I did. What I did was that enabled this load balancing in VPP and it used the GRE tunnel, you know. Yeah, this is this is this is a fantastic thing right. And so, you know, my my my counsel to you would be as long as you consider continue to think of that as a load balancer network service which you happen to have optimized by sticking it into the forwarder so you didn't have to jump through a second VPP. I think you're going to find your life extremely pleasant. But this is also something that I thought a bit about most of the stuff we're doing in the service. I mean, if you go end to end. And to get performance, the application would need to do a poll mode load for something like this. I mean, application and action and the real service. But the local stuff we're doing are just infrastructural things. And it would be nice to have an API down to the data plane to actually delegate those kind of stuff like remapping or be long tag or something like this. That's a fantastic thing to discuss. I mean one of the things that you're probably seeing in some of the work with the chain elements is if you look at the chain elements, the difference between anything that you can break up into multiple steps, you can collapse into a single step. Yeah. And that's that's very much by design. Because there are there are lots of places where you care more about flexibility in the performance, a lot of them, but there's definitely places you care way more about performance than flexibility. It makes a lot of sense and also I like the abstraction with the virtual virus and services because it's a very strong abstraction and you can model almost anything with this. I get the problem when we implement this we would get like native EPPs running in each container. I would like to push the data I mean push the implementation down to the data plane. Yep. No, it sounds like you're doing a reasonable thing you're thinking about it in a reasonable way. You know, and you have some additional thoughts that I'd like to understand better about how to make this a bit more flexible. So, and so we can definitely carry on some of that conversation about how to make things more flexible because I like flexible. Yeah, that that's that the net net is a PR to add a map to the context to use for the sort of thing is probably an unalloyed good. So, yeah, I was more like with the new with the new approach to implementing the forwarders probably the best thing is to just, you know, create the forwarder that suits your needs. It's effectively just another endpoint. So, just, I guess that that would be the best the best approach I was to why we were talking I was trying to figure out if we can do some things like whatever the CNIs were doing with, you know, kind of chaining different forwarders but I guess that's just going to get too complicated at least at this stage at least until we don't move. Maybe at some point we can think about this but probably not today. Yeah, I mean, in some sense, like you still you can change things in some sense. Yes. So a whole of network service measures, any things that speak the network service machine for them to collaborate, sometimes across, you know, space and time and sometimes complete mutual ignorance, in order to do the thing that you want. But I'm particularly heartened under is that you sort of arrived at this solution on your own, because it is almost exactly what I would have suggested. Which means that there's some intellectual coherence flowing through the system where it leads to the right sort of thoughts, which is good. With the new forwarder architecture, is that done? No, it's getting there. I just pushed up the command forwarder VPPA and stuff so there's definitely a code you can go look at. I'm going to go take a look at where that's going. But it's, we just got into the border command bit, and there are some other pieces still coming together. But the idea of how the pieces fit together. And the point that you made about it would be nice if you could go and insert something with additional logic into it. So what you're going to find is basically everything is a chain of small chain elements that do a piece of work. So if you were to go take a look at the command forwarder you'd see all it's doing is calling something that returns a cross-connect network service. And if you chase back to the SDK VPPA agent, you would discover all that's doing is putting together a list of things. So if you wanted to then say, well, I want to build my own forwarder, you go and write your own chain element that does the thing you wanted to do with fits, and you just build a slightly different chain. That sounds good. Because then we can do it. Exactly. That's exactly the idea. And part of it is to enable experimentation because quick, frankly, there are going to be all kinds of bespoke things that people want to do. Because the world is, you know, the networking world is incredibly diverse. And we're going to try and hold the reference implementation to the stuff to sort of what you might call the sort of optimized for the average base. But we want to make it as easy as possible for people to solve the other kinds of problems they encounter in their life. And generally speaking, you want to minimize the amount of code that they have to fork. And if the only code you have to fork is the assembly of a chain, then that's going to make your life much nicer. So, cool. Great. Okay, I want to bring one more topic. It's not on the list. I probably should have put it, but it's just a quick, quick one. And I think Watson is still here. So he probably have heard this. I don't want to put names here, but there was this discussion in I believe in the telecom user group. And it looks like folks are trying to pin for the telecom usage, of course, try to pin the specific networking solution. And we have a good support within certain communities. But effectively, when NSM was brought on the table, the pushback response was it is not production ready yet. I'm just saying this, not like, you know, panic, panic, panic, we should release whatever, whatever. It's mostly like we as a community should be aware that people are expecting more of us. You know, it's easier to say the implement, but you know, No, I think that's an important point to get front and center. And I appreciate you bringing it up. I don't have anything to comment. Just like, yeah, let's move forward to the best. Yeah, the, the unfortunate part is a lot of people come flight, like they'll compare NSM to, to multis or Dan M or, or genie or so on. And the problem that that ends up happening when you make those comparisons is that people will just look at the, at the CNI multiplexer and say, okay, we'll just thing inserts an interface into into a pod. And they compare that to NSM because that's with the context that they're only looking at that and then they say, oh, and it's not ready. But what they don't see is that NSM is actually solving a different problem and that we've had that specific feature for well over a year in in production quality, I dare say. But the problem is that NSM itself has is solving a bigger problem of like, how do we solve the, the changing aspect, how do we get multi organizational inter domain working, how do we get, how do we get these various vendors, like SDNs to cooperate with each other, and do it in a secure way. And so, and that's before you even look at things like, okay, well how do we line in a PNF or VNF into the chain as well. And so the, the scope of what, what we're doing is, is very different from others but people have inflated our project with, with basically a CNI multiplexer in terms of in terms of scope. And that's part of why we're getting this, this actions. And they are right, we do need to get a production version out there. But it's, but there's there's a little bit something, there's a little bit more that's, it's a little bit bigger that we need to try to, that we need to try to tackle. That's, that's clear. I'm saying like, if you are unbiased and you come to the Kubernetes landscape, and try to find the networking solution that will treat your needs being catelco. You would find certain projects, whatever they are, are they CNI based or whatever. It doesn't, it doesn't matter. I mean, you're just looking at what's available. Even if we manage to, you know, gather, you know, attention, which I think is good. And, you know, we do a lot of, you know, publicizing, you know, the Twitter, etc, etc. And it obviously, you know, helps people and hear about us want to look at us, you know, we get this nice blog posts. I think that we're doing great on that side, but on the other side, people are expecting, okay, and the same is greatest idea. I like it. I think it's going to solve my problems. I want to use it today. And I don't know, maybe we just have to try to figure out what to when is today, you know, that's just, you know, Yeah, I agree with that. But, um, yeah, we should be, we should be relatively close, especially when you compare to where we were at like the last, the last few condoms and before so. But we definitely need to work out with that. What those steps are to get something to get something out and I would argue the most important things that we needed to finish adding in where things related towards the inner domain. And specifically around the security and, and permissions use case and we now have some stuff in the repo that still needs a little bit more work, but it's very, very close to being done. And then at the very minimum, we should be able to release a version that works with the kernel with kernel interfaces, and, and wire things together in that respect, and we can spend more time with subsequent releases trying to push out, like, how do you get the SROV work that the community isn't working on and how do you get the things related towards, towards additional monitoring or additional, or how do you get additional things with like, with, with other, like VFs or so on. So, so I think there's, there's some interesting stuff in that respect that that we can push towards but yeah, I think we should try to carve, we should probably try to carve out sooner than later what the parts that are stable and build a story around that as an initial release. But that's my opinion towards it. Okay, maybe I don't know, I mean, I don't have any particular plan or answer today is just like, I think this is a healthy discussion I think your distinction Frederick is actually a good one because essentially the, the, the CNI multiplexers saw such a narrow corner of the problem space that, you know, it's essentially saying we're going to settle on the problem, we're not even sure we can use to solve the broader problem but because there's a thing there. We're going to settle on it. Because it literally it's not clear that there is a generic broader solution for problem I mean you can always go get it bespoke preventer thing, but I think that's what they're trying to avoid. Exactly. And that's the, and that's part of the part of the challenges like how do we, and the people who are in the in the calls for the most part. I don't think that there's like any malicious intent or anything like that like they're they're looking at it from the context of, as best as as best as they can and the stories that they're hearing our NSM is not ready. We have multis and Dan and and so on and those are ready. And so, and the thing that we want from NSM is multiple interfaces so why would we want anything. Why would we want anything more. And then when they started visiting the harder problems, because they've already pre selected a solution they're not thinking of NSM as a, as a thing. Those harder problems really are where I think, ultimately, life gets interesting, because effectively at a very fundamental level, I don't think you can solve those harder problems with a solution where everything is set in stone at pod start up time. And, and so I think that fundamentally makes any solution that is constructed that way a dead end at the end of the day. I will agree with you, but the thing that I get all the time is show me those examples where there is actually are needed. Like, which is the example that actually can demonstrate how NSM utilizes this dynamism this flexibility that it brings every CNF ever. I'd like to demo those examples. Every CNF ever because here's the thing. If you, if you, here's the very fundamental thing. If I have a CNF, right. And there are other things that need to connect to that CNF. If I want to do any kind of chaining at all, then I have, I need to have some kind of a connection between CNF one and CNF two. Now, those two CNFs, one of them, no matter what you do one of them is already going to have completed a startup process when the other comes up. The CNF completed startup process will not be able to change to accommodate the addition of a new client in any reasonable sort of way. Number one, but number two. If you have CNF one and CNF two and they're supposed to be connected to each other and CNF two crashes dies or otherwise goes away CNF one is now marooned. Not only is it no longer receiving the service that it needs, but it literally cannot get the service that it needs from anyone else. That means everything about its networking was set in stone at the beginning of time. That's not how services working Kubernetes. Services are doing a very different thing. Yeah, this is our carrying a payload over a Kubernetes intra cluster network. There's this fundamental presumption that every pod can reach every other pod. You want to reach CNF to reach every other CNF. Wow, I mean, you still have policies and things. Right, right. But do you truly want every CNF to be able to reach every other CNF? And do you want to pay the performance penalty? Okay, I don't think that we should try to resolve all the problems. But the really fundamental thing is that the better analogy is not to services. The better analogy is saying what if every TCP connection that you needed had to be established in your pod spec at the time your pod started up. That's the analogy. Okay, maybe I mean, I don't know, maybe. And if you're a server every client TCP connection for every client also had to be. That's the analogy to the situation and just like anybody would obviously see that requiring static at your static and fixed in time at pod startup time TCP connections is insane. It's the same thing. So, we're pretty much out of time so people have to continue this discussion at a later time. So, there's surely a there's surely a problem needs to be solved. And, and I think we're on track regardless of the difference of opinion they think we're on track to, to solve such a such a range of problems. Is there anything else in your last in the comments that or announcements that anyone else has cool with that. Thank you, everyone and we will see you all next week at the same time. Take care. See you. Bye.