 Good morning guys. We'll wait a few minutes for everyone else to turn up. You know what I'm saying? Let me rename myself here. There we go. Excellent. Hello, how are you? Oh, I'm fine. How are you? I'm doing quite well. I'm having a good morning. Hey, hey guys. Hey, turn on my video. There we go. You've gotten so much more rural, Andre. Yeah. Awesome. Cool. We'll do a few more minutes for folks to turn up. Joining the meeting is tricky sometimes. It's like, during the meeting, you need a different accounts, which like to switch accounts. No, join the meeting saying the problem. Join the meeting. Yes, I'll let you in. It's like, okay. I'm sorry, it's trying to be a little complicated. Like the logging in thing and just the various protections against Zoom bombing get to be a bit much. Well, what's funny is it's the same account every time I try and it fails the first one or two times and then it works. Oh, it starts to sound like my relationship with certain other web conferencing products that will remain nameless. There's one in particular that's infamous. I literally can't use it because every time I try and use it, it says you have to update your client. Click here to download. So there I download the client. And then I go back to log in with that client. It says you have to update your client. And so effectively it's never good enough. Like I've tried a dozen times in a row. And so when I get invitations with that web conferencing tool on it, I say, look, we're gonna have to use something else because quite literally they won't let me in. So... Yeah, it needs to be a recursive updater. For something. Yeah. Like if they can update you to the latest version, they should at least build something to do it recursively for you. Well, and the versioning stuff is hard. Like I know you having talked to some the various guys in Webex, like they've got all kinds of things depending on where various clients are on versioning. It's not an easy thing. And I've been involved with other software projects that are not even non-SAS cost software projects. I once was involved with one that had an entire director level organization that did nothing to keep track of the versions. Which is just madness, but madness happens. Yeah, this doesn't surprise me. I used to take, I used to do Windows updates. I would do it by starting a baseline of the entire file system, install the software, take a second baseline, diff them, deliver that as a package. It was very effective. It got all the registry in it. And everything else you could possibly think of that it could shove into the system. Nothing was left untouched. Yeah, yeah. All right, cool. So let me go ahead and share the issue PR screen. Cool. So we'll work really quickly right to left. So on the reviews in progress, I think, let me get back to those two. On the in progress column of stuff, we've got a few things that have come up as bugs actually in the model repo. And thank you all for being so responsive to people who raise those issues. One of the things from a community perspective gives us the space to do the fun refactoring work is the fact that we have working code that people can kick the tires on and that we're responsive as they kick those tires. So it's super important to continue to fix bugs over there and respond to issues. And you guys have been great about it. So we've got this one here, which I think you picked up Denise on the bulk registry NSE in the model repo. Oh, yep. I have looked at this issue and I cannot reproduce each of these kinds clusters. And I think the problem related to user setup, I'll try to set up user setup to reproduce this problem. Also, I have responded and shared it. No, you're responsive. This has been fantastic. I that that's absolutely done. I mean, it's really interesting. I had a team once and I remember pointing them to a particular set of research because the team was really struggling. They were much worse shape than any other team I've dealt with. I mean, we're in pretty good shape. They were really struggling. And one of the points that I made to them was that there's actually really good research that people care more about being heard than they care about having their problem solved. And so the fact that you were so responsive and I think you're probably right, it may have something to do with the setup, but it sounds like you've got a good back and forth there, so that's excellent. Cool. On the advanced OPA policies, I think I commented on this earlier today. So there's been a great conversation. Okay, maybe it wasn't on this issue. It may have been someplace else. I know there was probably a simple request. Ah, okay. Is the pull request referencing the issue so we can find it? No, we'll have to go look for the pull request. But we'll get there, I'm sure. We've got a long list still to go, but there's been some good movement forward on the OPA policy stuff. I'm actually super pleased by that. So Denise, on the industrial grade BL3, I know we've been talking a lot in the last week and you've been progressively refining things. Can you please open this issue? Sure. Oh, here I have added my status update for the week. You can scroll. Go ahead. Yes, currently we have merged for the last week, client in S chain elements. And also memory registry elements very close to merge. Yes, yes. Your comments, I'll fix them. We're chatting about that and whatnot. I think I may have misunderstood something in your PR. So we should definitely still talk. Also, I have updated spec document. You can scroll up, find the link to the document. And here my current status with this issue. Oh, this is a very nice picture. Thank you. Thank you. Oh, it is just abstract. And I appreciate the link to draw.io because quite frankly, like I have places I could use this in talking to people. And it's actually nice that you've shown three domains because it turns out that the two domains is easier to represent, but it actually really doesn't convey the fullness of what's going on. So. Yes, and also I have prepared a diagram for deploy well 3NC. It looks like it is logically right. So also here I'm planning to add some more use cases, but more of them is blocked because of we need to migrate most of stuff from Monoripo to SDK. Okay. Yeah, and logically right here, like this looks really complicated, but logically it is absolutely correct. The trick is a whole lot of things in here, I think when we get to the point and we don't have to do this out of the gate, where we introduce a caching chain element for the registry lookups, a lot of these calls will fall off because they'll hit the cache. So this is logically correct and it's the right place to start. But in practice, I think when we finally get to the finish line, we'll probably do something a little bit simpler, but this is definitely the right place to start. Cool. Yes, also here I have open question related to IP management. Yep. We have a few variants how to do it. And so, for example- So I think the project's been doing some preliminary work on some of that. And IPAM is super interesting because normally inter domain IPAM is really, really hard. But I think because of the way that network service mesh is doing this, we're going to get a situation that's substantially easier than it usually is. And the reason is because normally when you're doing inter domain IPAM, what you have is you have N domains and you're trying to reconcile N different IP domains with each other. So basically you've got to reconcile the power set of all the N domains. And that's a nasty factorial problem. In that service mesh, you only have to reconcile for the IPAM pairwise between the floating BL3 domain and each of the N domains. So instead of having an n factorial problem, what you have is a problem that is much, much, much smaller and your propensity to get lucky is going to be much, much, much greater. Yeah, that's already implemented. It's called the points to point IPAM. And another thing that we did as well is we separated out the journaling of what IP addresses are used into a separate space. And so, of course, the IPAM needs to manage its own list, but you can use the journal to replay and work out what IP addresses are assigned. Yeah, and the point to point IPAM in the journal is going to be super, super useful for the individual BL3 is managing the blocks that they're issued for the BL3 domain. I think the thing that we, unless I'm misremembering, I think the thing we still need to do on IPAM is the fact that a given BL3 domain is going to have, you're gonna have to have something somehow that is issuing blocks to the BL3 NSEs. Yeah, I've not implemented that and I have not implemented a recover from the journal yet. So the journal is created and it's actually storing to NATs, N-A-T-S. It is, so we need to have something that can also recover and replay the journal as well. So those are where the two gaps are. Yep, yep, and I think the thing is we basically also need to, for the most part, we need to try and keep things moving forward in simple steps, right? So for example, the journaling in that stuff, that's gonna be super crucial for getting to real industrial grade, but it's probably not the very first step, right? The very first step is probably the point-to-point IPAM with something simple to issue IP addresses out to it, right? And then you start bringing in the layers of additional stability into the system. Make sense? Oh, yep. Yep, cool. All right, excellent. And we should probably also then, once we started introducing things like NATs, we may end up having a conversation with Andre basically about whether or not VL3 needs an operator, which it may, right? Not directly a network service measures issue, but it may. So this is excellent work, I appreciate this. Cool. The WireGuard VPP plugin, or anything else on industrial grade VL3 before we move on? Nope, that's it. Cool. So the WireGuard VPP plugin, how is that going? Okay. Yeah, I continue work at it. I did a handshake initiation packet for server, the WireGuard server, and I can send it, but now I have trouble because I don't get a response from server. Sorry, you say you don't get the response back from the server. Do you mean that it's not showing up at your VPP plugin or the server is not sending it? I send it to, I run WireGuard server on WireGuard Go, and I send it to this handshake packet to it, and I can debug this packet. I can see the errors, and now I'm working at it. So question is, go WireGuard server, did it send the response for a handshake? Yes, it should send me a response. Is it sending the response though, or is it not? No, it is not because... Okay, so something wrong with a handshake packet. Yes, yes. I tried to find my error in this handshake. So... So have you ever worked with WireShark, the network analyzer? Yes, I use WireShark. I see that packet. I see the example for, from examples packets send it from original WireGuard, and I try to figure out where is an error. You don't find any difference between original handshake packet? There are... It is... So let me ask a better question. How do you know that the go WireGuard is not sending back a response packet? Because I use WireShark, and I see that it is not a response. Yes, yes, yes. And I use the debug in WireGuard Go. I run a debug mode, and I see that the response is not created. I'm not sure why. These are both fantastic answers. So there's a saying in networking. What's on the wire is true. And what we often mean by that is, and this is something that's counterintuitive when you're doing network level programming, because you're used to software where you go look at the debug logs, you throw the debugger at it, you see what the software is doing. But at the end of the day, if you don't see it on the wire, it's not true. And that's why I'm so pleased that you were using WireShark to make sure that you weren't seeing the packet sent back from the WireGuard Go server because that tells you really, really it didn't send it. All kinds of things can be squirrely about the debug messages that might deceive you. But if it's really not on the line on the wire, that it's really not on the wire. Because when you look at the places that this can break down, you sort of gone through and done a great job of debugging it. You sent the handshake packet. You used WireShark to see that it arrived. You used the debug logs in WireGuard Go to see that it arrived. You used the debug logs to see that you didn't send a response. You used WireGuard to see that you didn't send a response. So you know that your problem is somewhere in that packet, as opposed to if you had not been so diligent in your debugging, it's possible that the packet was being sent and you were losing it somewhere in VPP. So far you've done a good job of sort of drilling into the debugging, but you may want to sort of compare the packets. The other thing you might want to try playing with just for your own sanity is you said that you have an example packet of a handshake, correct? I apologize, I talked too fast. Yeah, but can you say it again? I don't get it. Yeah, yeah, yeah. He has example packet with WireShark, just go implementation for WireGuard. Okay, yeah, and so you'll compare with those. Yeah, comparing those is probably your best first bet and see how that goes. Okay, okay. So my experience is, Artem, this kind of problem, it just takes a little time to work through, right? And it sounds like you're doing the right things. So I'm sure that you will make progress, even though it may seem slightly frustrating right now. Cool, all right. So the command network service manager application and testing staff that you're working at right. It's still in progress. Yeah, at the moment I'm working into registry stuff to do a registration of NACs and some work need to be done for using a proper use of a call by KPI inside the SDK. I've almost prepared the pull request with some basic tests, I'll need some stuff to finish. So I'm excited about the call back stuff. Yeah, with security and client identification. So the, I'm also been poking pretty hard at the, I actually had a super funny experience yesterday because as I was debugging and getting things going again with the forder VPP agent, I sort of got to a place where it looked like the problem was in the SPIR stuff that you had updated. Turns out, no, the problem is not in the code that you updated, but for a while it did. When I finally tracked it down, your code was doing the right thing in all cases. So it's someplace else in my code. Cool. So the metric stuff I think for the moment is a little bit on pause. The SDK kernel migration, good. All metric stuff is already done. Well, so all the metric stuff that we've done so far is in, but we're not to a place where we can actually try and do anything with it. So if this is already done, let's go ahead and close this issue. I remember probably it was still a pull request about do we need a docker testing inside of SDK or not? Oh, yes, yes, yes, absolutely. I remember that. So I'm actually coming around a little bit on that. I'm coming around a little bit on that. I've been poking at making the whole docker thing simpler. And so in the course of making that simpler, I'm warming to the idea of potentially doing that there. So, but yeah, I definitely need to come back to that, but we can do that in a little bit. On the migrating the kernel forwarder to new style cross-connected network service. Is anyone currently working on that? I don't think we have any of the people who traditionally have been working on that on the call. No, it's momentarily not nobody from our side. Okay. And then the SRIOV stuff, I know that we're looking at that a little bit. Yvonne is not here for the visualized traffic. Where are we talked about default policy examples? I was super happy about this. This is actually coming together nicely. So yeah, this I'm pleased with how this is going. It does appear to be a good start on this. And then I think there's also a PR to be reviewed, but I unfortunately have not had a chance to look at the PR yet. All right, so on the to-do front, I'm actually gonna start at the bottom. So we had an installation failure reported. And I think it appears you've been interacting with this a bit. Andre? Yeah, but it was, I think, a few days ago. Yeah, we still have a health version three in pull request for a monorepo, and it was not merged. So... Is there anything holding it up? Actually, I don't remember why it is not merged actually. Okay. Is that something you could take a look at or get someone to take a look at? Yeah, probably we could check if it's still correct, we could merge it, and probably it will be resolved. It definitely is gonna need a little bit of rebasing. Yeah. So, cool. And then the, we had this one that you'd open Denise on a bug about, and this one I think is just you're doing an excellent job of documenting that we have sort of an intermittent leak of GoRoutines here that we should probably eventually track down. I don't think this one is super urgent, but I think it's important to keep track of it because it will be something that will need to be fixed. But I don't think it's blocking anything. Is it blocking anything at this time, Denise? Oh, sometimes this unit tests block CI drop, and I think we can fix it, fix this. Are you, yeah, I mean, fixing it would definitely be good. Are you able, when it blocks the CI job, are you able to rerun the CI job? Oh, I am able to run CI drop with force push. Ah, okay, that's annoying. So yeah, this does become a bigger problem because annoying your contributors is not good. And that is definitely annoying. So we also had this issue about being able to I need to remember not to use pound signs and comments when I don't mean them. We had this issue that this person was having trying to do a GoGet with just the K8s module where it was not actually working correctly. And it's not actually working correctly for reasons that are extremely well understood having to do with the multi-module nature of the current repo. GoGet really doesn't get along with multiple modules that cross refer to each other with local replaces in the same repo. I'm not sure, I haven't had a chance to look at what he's suggesting here and to see what Kubernetes may have done around it. But we definitely want to encourage our friends who are playing with Skydive. They've been really good to us and it would be good to encourage them and to help solve their problems. We also have this thing that was opened with the wrong test in secure intranet. I don't know, is there any, is there anyone who can take a look at this? Okay, he still looks like he wants to submit the PR himself. So let's leave into that. And then there's this that I think is legitimately a bug that's been raised by Albrecht. I can't pronounce his name there. Or is it, you can have ID. Effectively he's got a problem. And this one we really probably should get someone on the monorepo. Effectively his problem is that he's connecting to two network services. And it looks like he's getting two connections to the same network service instead of to the two distinct network services. So it's not falling over to the next one. So we really do probably want to take a look at what is going on there and get that fixed. Is there someone who we have picked this up? Oh, I can take a look. Can you assign this issue on me? Yep, absolutely. There we go. Cool, thank you very much. I appreciate it. Because you'll see from the comments that I managed to confuse myself very badly about this one. And I think the rest of it, we should probably go jump on the other call because it can't start without us. But I think we've gone through enough for today. We'll go jump on the community call. So I'll see you there. Yeah, see you. Thank you.