 Good morning. How are you? Hey, I'm fine. How are you? Good. Just a second. We've used the air buds. I keep forgetting it's echo-y here now. Yeah, okay. Hopefully that's better now. Yep. So apologies guys for having been missing the last few weeks. There's been, as you've noticed, there's a lot of crazy in the world and I've been pulled in a lot of odd directions because of it. So in the normal course of affairs, MSM is still my main focus, but I've been at Cisco for 20 years. I have my hands on a lot of weird things. Hi everyone. Good day. It's welcome. I can see if Frederick is coming. Hey, Nikolai. Hello. Yeah, Zoom has funny visual background functionality. Yeah, I know. It's great fun and there's been a lot of stuff lately with everyone switching to Zoom. Apparently, you can even use MPEG videos as your backgrounds. They move around and everything. Cool. All right. Let me, that's right. I was going to go ping Frederick. One moment. Welcome Nikolai. Yep. Hello. Yeah, I was just apologizing to the guys for having been missing the last couple of weeks. It turns out that even though MSM is in the normal course of affairs, my primary focus, that with the world in a strange state, all kinds of weird and needful things have been popping up on my agenda. Some of them with very short notice. All right. So I think, Frederick, should we go ahead and jump in then? Let's see. So in my absence, have we been reviewing the PR issue board or how we've been working through the meetings? Mostly review it, but a bit. Not so many. So we mostly discuss the current status the guys working on. Okay. And that's mostly the same. Yeah. But that's completely fine with me. Do you, however you guys have been running it, feel free to continue and I'll just skip it for the moment. Boards is mostly up to date. There is main four topics for us in the board at the moment. It's advanced OPA policies, industrial grade, inter-domain, ordnance, and we're guard VPP plugin, which we decided to add to VPP. And is the care-related stuff like Sam Dennis manager. Awesome. So who's working on the where guard VPP plugin? Our new engineer, Artyom, is working on a var guard VPP plugin. He already has some prototype based on the IPsec and started to work with encryption for a var guard. So we plan to use it as a template. He already has some examples with two IPsec implementations working with two VPP containers, thinking each other and plan to start rework it to a var guard plugin. Okay. That's cool. When I went and I asked the VPP folks like what they would suggest cribbing from, their suggestion was to go look at the IPsec stuff and apparently the underlying IP and IP things that it's doing. So one hint I'll give you Artyom, which is that generally speaking VPP when people are working on VPP there's sort of two cycles to it. The first one is that you write a plugin that works and it turns out that's not too terribly hard usually. And then you make it go fast. You basically optimize it and it turns out that is much more interesting. Optimizing the plugin. Because there's a lot of you, I don't know if you've noticed in your research, but there's a lot of dark magic in VPP where you know it's doing a lot of work to make sure that it stays in cache. It's doing a lot of work to make sure that anything it needs is fetched into cache by the time it needs it. It's doing a lot of work to even prime the pipeline so that it gets the pipeline behavior that it wants or that it's because it turns out that we're smarter about the pipeline and the pipeline is generically, which is not hard. It's got to be generic. They're even doing work to parallelize instructions so that multiple instructions per CPU cycle are going through. So yep. Cool. So don't hesitate to reach out Artyom if you need anything. I am not the deepest expert on VPP, but I do know a lot about it. And I've got a lot of friends in that community. I'm the technical steering committee chair over there. So cool. What else? Yeah, I still have an open question for the docker test like functionality for VPP agent SDK. So I have a pull request for it. It just adds the basic stuff to set up a VPP agent in a docker and it's actually working blazing fast for about two or three seconds per test. So potentially we could have a test for our chain elements against the real VPP agent in a SDK. So I have a chance that you can take a look. Yeah, I I think I need to go back and rethink that my initial thinking has been I didn't want to bring in something that complicated into SDK VPP agent, but given that you've got it moving very fast, it may be worth it. It's very well worth doing. And then you can actually make sure that the configuration that you expect is turning up on VPP, which is probably worthwhile as well because you can run into a little weird little things like I ran into one just yesterday, where apparently they've deprecated post interface for AF packet in favor of a new parameter. They call Linux interface and it turns out it wasn't quite the same behavior. So I had to switch back. So yeah, it may be worth going and doing that check. So I'll go take another closer look at that and see because that would also, as you said, give us a lot of space to work with. Yeah, at the moment, it's more like a draft because it uses a direct VPP interface to configure example, but I think we should use our chain elements to do all the configurations for our testing. So it will check most of the chain elements. Yeah, I mean, you could absolutely set that up. You could absolutely set that up. And the really interesting thing is it turns out that the VPP agent, you can relatively easily ask VPP agent to reset the VPP instance. So, you know, you could basically start up one and run a bunch of tests against it. Yeah, yeah, yeah, yeah. And because it's just a pure doc app, it starts very fast. Yeah, yeah. I mean, the other one that I want to sort of take a look at is, and you probably saw this in some of the work I'm doing with the command forwarder, I've been experimenting basically with rolling a single docker file that will optionally produce a test stage that we can run. And the ability to pull in locally built binaries and tests for that. And that seems to be working relatively well. So that may be another approach. Yeah, it ends up looking a tiny bit involved, but effectively you've got a single docker file. At the end of the day, it produces the runtime you want. But if you use a different target flag, it'll spit out a container that will run your tests or a container that will run your tests. It'll spit out a container that will run your tests or a container that will let you debug or whatever. Yeah, it actually sounds very good. Yeah, that was the hope. So yeah, I'll definitely take another look at that. I know I've also been looking at the stuff that you did for callback, and that's actually starting to look very attractive to me. I still want to try and get the usability to be as close as humanly possible to the normal patterns that people use with GRPC. And you've sort of gone part of the way there. Then I made some comments on the PR about sort of some musings in that direction. The alternative implementation for it could be, but I've not checked how hard it will be to implement, is to provide it a different way. Is to use a more low level and just passing bytes to and from. Yeah, that was kind of like, that was one of the thoughts that occurred to me as I was sort of poking out of her usability point of view, which is if you get at it at the level of the connection, then effectively, you could potentially look at whether or not you could do this with a essentially, if you get to the level of the connection, you could look at doing it with a GRPC option to, basically with a GRPC option that would use an alternate dialer. You have the dialer just being there. And that would end up being extremely smooth, because then you just do exactly the pattern you're used to, but you pass in this option. And if you have the option basically, if you have the option basically keying off of some kind of a URL, but if it's not of the form of a callback URL, just no ops itself. Yeah, the problem actually, I have not yet found a way how to send the bytes to real GRPC server on a client side. So on a server side, it's easy, but on a client side, without changing for a GRPC packet itself, it's very hard. So just by using the public API available at the moment. So probably some changes to the GRPC packet itself will break right. Ah, that may be true, because essentially with the connection, you know, the connection that's presuming it's getting a stream of bytes. Yep, and that stream of bytes includes things like HTTP headers and TLS and all of that shit. So, yeah. Probably possible, but I can dig into this area a bit more. Okay. The other thing you might want to look into, maybe, you might want to look into some of the stuff going lower level. You do get things like WebSocket implementations, which essentially are running down the level of HTTP-ish stuff. Yeah, yeah, I will check. Probably we'll find some solution in this area. It'll be more simple. The performance will be better, I think, because packets will be simpler. Yep. Anyway, but I like that approach to the problem very much. If we can get there, as long as we can keep it relatively simple and transparent. Okay, cool. So other stuff that folks are working on. Who's working on the advanced OPA policies? I think Dmitry, but he's not in a call today, to take the PTO today. Okay. He does have something in the area of industrial grading through the main. As I've seen in the example, he's writing something from the ad. Yeah, I've been chatting a little bit with Denise. I've been trying to put together a quick document for him, describing the pieces, because it's one of those things that's not terribly complicated if you're an networking person. But if you're not a networking person, it's not at all obvious where to start. So yeah. And this is one of the things that I've learned over the years. It's really funny when you are a networking person who also writes software. Because there was a whole SDN revolution that happened with software-defined networking. Where suddenly you were trying to program the network. And so the first thing you discovered was that networking people weren't usually very good at programming, and programming people usually didn't understand what was going on in the network. And so the first thing we tried to do was to put two of them together. And that sometimes produced very interesting results, many of them not good. Because the networking people would try and persuade the software people to replicate all the things that they had done in the network before, which didn't necessarily map to the limitations that had been lifted for the programming. So you ended up doing things much harder, the much harder way that you needed to. So, cool. All right. And then other stuff that folks want to talk about. I think a new area we started to work just started as an IOL. We previously had in V2 and V1 of Monorepo some stuff already. So we just started to work into this direction. If you have any pointers additional to this, it will be very good. Okay. I'll try and dig up some of the existing conversations on that. Because a lot of this has been talked about back and forth. And the good news is that we've got a lot of work that Prem is done. I would definitely recommend reaching out to Prem and trying to collaborate with him. Or sorry, not Prem, Xemic, and collaborate with him. Because he's actually been thinking a lot here. And also you may want to reach out to Radislav. Nikolaj, do you happen to know if Radislav is still cooking in this area? Yep, he is. Okay. Am I muted? No. Nope, I hear you. Yeah, he's, yes, he's reached out to him. He'll be joining the call after, so yeah. Fantastic. Hi, Fredrik. Hey, hello. So we solved all the problems now, Fredrik. It's okay, you can go home now. Okay, well, it's handy, I'm already home. Yeah, mostly all from the homes. I don't know, I've always been from home. So, yeah, it's remarkable how similar to my normal life all of this is. Although the sort of general hysteria. Yeah, good idea is to build a home near the home to work without being disturbed. This is definitely true. I have heard a lot of that going on. I've sometimes joked that the real, we're missing the real health crisis here, which is the mental health crisis for all the parents stuck home with their children for weeks on end. Okay. Cool. I happen to have a home next to my home. It's a tiny one, you know, it's like two rooms, but it's there. Good. Mostly I think we discussed most of the stuff. You can like, do you have anything, or Fredrik? Not at the moment. We'll discuss it on public requests. And I have added a few comments, mostly minor. To the CMD forwarder with PPA agent. The most interesting, I think it's about how it will register to the MS manager. Yep. And I think I responded to those comments at this point. So you had a couple about with GRPC with block. And basically, I've been trying to minimize the sort of latency of everything in the system. And I gave over optimized that a couple of places in that regard. So, okay. Also, forward pull request on the MS manager. Do you plan to commit anything inside? Or I could just take the same patterns from the forwarder. Oh yeah. Please just go ahead and lift whatever from there. And I can go ahead and close out that PR. Mostly I was pushing that up. I was mostly pushing that stuff up sort of as an example. The one thing that I think you'll find in there, and we probably want to go chase this PR down, is that I've got the network service manager chain. I've got a PR for that in the SDK repo. And so we probably want to work from there with that pattern from there, because we probably want to keep the binary, the pieces in the command repo, just to the stuff necessary to get the binary up and going. Yep. Yep. Yep. The plus is that it should be a few things. The binary, the manager PC socket, probably it could be one. If you will go with the client callback approach, or it could be the same way as at the moment with multiple, with different folders for different clients based on the device plugin. And it should be the registry endpoint to register the NSE. So mostly for parts of it. And if you go look at the PR for the network service manager, yeah, it is basically exposing all the basically the network service, the monitor connection, and the registry APIs currently. Yep. So it should be pretty simple. Yeah, okay. Cool. Awesome. And I'm sure we'll find things as we go. I've been hitting all kinds of interesting things as I go through the system. So Frederick has been making a flurry of small changes to SDK as I find things and fix them. Oh, that's actually one of the things I did want to bring up to you, because you did this work originally with the span helper stuff. So I've been using a lot the span helper stuff, because it ends up giving really nice logging. In fact, one of the things I just did was essentially have all the wrapping for stack tracing handled by the trace chain element. So that now basically the errors that come flying off the system, if they're not already wrapped with a stack trace, they get wrapped with a stack trace. But as I've done this, one of the things I found is that in the logging that we're getting from span helper, basically if I return an error, 15 elements deep into a chain, then the stack trace gets logged 15 times. Goat. It makes it less easy to see what's going on. And so I was looking at doing something to essentially cause to suppress that, so it only gets launched. So we only see the full stack trace on the inner logging. And then after that, we just get something as the error propagates up the chain, so we can watch it propagate. But we don't get the full stack trace, because the full stack trace gets annoying really fast. I have to think a little bit about this. But one of the things we could do is we could probably have some type of a... Since we've created our own wrapper earlier on around some of these things, like in terms of returning errors or returning or logging or so on, I'm wondering if there's some part of the context that we can add in or we can keep track of some recently seen errors. But that's something that we'll have to think about. I'm not sure. I can think of an out-of-the-box solution to that problem. The one thought that I actually had on this was that effectively... So right now we're using the errors package, which is really nice. And the way the errors package works is that the error has a format method. And if you pass the format method to something that is formatting with percent plus V, so if you pass a thing to something with percent plus V, then it calls format instead of string on the object. And effectively what occurred to me is that one of the things that we could do is since we are wrapping all of those errors in the trace chain element, is simply in the trace chain element to build a simple wrapper that effectively has in addition a format once method that essentially... You wrap the error in this thing, it carries a once with it. If you call format once, the first time you'll get the result of format, and every time after that you'll get some stringing thing. Does that make sense? Yeah, that makes sense because it does... As you're propagating the error, there is information there that is present in the actual propagation itself. And so we want to be a little bit careful because just because you have a stack trace, that stack trace represents the current stack that it's on and we're not called, but the information that you may want might have been on another stack that passed it in information and when it returns, we want to be careful on how we squelch some of this information. Agreed. And so another option that we have as well is in the logging, when we could just log without the stack traces, and something we could potentially do is have a second channel or a second notation where the stack traces themselves actually get logged. So that way the log messages stay somewhat clean. But I don't quite like that as much though. Yeah, I don't either because it turns out that having the stack trace in context is unbelievably useful. It just turns out that having 16 of them in context in a row is not. Exactly. So I'll go poke at that a little bit because I think I have a clear line of sight to it. And what I would probably do just to make sure that we don't get confusion is for the subsequent lines where we are just returning essentially the way we get from string might also include an error ID. So you can correlate the errors if you have to and go, for example, search through to find the actual stack trace. You know, if the error ID ends up being squelched in a place that maybe it shouldn't have been. Or those sectors. Yeah, it may work actually. We did similar in logs for Monorepo and we look some numbers, like establishing of a new connection to the networks service with like 6.1, 6.2, 6.3 and so on. And add this information to the errors so it's possible to find. Yeah. So our current Monorepo uses a similar approach. Yeah. So I'll probably put something up for that today because it's making it hard for me to debug things in places. It took me a while to figure out what the hell was going on. Why was I seeing 16 copies of the error? Because by the time you've got 16 copies of the stack trace to the error, you don't even have the context anymore that you were in the middle of a stack trace. So cool. Anything else? I mean it before. Yep. Anything else before we go jump on the next call? Nope. Guys. Cool. All right. See you guys there. Yep.