 Hello, hello. So a reminder to everyone that we are being recorded and these will be posted to YouTube. Let me get the meeting minutes link and encourage folks to please add themselves to the list of attendees. So reminder to folks who are joining these calls are recorded they will be posted to YouTube we usually start about five minutes or so after I've pasted the link into the chat. if you could please go add yourself to the list of attendees in the meeting minutes. Also for those of you who are a little bit less familiar with the sort of way agendas are handled in network service mesh, we're a really open regime when it comes to agenda. It is not at all uncommon for people to add themselves and the things they want to talk about to the agenda themselves, including doing so as the meeting is in flight. So if there's something you would like to see added to the agenda, I would actually encourage you to go ahead and add it. We'll give it up until eight or five then we'll get started. All right, that's good moving. So welcome to the next network service mesh meeting. So we have this meeting every Tuesday at 8 a.m. Pacific time. And so we are also participating in the telecom user group, which occurs every first Monday. They are now rotating the time schedule. So the third Monday is no longer a thing, instead what they're doing is every month at 8 a.m. and then 3 a.m. Pacific is my understanding. I'll double check the time of the tailor. We had one yesterday at 8 a.m. Pacific so the next time should be one that is more friendly for Asia time zone. Again, I'll get the specifics and verify that before the next meeting. We also participate in the CNCF SIG network, which occurs every first and third Thursday of every month, which means there should be one this week. And that occurs at 11 a.m. There is a link in the meeting notes. Major events. So KubeCon China has gone virtual. We also participated in a cloud native zero trust talk, which I'm waiting for the recording to be sent over to me. And once I get that information, I will post it on here. And that was in last May for the OpenShift comments group. You just got very faint, Fredrick. I can tell you're talking, but I can't hear you. Did you miss everything or should I restart? I think I missed maybe the last 30 seconds or so. I don't know what everyone else heard or didn't hear. Is this any better? How we, where did you last hear me leave off? Cause that's probably what happened. I think you were going through events. You were just talking about Kubernetes, KubeCon being going virtual, I think. Yeah, so KubeCon has gone virtual for KubeCon China. We'll have more information soon. We have, in May 26th, we had a webinar to talk about cloud native zero trust, which had NSM highlighted inside of it. That was through the OpenShift comments. I will get the link to the recorded webinar soon. And once I have the link, I will post it in here. On June 9th, my company is going to be doing a webinar on federated learning of which NSM is going to be a key component of. And so I'm not going to be talking at the specific one. This is more for more of a high level in terms of federated learning. But if you're interested in federated learning or the use case that we're going to be adding on top of NSM, then I definitely recommend showing up to it. On August 17th is going to be KubeCon cloud native con Europe with a virtual experience. So what this is going to be, right, just the way that this is going to work now is that we're going to prerecord all the talks. And what will happen is that the talks will be played and the speaker will ideally be available for discussion. So this removes a little bit of the interactiveness a bit, but it also should result in more professional videos. And my guess is that it'll all be played on a European friendly time zone. So we also have ONES North America, which is going to, which is, we're still waiting for information and guidance on what's going to happen in there and the same with ONES Europe. My guess is that they'll have virtual showing on both. KubeCon North America is still proceeding as currently planned and we will be seeing a, they're currently putting together the committees to review the CFPs and we should see a schedule announcement sometime in September. In terms of announcements, it's not really an NSM announcement, but just to make sure everyone is available or aware rather, Dan Cohn is no longer heading the CSCF, but he is still with the Linux Foundation and he's taken over a new effort called the Linux Foundation Public Health. And so we'll still see lots of Dan, but they've announced a new person to run it and not have to get her name and add it to the agenda. But this person was the, I believe the head evangelizer over at GitLab and we should see some really good work in the CSCF to come from her leadership. In terms of the social media community team, we have an additional follower and are following five more people. We've posted out 25 tweets which includes the last week's video recap, call reminders, the weekly webinars, the introduction of T-Mobile has a gold member. So welcome T-Mobile to the CSCF. And there is also a virtual Linux Foundation AI Day EU it's gonna be on June 22nd. So if you're interested in artificial intelligence, definitely consider signing up. There is also information on Linux Foundation training and there's information on a CNTT white paper. CNTT is something that I personally have been involved with. My involvement has been primarily making sure that the wording of it is as generic as possible and try to minimize vendor lock-in. It's still early in its life cycle so there's still a lot of work to do in that space. So this is just one of the early releases that's being done. There's also information on 5G and Wi-Fi 6 in telecom TV and network architecture and we're cloud native, what does cloud native mean to that space? We also have a post in VMware open source about the latest earnings from the open source community. And there is information on CNFTestbed where the mobile telecom industry is making rapid progress in conjunction with the CNFTestbed. LinkedIn posted the same information and we've added an additional follower there. And we have pending updates on NSMCon EU 2020 and we are going to, once we get some clarifications from CNCF, we will begin promoting registration and sponsorship and schedules and so on. So in terms of the main agenda, we have at the top, we make sure to put your names on this stuff as well if you're the speaker, so I know who to hand it off to. With the start of it, we have a proposal on control plane and data plane separation. So who's going to take that? Is it us? Is it Harik or Roshina telling you this about this? Sorry, I don't quite understand that. Can you repeat that please? Is this one? Did Roshina put this one on the other end? Yeah, I think she reached out and said that that it was going to be next week. I think it might be delayed for next week, yeah. Yeah. But I suggested that we keep it on the agenda just to give someone an opportunity to say a few words about what we could expect to hear about next week in the hopes that it will prove of interest to people and encourage more folks to come around and hear you talk about it. I can speak a bit about it. It's a pop we have done. And the naming is maybe not, it's about application data and control plane separation. So it's actually to implement a service that will be the control, the control of a service would be like the IPAMS after that but we would have a data forwarder directing this stream in a completely other direction. So that's what's more, that's about. Oh, fantastic, that's interesting. You and I talked about this before a long time ago. Oh, yeah, yeah. Well, I mean, I'm sympathetic as to the naming things problem. I mean, I'm sure you've heard the old joke, there are only really three problems in computer science. You know, basically it's cash coherence off by one errors in naming things. Yeah, that'll be fantastic. So we'll make sure that ends up in the main agenda next week. And so I'm definitely interested in hearing what has to be said there. Yeah, we think it's also really good with the latest development of the forwarding. I mean, the new architecture of NSM because I think that could help us to do this. I hope so. Yeah, and one of the things just to not to dive too deeply into this but one of the things that I think is going to happen is that historically the application would always weld itself to the infrastructure of your environment. And we've progressed in time through a variety of advancements so that the application can bring some of its own architecture. So they're no longer told, you will bring in this specific database and you will use these specific primitives. Instead, the developer might say, oh, I'm going to use Node.js and I'm going to use MySQL or Redis or MongoDB or whatever. And so it gave a lot of power to the developers which gave rise to the entire DevOps style as part of the development team. But one of the things is that as we progress forward I think we're going to see as you're describing your application-based control planes and data plane separations. And I think that's, to me it looks like this stuff is leading towards instead of the application responding to the architecture of the data center, the data centers could also respond to the architecture of the applications and can form themselves based upon that because the control planes are eventually maturing to the point where they can take such input and do them in a safer way. And so I'm looking, I'm definitely looking forward to seeing what you all have to say in that space and we'll make sure it definitely happens. And we'll start to add a bit of background and so on and put it in context also so it's digestible for everybody. Fantastic. Sorry, I'm one of the colleagues here. Context, context is always good. Especially because one of the exciting things with network service mesh is we have people coming from a bunch of different directions. So we've got a bunch of deep networking people. We've got a bunch of sort of more cloud native people. And one of the exciting parts is we all share very different contexts. So there's a lot of stuff that's intuitive to one group of folks that's not to others. So explaining context is always very helpful. So next week hopefully, when we will show- I'm excited. Cool, we have NSM operator updates by Alex. So Alex, you have the floor. Oh, thanks. Hello, everyone. Can you hear me well? I don't know if my microphone is working. We can hear you. Oh, cool, cool, nice. So basically, I think I already shared with Fred and also add a little bit of what I'm struggling with in the NSM operator, especially in trying to accommodate the network search mesh inside OpenShift, which is also a goal for the operator. So it happens that OpenShift works with a different API server that has a lot of networking, especially network objects that we can retrieve information from that. And I know that the network search mesh application needs to retrieve all the prefixes that are being used by the platform, maybe OpenShift or Kubernetes. And there is a different way of doing that. So I'm working on that right now. This week I'll be working probably a big slice of the week on that. But I have a big question on that specific thing that should change a little bit the game and running network search mesh on top of OpenShift. And the question would be how often client goal or something like client goal is being used to talk to the QB API server in Kubernetes in the NSM code base. Like I've seen a few bits, but I don't know if I will need to do that a lot of times. So this is one thing. It may be kind of a silly question because I'm not completely familiarized with the code base. But from the perspective of getting the prefixes out of the platform to make the network search mesh application aware of those prefixes and say, hey, we need to exclude those because they are in use. That one I've seen. I don't know if I need to do other networking queries to the platform because in that case I need to use a different client because OpenShift has its own client goal. It's the same as Kubernetes when I need to talk to the QB API server. But if I need special objects that are already developed inside OpenShift, I need a different client because that one will have network getters and that method will bring a lot of networking information that OpenShift is using. So I mean, this is turning the field to me like a modularity for the win kind of problem, which is that effectively the... So you basically have said OpenShift has its own client goal that is a drop in replacement for client goal if you're doing things the client goal normally does but those are the things besides, right? Not exactly. Not exactly. The client goal for Kubernetes is exactly the same for OpenShift. But when you need to talk to something special that's only inside OpenShift, depending on what it is, then I need a different one only in that case. So networking in that case, I have a very special way of doing things that will be pretty easy if I use a different client. But for all the rest is a regular client goal. Okay, okay, got it. I got it, I got it. And so like the thing to understand is that the prefixes, the go figure out what prefixes you should exclude. If you, it's, there's an attempt made to actually keep that at arm's length from the core of network, the implementation of that at arm's length from the core of network service mesh because even though network service mesh fits hand and glove with Kubernetes, we don't weld to Kubernetes, right? And so my general sense is that what you would probably end up doing is effectively writing something that simply gathers the exclude prefixes differently as a separate sort of package. And then either we already have or we would need to make sure that we arrange to have the proper modularity so that one could configure the choice of using that to get exclude prefixes in. Or we could automatically figure out that that's the right thing, right? I presume there's some way to figure out that you're running an open shift, correct? Yeah, there is, yes, that's correct, that's correct. So I mean, you could literally essentially write it as a separate package so that you've got nice type modularity and then go back to where we're making the decision and say, okay, we need to make sure we have proper modularity and something here needs to sort of look around the world and say, okay, this is the environment I find myself in, therefore I'm going to use this to get the exclude prefixes. Sure, okay, cool. Because you know what's even better than configuration? Just working. Yeah, yeah, yeah, sure, sure, sure. No, that's totally fine. I'll try to modularize that and then make it, let's say independent of everything else. And yeah, the other two things I'm working on, one is capability tracing. I'm doing that in parallel. So I have a few tools here because in OpenShift, the privilege permission is a problem because we'll need to touch some knobs to make it work and it's better if I can really narrow down the permissions and I've seen the neat container that is injected from the admission controller is asking for permissions. Forwarding plane, network service manager. The unique container should not require any special permissions. Yeah, it should just be an unique container. So if you're seeing it asking for any permissions, let us know because that sounds like a bug. Okay, go ahead. No, please, please. Yeah, it's not requesting the permissions, actually. It doesn't request the permissions and running inside OpenShift, it gets rejected, it gets permission denied. So it's actually trying to do something that requires permission. Okay, okay. So basically the OpenShift security model, we need to figure out what it is and what it is that OpenShift thinks it needs to do it and then decide from there how to proceed, right? Because how to proceed is gonna depend entirely on sort of why OpenShift is grumpy because it may be, I can't tell you how many times I've seen something where like exactly like the situation, where it turns out there were five possible ways we could have done the thing. And we picked number three and the place that is now grumpy would have been perfectly delighted with numbers two or number five. And so why don't we just pick one that works more broadly, right? Yeah, that's true. Also, does this get flat out rejected or does it work for, does it run for a short period of time and then get rejected? Now it actually gets rejected. It's working because I made my own build of the admission controller asking for those permissions. So it works, but the permissions are too large like privileged or something like that. And I know this is a very tiny thing, it's not something big, right? So it's just a matter of actually tracing the capabilities understanding why they are there. And once I have that, it will be good to go. So this is why I'm working on tracing capabilities from the process perspective to understand how I can translate that into the security context, both from the pod and the container and then translate that into a policy inside OpenShift that say, hey, you are allowed to ask for those because you are working with that in that. Yeah, as a general rule, my vast preference would be to try and figure out a way that we don't have to ask OpenShift for anything magic or special there. Okay. Because, I mean, don't get me wrong, it may not be possible, right? But generally speaking, in every other environment we've dealt with, the init container is able to do its work in a completely unprivileged way. I mean, fundamentally, because like literally all the init container is doing, 100% of all it's doing is coming up, opening a Unix file socket that has been presented by the device plugin and making a GRPC call. That's basically it. That's all it does. Right. I guess the socket is in the slash var directory. Am I correct? Okay. So that, yes, it is in the slash var directory just because that's sort of the traditional Unix place to push such things. Yeah. And so as a good example of like sort of the, as a good example of the point I was making earlier about sometimes you just discover that you had five choices, you picked one and the other, and that's the one that makes everything grumpy. If the issue really is that OpenShift is for whatever reason grumpy about us using a Unix file socket in slash var, I would be enormously more inclined to adapt to that and put the socket someplace that is not going to make for a grumpy OpenShift than to convince OpenShift that it should open up what it considers to be a broader set of security permissions. Perfect. Perfect. I'll try that. Instead of trying to trace the capabilities, I'll just change the folders and see if it is about that. Yeah. If it is about that, then we can have a really, really good conversation about that. I'm ever so slightly curious as to why OpenShift is grumpy about having Unix file sockets inside a container in slash var. But I also know that there's limitations to my understanding of security in the world. And so sometimes you just sort of shrug and say security people, you know? Yeah, yeah, that's true. That's true. That's totally true. Yeah. I've been struggling with that myself. That's totally true. Yeah. But anyway, that's it. I mean, like one of the central tenants of cloud native is immutable infrastructure. And we try and respect that as much as we possibly can because it does end up making the world a much simpler place because you could easily imagine, right? Like a world where somebody wants to install network service mesh and they don't have permissions to loosen the security constraints in OpenShift for whatever reason, right? Or they need to now go convince someone to loosen that security constraint and nobody really understands why it's there because quite frankly, like I can't tell you off the top of my head why you would have that. I'm sure there's a good reason. You mean the security constraints? No, the one that says you can't have a socket and bar inside a pod, right? Oh, sure, sure, sure. That one, that one, I think that one is there for historical reasons and it's more Linux related than OpenShift itself because this LashVar directory, I think needs some root access but I don't know how far it should be secured. So I need to check that yet too. Yeah, I mean, like I said, it's always easier to figure out a way to get around something like this by just letting the security people be pedantic about what they're pedantic about, right? So, but what I don't wanna have happen is some poor user have to go and force some weird SE Linux policy into the world that they don't understand why it was there to begin with which means it's marginally dangerous for them to change it because it's generally not a good idea to change security things you don't understand. We don't want them to have to do that. That gets to be ugly. It also makes the conversations with the security teams much easier because when you're applying something that says, hey, you need to go apply this SE Linux thing and then you have to explain to the security team why you're doing it, it's a lot more painful to have that conversation than to just work in it, I'm privileged. So this is fantastic information and thanks for tracking that down. And I would love to see whether or not just moving it into maybe slash run slash network service mesh or some other similar location to see if that just works. Yeah, and that's an easy nothing to try. And then we get to the really fun discussion of, okay, where exactly should we put it and under what circumstances? Because then sort of two pieces of tension, one of which is as a general rule, you want to, as a general rule, you want to try and follow tradition as much as you can, which would say put it in bar run. And so we're not gonna be able to do that for security reasons that we aren't even interested in figuring, we don't care because we don't care to argue about them, right? There are security reasons, mumble mumble. And so, okay, what's the next best choice that we can make that hues towards tradition? Although if you do hear why it's like that, I do care from a personal perspective, not just from an NSM perspective. Yeah, I do too, just because the more I understand what's going on in the heads of the brains of security people, the more easily I can actually work with them. But I'm not asking you to try to chase them down, that reason down, just if you happen to go across it or if someone happens to talk about it, then that'd be an interesting story. But yeah, in terms of this, let's just try different directories and see if it works. And if it doesn't work, let us know and we'll help you with trying to work out why it's doing that and try to find a resolution that is uniform across as many systems as possible. Sure, anyway, the tracing the capabilities will actually tell me exactly what is happening because it will be like the process will try to do some sort of system call at some point behind the scenes, trying to do something and the kernel will reject that because of some policy that it's on top of it. So I'll be able to see what is happening and looking a little bit deeper into that. So, but for sure, I'll try using a non-privileged path and see what happens. And that leads us to a third topic which is pretty much related, which is the service accounts that are used in Kubernetes to run all the workloads. They need to be attached with a security context constraint in order to allow everything to run and it includes the examples. So if I try to deploy a new network service that one also needs to be ran under a specific service account. And in terms of the operator, the operator lifecycle manager is capable of creating those accounts and let them like pre-created for any service account. And I don't know if it is a good practice in terms of network service mesh to have one specific service account for all the network services that are being deployed or if it is too much, I don't know. Yeah, so I think part of what you're gonna run into there is that much of what we're using the service accounts for is that Spire can trigger off of the service accounts in order to issue spiffy IDs. So what you're probably going to discover is that it is perfectly okay to have organized your network services by service accounts in whatever way makes sense to you. And I tend to be on team as granular as you can get away with without driving yourself insane. So I like granular groupings of things. So I would tend to, you know, one more granular service counts. But it is going to need to be the case then that there is a spiffy selector for Spire that will cause it to issue an identity to those service accounts. Does that make sense? Yeah, totally. So yeah, I gotta take a look at that too. Okay, I see. I actually need to run with the Spire enabled and see how things will work with this specific service account. And I guess one thing that you guys mentioned on Slack is about the NSC and the NSC, how an operator could help with those guys. I don't know if you have a specific question on that if you wanna try to discuss that later. Well, I mean, I can sort of, I can hum a few bars if that's helpful. So I think the reason, the thing we sort of mentioned on Slack was as we look at things like the BL3 network service endpoint, right, the BL3 network service, which has a bunch of NSCs, and they sort of interrelate in slightly more complicated ways than some of the things that we've built so far. So what do you do in a Kubernetes system when you have slightly more complicated systems of things? Your mind immediately turns towards an operator. And I don't know if this ends up being a separate BL3 NSC operator, or if it ends up being some aspect of the network service mesh operator. My knee-jerk reaction, just cause I tend towards modularity is towards having a separate operator. But as I think you sort of mentioned persuasively in our Slack discussions, there may be very elegant ways of just rolling these in a generic sense under the existing operator. And I'm a huge fan of elegance. The other thing though, that's a little bit interesting and tricky when you talk about the BL3 is it ends up being something that spans potentially many, many clusters. So there's no one cluster in which you would run its operator, if that makes sense. Okay. And I'm not sure what the story is around that yet. Okay, so I need to understand how these NSCs are working together in order to- So if you ping Denise on Slack, he can point you to sort of, he's currently building out some design documents on this that can give you some notion of where we think we're going. And again, this is not like a burning issue yet. It's just, you know, I find the world is much easier when I can sort of like think about things in the back of my head for more time, not like actively think about them, just have them bouncing around in my imagination. And so I will often mention sort of perspective things to people for this reason. Okay, cool, cool. I'll definitely do that. Yeah, I'll try to address those first permission problems. So this week, I will have some resolution in some of them and then I'll see how the operator could embrace another API maybe if not, it may be a new operator. It's not a problem at all. Yeah, and one of the things that I think will win the patenting is from the operator's side, is I can see a couple possible integrations for how the endpoints themselves could make use of it. So one of them could be if you have dependent services that you need to run like, suppose you have an intrusion detection system that requires access to some Q or to some database. And so the operator can help with configuring and maintaining all of that information and helping out with some of the state in regards to like how do you upgrade it or how do you recover from certain types of failures and the operator can be that thing that helps with that. Now the second one that we can drive towards as well is when we start to deploy certain types of workloads out there. For example, in the series use case we have, we have the example where you have a pod connected to a firewall through intrusion detection system to a VPN. And in those scenarios, the management of the top level objects and ensuring that they get created in the correct way in a uniform way that is that's easy to manage becomes very, very useful. And I'll drive down a little bit more deeply into that. But when I say management, management of this, I don't just simply go run Q control apply and then you're done. Like when you started running a larger number of these across a higher number of pups, then you start to see environments where you have to manage all of these things in a higher number and it becomes unmanageable if you have to touch a bunch of things. So how do you make sure that you update all of your wiring so that you keep that uniformity across the lot of them. The operator is also a potential path towards making sure that synchronization happens and takes place. So I'm also thinking of use cases like that that help us with the unified management of some of these systems as opposed to just as a potential solution for that. And not just aiming towards, let's make sure this thing stays online. Does that make sense? Totally makes sense. What I see there is like modeling if I have a custom resource that could represent that system, then I can model the status field on that resource with a lot of very specific information and the operator as a control loop, as a controller will be watching that status field all the time, watching for all of those resources and it will take action upon any kind of state that you were retrieving from the various components for that specific custom resource. So that's kind of how the operator works. And so I totally agree that this is a very good use case. Yeah. There's a whole lot of interesting stuff here to explore. No question. And we're just getting started. I'm pretty sure we're going to find a whole range of patterns that we're going to be able to make use of this stuff. And the trick behind it is that all the communication to perform the actual connection and discovery is still NSM driven, but that still occurs. But on the northbound side of that, we can have something that consumes the monitoring and we ask things that consume, that consume things about the environment that it's able to access and is able to help with the management of those systems. Like I have a firewall. Maybe it's a bump in the firewall. It's a stable firewall. Just think it's a super simple example. Yeah. And am I scaling that properly? Like even that one question alone is something that an operator can help out with tremendously, especially when you start looking at, let's go pull the CPU stats out of Kubernetes and let's go pull the connection stats out of network service and try to work out if the firewall is going to start running into scaling issues with current configuration and expand that configuration out. So just like that simple approach and technology underneath of it and the operator on top of acting and potentially help do that, not just from a single cluster, but you could do some of this through the standard, through some of the standard objects in Kubernetes a little minute away, but to be able to do this potentially across clusters as well. So there's a lot of really interesting things that we can look at that will enable interesting behavior. And I would love to get a repository of templates or samples that people who are building these type of things on top of NSM can start off with one of those things and eventually just use that as a template to add additional functionality on top of it or to have some pluggable systems that they can configure and just tell it to go. So just some ideas moving towards towards the future. Sure, sure. Yeah, it's pretty interesting. Yeah. There's a lot, a lot to explore for sure. Cool. We need to make sure we get into the open policy agent stuff as well. Do you have anything else before we move on? Cool. I'm not hearing you. So I assume the answer is that you're good. Cool. So open policy agent, who's the main person for that? So I think Denise has been doing a lot of the work. I just wanted to make sure that we highlighted that there's cool stuff going on there. Do you want to say a few things about it, Denise? We just started to implement open policy policies for NSM. We have added two policies for token exploration and also for token validation. Mostly that's it. There's also a nice modularization of how these policies are handled. So effectively it gets very easy then to mix and match a bunch of really well done default policies in with custom behaviors that you may want to have. So you can, for example, say, well, I actually have custom policy behaviors about who I want to allow to talk to this network service in point. But here's a few characters to also mix in making sure that's default behaviors around token expiry and validation are actually being done correctly as well. So it should make the system very easy and flexible from a policy point of view. So in terms of policy, so this is an area that we're going to continue to work on to try to find ways to simplify the, not only how do you build and consume the policy within the network service mesh chain itself, but to also focus on the use of integrating and using policy as an operator. So if you have any suggestions or something you want to jump in and get involved with and help as well, definitely let me know and we'll help you get involved in that area. Let's see. We have 10 minutes left in the call. Is there any other topics that anyone would like to bring up? Okay. Well, as a reminder, we have the application control and data plane separation next week. So we will make sure that there's time for that. And with that, I want to thank everyone for your time and we will see you all again at the same time next week. Take care.