 Hi, Lee. This is Sanku. Hi, Sanku. How are you? I'm doing good. Thank you. Good to finally talk to you. Yeah, Sanku, remind me. Is it, uh, wait, am I affiliating Intel with you correctly? Yeah, perfect. Yeah, remember it. Yeah. Nice. Yeah. Yeah, good to talk to you. Yeah, I recently started working on service mesh and so being ramping up on all these details and also the community perspective of what's going on. Yeah. Yes, say more. Oh, yeah, so right now looking into on why and its usage models and its performance aspects. So that's where I reached out in Slack, join SMP and SMI. So, and it's all through the documentation that's been there, which I should say it's been pretty well documented on a lot of these things compared to the other communities have been part of in the past. So, really good starting point. Yeah, still in the learning phase. So trying to participate and understand what's going on and looking to contribute going forward. Nice. All right. Well, you signed yourself up now. I think. Yeah, there could be some symbiosis happening to the extent that that you that you getting engaged a bit helps give you a leg up on, you know, your your daytime focus like to help you sort of achieve part of your goals more quickly. It could help and then at the same time help some of the projects actually we're going to talk about a couple of them now. So, and in part of that it's funny funny that you mentioned it because part of it is about organization. So, yeah, absolutely. It's interesting to see a few years from the topic because that's, that's been my background last few years now. And that's, that's the space we're looking at leveraging on why, for example, among other things, but yeah so looking to see what that discussion entails. Oh, nice. Did you say I'm sorry I'm doing like way too many conversations. Did you say, did you say performance. Yeah, NFE as well. That's been my background. And yeah, I'm looking at some of the performance aspects to start with. But yeah, to understand performance better understand the stack so that's where I'm right now learning the stack. Gotcha. Okay. Yeah. All right, good, good, good. We, we, if we don't do it on this call. Let's make a point it will just make a point of exploring a bit more like there's there's a number of things that for me to share with you I think that might might be helpful. Absolutely. So, yeah. Amy, I think, yeah, I'm not sure I didn't start recording. Yeah, that's not good. That's, that's actually quite bad, because these are all set up to be automatically recording. However, you're here it's good. I'm assuming that you all are canceling your meetings for the next two weeks, because there will be no staff. Good. And we'll pick back up in January. So, okay. All right, fair enough. Good. I'm recording now it seems like everything is working so sorry. Yikes. No, very good. You know what it may be. We will see when we're done maybe it was recording the whole time and I just couldn't tell, but So, fair enough. What we've got a set of a small collection of dedicated individuals on today's call. And we've got two other individuals who are Lacking with me and And I'm trying to get them to join the call because it would be easier to do it verbally. So, okay, very good. Okay, well, the good thanks. Ken very nice to see you. Frederick. How's it going. Pretty well. We have any other guys who are planning to present joining today. There is. I'm Ian Wells from Cisco was wanting to talk about NFV. And, and so I've invited him to the call. I'm not sure if he's going to join or not. I'm going to talk about a couple of things somehow a little bit housekeeping housekeeping around the work streams inside the service mesh working group to talk about. And since Frederick is here is a little bit to talk about with respect to SMI in general. Cool. And actually, I'm Ken, since you're here, there's another topic that relates. Let's get right down. It's just the way all the topics are service mesh topics except for NFV. Okay, fair enough. So, thank you all if anyone has any other topics, please feel free to list them because we'll, we will get through them. So today is the December 17th. This is the CNCF signal work call. So we have a couple of topics of the service mesh working group. And so as a refresher for, for most of us, a working group have been formed on our some many months ago. The efforts that have been given on those work streams have been presented the last couple of cube cons. So we've given kind of an intro and a deep dive to, to those work streams. I'm highlighting a few of them. And, and really the first thing I think I want to ask about is, is a topic that we've had in the past and it's been, hey, for these more for these deeper conversations. Is there at the time we didn't know if it was appropriate to use this. So it's a big networks time to go through those. And we decided like, hey, there's enough. There's enough room in the agenda to go deep on these. And so we can part of the challenge is that some of the folks who are critical to those meetings aren't able to make this time. And so, Amy, one of the things that I'm doing, you know, I've struggled with either suggesting that this SIG network meet at a different time, or tossing out a separate meeting time for service mesh working group. We've got a CNCF calendar that overspills. But Amy, other than changing this, what did you have a suggestion on whether or not we should change this, this meeting time and just use it to go deep on some of these projects. Yes, you should just use the meeting time that you have adding more meeting time is not going to improve the situation of calendar collision. Sure. Sorry. Yeah, no, no, yeah, the problem is some of those people can't meet at this hour. And so it might be like, that is separate. If we need to find a different time for this meeting that is completely fine. Okay. Okay, I would just like to be consistent about when this group is planning on meeting so that we have a chance of being able to have some sort of like, you know, show up every week at the Yeah, it's like, I'd like to get us away from doing like one off meetings. That was all. Yeah, make sense. There's a lot of struggle in there so we'll probably send out a doodle or a or a survey or a request for times. So one of the reasons why, why this time is an inconvenience is based on time zones of the people who are working on a couple of things within here so there's under this focus is about performance and service mesh service mesh performance. It's about a few things trying to standardize the way in which service meshes and their workloads that their performance is analyzed. It's not entirely about hard statistics with respect to performance it's also with respect to measuring the value that's being derived from a mesh and how it's being run. It's to say how much you asking a service mesh to do and how much weight do you give how much do you value those things is part of the conversation part of that same conversation is about distributed performance analysis. A lot of times performance analysis at least in context of service meshes is done from a single vector. By generating load against an endpoint and measuring the throughput and the latency of those responses and watching overhead in of memory and CPU and that kind of a thing. Well, then, probably a fairly one D perspective a one dimensional perspective. And so there's been part of what that group has been working on is a 3d perspective is of how to characterize the performance of different of your microservices and using a service mesh to do that. What is taking on a name. Well, or is is taking on a project name called get nighthawk. If you're not familiar with nighthawk, it's the load generator for of the envoy project. And so there's a few load generators out there actually many load generators out there. The envoy's nighthawk load generator is is becoming more and more popular, and that working group is trying to help it be so it's creating a get night, a number of distributions of nighthawk, and kind of a get nighthawk dot, you know, dot I O project. So bringing this as a little bit of a review for those who might have heard some of this before. You can learn more about the projects at probably at this first link on the slides. So it is a call for participation, but it's also an acknowledgement that this time probably doesn't work for some of those that have been real interested. Earlier this week met with Microsoft, a couple of folks of Microsoft those that are working on OSM, and they're quite interested in participating in service mesh performance. And our I think that the project's probably bigger challenge has just been finding a venue to have that meeting and from my part I've been really hesitant to bring that holistically to this. Because I feel like it would do a disservice to, or, you know, going really deep on one thing and only talking about that thing would do a disservice to someone who wants to talk about core DNS or, you know, ambassador or other things. That's been my personal challenge probably about guilt over. Anyway, I should. No, no reason to feel like upset about this. We are we're currently working on being able to figure out better processes for, hey, how could we actually get projects that want to be incubating in. So, I mean, like I would put this on the SIG updates for January around help support the SIGs in this process. Is that fair. Yeah, yeah, cool. Nice. Okay. The other thing. Another topic is about let's talk about service mesh interface a little bit. A couple of couple of recent conversations with respect to the project. And I think there's there's something of I had called a meeting with a couple of the maintainers this week to discuss how to breathe some more life into the project. And how to help acknowledge that the leading service mesh are the leading service mesh by amount of adoption and mind share is steel. Trying to try to be thoughtful with my words but just as the, you know, as the lead as I'm hasn't shown a lot of interest in SMI and having all of the measures participate is most helpful to an effort like SMI to making it more useful and achieving its purpose. That's that's most that's just a statement and so I'm how to help make sure that the current SMI controller for Istio continues to be up to date as Istio goes through changes. We're talking about how to how to do that. Talking about maybe adding some new specs to SMI or or minimally enhancing the ones that are there. So adding things like the ability to define a retry to define a circuit breaker. To define how a canary would look like part of this ties into something that I've spoken to in the past and that is that there are any number of service mesh patterns that that working group has been working on documenting and and a number of those patterns like you would ideally be able to characterize those patterns in SMI like in SMI parlance like hey here's a pattern here's how you could here's how that would be fulfilled through SMI well. A lot of the patterns aren't able to be fulfilled through SMI based on its surface area today the surface area of the API. So the discussion is and the agreement was, you know, the land of those that are involved and focused and have skin in the game. Some of that has changed over time. And right now it looks like there's a number of people of the maintainers that are involved are ready to move some things forward. So for my part I'm one of those inside I think. So, so for most of the people here this is this these updates aren't intended to be a monologue for me. Please interrupt I'll try to pause as we're going. There I'm really giving part of these updates to say there's a lot going on in this this on these work streams and organization of them and getting them like part of this is I'm sort of pointing the finger back at myself and kind of having Ken on the side of the line as well where I'm saying hey there's a bit more organization for us to do to make to to bring some of these together to like have consistent meeting times to make sure that we can meet and go through these and begin publishing some of these things there's a bunch of well documented work that's been done around each of these but that but it's not. But for my part I'm falling short and so are some of the other people participating falling short on getting the word out there. These activities are going on, because I happened to bump into a lot of people who are very interested in them. One of those is as a related note is like the service mesh survey. So Amy you you might be in a position to help characterize this a bit more as Amy still with us. So some of you have seen. It's been once or twice there've been the CNCF has done it does a number of surveys I'm not sure what the the latest ones have been called but they're basically usage surveys. And one of them had been about how different service measures are being used and being adopted, and was looking the last you know a year and something ago when I looked at one. It was embarrassingly wrong. And I made comments about that privately probably. And then there was some recent chatter about another one that was published, and it was less embarrassingly wrong, but still quite wrong, or quite pissed poorly done. And as I went to go make a public comment about, or I wasn't anyway as I went to go comment on that I thought, oh, shit. And people might think that the, the service mesh working group or the SIG network might be responsible for that kind of a survey. Maybe we should get it right. And in the service mesh and the SMI by weekly meetings there was an ask for to survey SMI users or potential SMI users for use cases like ones that Frederick had recently mentioned. And so I took up an action item to help facilitate such a survey. I'm taking it through the SIG network here. So for my part I think I think it's high time that we do a real, an actual survey that there's there's no doubt. And maybe December is the month there's no doubt and can you may know. There's no doubt a was the CNCF radar, the use end user radar and user technology radar that will eventually there will be done one done on service mesh. And the clock is ticking. And for my part. I think those can be kind of helpful things. They're, for the most part, also not well done and I understand that those are hard to do. And we definitely want to be involved in those I know that it's not it's not registered a schedule gap but we do want to definitely support that leak. Nice. Okay. Yeah, good. Yeah. I think a barrier had been recently broken down it. I've been invited to go to hang out on the end user call, which is great because for my part I want to help bring those use cases back to the service. And so yeah, so I was going on public record to say that I caught myself before I made a silly comment like, Hey, this survey isn't well done. Well done. And so, so maybe we should step up and do something. I'm going to, if anyone's interested in participating in organizing this survey. Comment here, or comment verbally. I think that the ones that have been done to date they feel more damaging than they are helpful. I think because we've got things like are you using this project name and turns out that project isn't even a service mesh. So, I do want to point out that Ian has joined us and we have missed him in the start of the conversation. And so, so I want to point that out in case we want to read this BNF B stuff. Yeah, I'm here, mainly because Lee wanted to know whether I was going as gray as fast as he was been the answer is easily. Yes. Wisdom, it's all it's all wisdom just. Yes, genetics one of the other. Yeah, if we want to talk NFE again I would very much like to have that conversation, which which is obviously way off the top of service measures but a topic of service measures but to be fair that that's part of the the reason I want to raise this here. Ian has a moment. Hey, Ken, it's been a very long while actually. Yeah. Apparently, this is the ex Cisco complication that's really quite disturbing. So, NFE, as a general statement is trying to make things that usually end up not exclusively but usually end up in service provider networks things that are trying to do, you know, treat network traffic as more packets or do weird things with weird protocols because you know service provider standards are generally weird low level more low level networking than you would normally find in Kubernetes. These are the things where that I'm trying to deal with. And to be fair, there's a lot of people from different companies also interested in this for obvious reasons, and the puzzle that we have, which has been tinkered I think not necessarily solved in the Kubernetes world is that we want things from networking that CNI is and therefore service measures built on CNI is not providing. And the reason for that is that the CNI's job is to simplify developing conventional applications, and the sort of things we're talking here are anything but conventional with their use of networking. We all know that networks, networking is made of packets but most of us never actually have to give any thought to that because the BSD socket API we consume is not relevant to, you know, the packets that are broken out. So, the problem we have here is that the CNI, again, is nearly 100% of its users. It is delivering exactly the functionality that they're looking for. And again, the fact that when we want extensions we can just build on top of it with a which says to me that it's doing the right job. So, the question I'm left with is, firstly, can we enumerate our requirements on the NFE side of things the things we want to do without regard to the solutions like CNI out there to actually say if we can do this then we can get our job done. The second question is, what's the best approach to getting those things done when we're working when we're trying to make an application run on top of Kubernetes. And my feeling here is that the more we try and bend the CNI to the will of a very small minority of use cases, the more we're doing the wrong thing. So, that's a personal feeling. How the general question is, if I want to run a network function that plays with packets, the other network function I want to concern myself with is if I want to run a network serve function that consists of multiple reaching domains verse on the network. How can I get those things to work in Kubernetes and what would I need to change. So, just to for more context in is, is part of the use cases and the conversations that you've been having to date have those. Any of those been in and around the network service mesh project. At the start the network service mesh, Ed took it and run with it but it was me and him and Kyle who had the original idea so yes. The original idea of NSM was something a little bit more raw than it's turned out to be. And I think it might have a potential part of this solution, although, as I've seen it develop I think it's developing more in the direction of cloud to cloud connectivity, which isn't our problem in NFE. It's actually more dealing with things like wide area networks controlled by independent operations teams who would like you not to stuff up their network and get them in trouble. Who would also like to say that them playing with their network is not stuffing up your application and getting it in trouble. So, the it and the NSM use case don't seem to gel quite the same stand in fact my feeling is my suspicion is that in much the same way as you've got CNI and then you've got service measuring on top. NSM fills the role of that service mesh in raw networking, but there isn't an underlying layer there and there's a more basic question of how it should be dealing with the network at the lower level that was is the one that we're missing right now. Again, again to like to just help characterize the problem statement or like part of the question that you're trying is, would it be a mischaracterization to say that that you're in search of extending CNI surface area. I want what I think is a far enough different kind of networking that it's probably not sensible to add it to the CNI. If we're being honest, in much the same way that I don't try and make block and subject storage answer to the same API is I don't think these two sorts of networking should answer to the same API is a little bit towards that as well there have been years worth of efforts to try to extend out CNI scope from variety of different parties and CNI remains mostly unchanged in this respect. So, I even I agree that I with with the end but I don't think it's the right approach anyway, but even if we thought it was the right approach the chances of getting a community to accept the meeting in that area is makes the point most likely moved anyway. It doesn't. Again, when you consider, firstly, the, you know, the kind of networking we do is way off the wall compared to, you know, anything that your average Netflix or whoever is going to do for a web app or similar. It's, you know, dramatically different to that. So, trying to combine those two things into a single API leaves you with all the complexity that we might need thrown into, you know, all of the standard use cases that don't use it. I don't think it's going to make anyone's lives happier. And I'll sum up the contract in a phrase. I, it's an API that when you call you say please, please add networking to this network name space Linux never came space and tell me what IP address you chose. Tell me what interface name and IP address you chose. They're not even the summit like it's very, very, very minimal down. So it's focused around pure pure L3 connectivity for landing an interface into into a pod. And anything beyond that is not part of the technically not part of the spec like there's, there's people little drop things like oh let's do something with SRV let's do something with MPLS let's do something with with only additional and these are technically all out of out of spec and they're running into the same problems that neutron ran into earlier on which is that neutron did not meet their needs and so they ended up co-opting things like oh yeah this Mac address label that's an MPLS label instead. And until they worked out they could actually get underneath to the to the underlying Q and inject their own messages and then became a nightmare. So what it comes down to is without necessarily judging what the implementation is going to be whether it's going to be the CNI or not but at least bearing in mind that it might not be. What I would like to do is I'd like to make sure we've got some degree of quantification of the problem statement, rather than the solution, because this is one of those problems that you tend to find people are basically saying I see these open source projects they've been out there for so long. They must solve my problem. I don't think people necessarily appreciate how different their problem is. So I think it's more. There's an element of just defining what would be a working solution how I would know if a solution was enough for what I wanted. And then, you know, kind of exploring some of the more perhaps off the wall options than then again trying to adapt the CNI to make this work. Yeah, one question here. Some of these requirements are being discussed by service providers and operators and CNTD or what's going to be, for example, the reference models or their CNF working group. So how are we talking about the requirements from there that we could address via CNI or outside of CNI or. Well, I'm here in part because I've been involved with the CNF working group, but I just want to be clear about an audience problem that we have here. We talk as if service providers are our audience for this level of feature and that's simply not true. The people who consume API is that do networking and the people that write applications that do networking. Service providers then try and run those applications but service providers don't generally. I mean there are exceptions to every rule but service providers are generally not the authors of those applications. So if we jump in and say that a group of service providers, like for instance the telco user group or CNT which is service provider dominated are going to, you know, come up with the right thing to make an application run. Yeah, it's the tail wagging the dog in much the same way that Kubernetes APIs were developed by people who are writing applications. These APIs, first and foremost, need to satisfy the people who are writing those applications. And then as a secondary factor they also want to satisfy the requirements of network architects. They want to more accurately produce results that make network architects happy. It's probably not the API is themselves the architects are calling but yeah. So I guess then the question is like, how do we capture those requirements and have a solution for lack of better term solution via CNI. Is that the kind of question we're discussing here. We are right. This is a judgment. Firstly we define the problem before we start saying how it's solved but my best guess here is that whatever that solution looks like it's so far removed from what CNI does today that it would make more sense not to mingle it with the CNI. But that is a best guess it's not necessarily true right now until you know what the problem is we can't say what the answer is. Yep. Yeah, and to be clear, another thing in terms of CNI is there's also a northbound and southbound API issue so one of the problems you run into is, well what problem are you trying to solve so if you're if the problem you're trying to solve is I want to get multiple interfaces within a pod, then a CNI approach may be okay because you can do like CNI multiplexer. One of the one of the problems that CNI multiplexers tend to run into is that they're commonly configured through a custom resource. They bubble up the southbound CNI API to the northbound, which means that if Kubernetes ever decides they want to change the CNI spec that those low level details have been have been exposed. That also means that the many of these the subnets and similar and or parameters like what like VLAN, the XLAN parameters and so on often get welded into those as well rather than negotiate it so we do want to be careful there. It's interesting what you've just written down Lee, because I'll come to that in a moment but but starting from what Frederick's just said right if we look at something like maltas. Right it said that a CNI networking the CNI networking API is largely adequate to the task it describes the sort of networking that we want. And then it said, but the only problem here that we used to be able to do with virtual machines and we can't do with containers which we can't get multiple interfaces into a single container. Now, there are flaws with both of those statements statement one to one number one that CNI gives us the kind of networking we want, which is that there's a single network domain. It is layer three addressed it's all about reachability that there's one egress from the cloud that I use for for the networking that's coming across the CNI. Literally none of that is true. In terms of NFP, I want multiple points of egress from the cloud. And the only reason I have multiple interfaces in the container is because that's the only way I can point at different points of cloud egress at the same time. Then we get things like whenever we get into this it always seems to turn up with IP address management. I might have an IP address, and it doesn't always rely on one interface I might be doing things like ECMP load balancing so my interface, my address, my grades, I might be doing MPLS where I don't have an address at all. So, all of these assumptions. It's easy to find a use case that invalidates them. And then finally, that namespace thing. I am trying to get multiple interfaces into my namespace actually no no I'm not literally don't want to interfaces in the namespace that's exactly the thing I don't want. I might want them into independent namespaces. I might want one of them to be passed through interface but I absolutely never want to interfaces in the same namespace. By the way tying this to to the layers as well so most of these are at the L2 and L3 layers and we have people who are service mesh experts here. So, one of the one of the things one of the challenges as well that we that we need to look at is, is how to build in these things so that they complement each other so in other words. Most service measures focus on the L4, L7 managing PCP sockets managing application related things like HTTP or GRPC or similar. And at this particular level, you're looking at how do you actually establish that initial connectivity like multiple clusters how do you establish connectivity between those clusters or maybe a cluster with connectivity to external systems could be a VPN to a client or a partner. In the case of telecom they could also be how do I establish connectivity across multiple sites now that we're starting to see edge data center start to start to come up and how do we ensure that we don't get things like IP complex or localize the global and scope trying to localize them. And these are problems domains that are outside of the standard service mesh, but they have, they have massive impact on on the service mesh space, depending on the type of service that that is provided. And the ideal state is that it's fully abstract again that you don't have to worry about those low level details as long as like, you know how to connect to something that it just works. But we're not at that at that state just yet. And so just just some just some thoughts as well because I want to make sure it's relevant to to the people who are who are on the call as well. So you need both is what I'm saying. Yeah, you totally do as a matter of fact you know there's an any number to your point. Frederick there's any number of like sort of core service mesh use cases that that when you have a service mesh, as it's defined today or as it's commonly spoken to today, we have a service mesh discussion. But don't. They don't go low enough to rely. They don't go low enough so that you can actually realize a number of those value propositions, because the way that you would deliver those value propositions is by use of lower layer networking or things that aren't just And I would argue that they shouldn't drop down to that level as well because they over complicate the, the solution and they start to make assumptions in the space and so it's better to keep loose coupling and keep that abstraction. But the problem with it is that all all these abstractions end up being bleaky in some areas so you end up making some, some requirements somewhere. One thing that will help in this particular space is there's right now if you look at how have what the underlying assumptions are the underlying assumptions are primarily around networks are the identifier of who you are and IP addresses are the identifier of who you are. And when you establish a service, like one problem I ran into this actually was on an enterprise not a telecom site but we'll see the same problem arise on on telecom and service again especially with with edge data centers coming along. And so I had one, I had two companies I was joining together, one of them was on Google and everyone had a major on on prem solution. One site said that they, they had a range of IP address they're able to use. The other said that those range of IP addresses were completely taken you're not allowed to use them and and join them, and the cloud prevented it was using Google at the time and Google prevented them from making use of other set of addresses. And that particular system took around six months to resolve because of IP conflicts and firewall rules that have to be set and have to be mitigated and this problem is not uncommon, the, the average time with many large organizations could be literally several providers just to establish that initial connectivity. And so the implications on that are huge. And so when we start looking at things like application method and large enterprises tend to actually look like service providers because their service providers for their be used, and they have a lot of infrastructure for their views. And so these problems are not isolated only to do telecom. But one of the things that I strongly recommend is not using the network, or IP address as the identifier anymore, because that's now dynamic we have, we have, it makes the, it makes the infrastructure. When you bind identity to it it makes your, you've welded the security and infrastructure together in a way that makes it brittle. And so one of, we have within most of those meshes, the spiffy identities or we have cryptographic identities are able to use. So binding against the identity against those trust domains is a much stronger proposition, because I could say this trust domain and that trust domain, we can move together. And it doesn't matter what IP address they move to, we can say, yeah, that's using 10 dot something it's using when I do it's using a public address, don't care. So prove to me who you are based upon that cryptographic certificate across across trust domains. One of the problems to run into is that the service messages don't tend to implement the Federation portion of speed they implement the workload API, but not the, the Federation portion of API, which then complicates the Federation of these of these identities. So by providing some form of Federation, whether it's allowing your system to to run with spire and building against spire, or building the Federation API out and not a complex API. It doesn't fully solve the problem, but it gets us a major step towards that level of independence and then that helps separate you from those L2L3 concerns. And that means we could even drive you to a different protocol, which could be non IP based, you don't care, because you've asked for something, as long as you can get that socket to what you need, you've maintained that that independence. So, does that make sense. Oh, yeah. Yeah, I think that the day had, the day had long past sense some, it was appropriate to use an IP address to identify much or at least, at least in my area of focus and when you said spire a moment ago, was were you using that interchangeably with an, an SFID, like, let me let me say like this that I'm being, I'll be, I'll be very specific. So spire is a is a reference implementation of spiffy. There's two APIs, you have a workload API that pushes down the SFIDs down, and then there's a Federation API, which is how do you actually join things together. How do you join together multiple trust domains. And so, spire is a reference limitation doesn't have to be spire. I'm bringing that in as an example because it fully implements both sides but it could be, it could be Istio or it could be Kuma or be something else. As long as those Federation APIs are a dear to in the spiffy spec, the spiffy is the is the spec. Then that gives that would help tremendously towards getting us towards the multi, towards a multi cluster approach. That's actually something that we're really interested in. On the side here. Well, wondering, like, what is the status of other service matches interest in either like the Federation spec of spiffy or the Hamlet spec that was proposed by VMware and seems so kind of like lost steam over the past like year to six months. It's definitely something that we're looking at for internal use cases of federating like console servers with console plans with other console deployments independent that are independent. But we're definitely interested in hearing from other measures about what sort of interest they might have in those protocols. Mike that the Andrew Jessup or a representative of spiffy inspire projects had wanted to put together a definitive list of which meshes have are have implemented spiffy or have implemented spire. And he did so. Here as a reference under under this section here under functional just there's a spiffy inspire section. And so that's put together that the yet the yays or nays are well put together by project project maintainer of spiffy inspire. And so, you know notably, and you know Frederick is sort of hinting around this that one of the one of the more prominent service messages doesn't you know doesn't necessarily use either the two of those, or like sort of halfway uses spiffy and that that if these were all green checkboxes I mean one way of saying this if if sort of the spiffy column was showing a lot of green. Some of some of the challenges that Frederick it was as articulating begins to and some of the power of, like, in my mind, while there are all kinds of non technical concerns and politics to be addressed around the projects. As long as service mesh as a technology makes it, so to speak that it's an inevitability that federate in Federation of these identities of these catalogs of these services is either SMI, or maybe an NFVI or a or a Hamlet. You know needs to become a real boy and and one place to sort of make it a real boy if you will is is here in this forum, run this in the CNCF sort of in this forum. And those project maintainers had asked a couple of times about interest, and then we're interested I mean we're, I think there. I know for a couple of individuals that are involved from VMware that they were. It was pre pre release of Tanzu service mesh, and they weren't feeling the love and Istio environments working group, so to speak. And we're feeling placated or we're being placated in the SMI community meetings. And between that and I think, you know, yeah, I think that you've got an accurate sentiment as you characterize, so where the projects at. Yeah, some, some information on VMware, they hired three of the top spiffy spire engineers and are having them work primarily on on spiffy and and spire. And so that that was done. That was done relatively, relatively recent. So, not that they have a service mission in the game but HPE acquired site tail which was the company that was driving spiffy and HP has been doing a lot of work to continue that those engagements. We've been actively working with several service measures to help with that with that integration, including in some scenarios, providing manpower in some limited cases, based upon their based upon that teams capabilities because it's still a relatively new acquisition so they haven't fully scaled up yet. So there's. So there are commitments from Friday groups. I've also had conversations with with red hat and they are they are interested in the, they don't have a good Federation story. And they are interested in this has a potential approach, which has some SDO implications but there's, I think the SDO one is going to come down to whether the governance has been sufficiently decentralized, because while he was being controlled by by Google, it, the, the, the will just wasn't there for for some of these like they've been you, they supported the, the SFID downstream but not the, not the not the Federation portions. And so I, if, if the governance is decentralized, there may be more, more interest in this type of things because it helps with with a broader set of use cases then the initial scope that this deal was originally designed for which was only Kubernetes cluster because it welded the big problem actually comes from where do you, where do you root your identities to is do you root them to to an individual cluster because right now the cluster is the is the identity provider, or can you can you weave it into a wider story, which as a company or as a business unit, you're able to control your identities, regardless as to whether they're Kubernetes, some or something else, and to provide that that integration across the across the board. And, again, going back to tying it back to the end of the portion. This allows us to treat entity as a layer, and we don't have to worry about from your service website you don't have to worry about worry about any level detail as much because you're not bound to a specific IP address you're not binding to a certain VXLan protocol or similar those are those are all abstracted away at that point. Ian, with the time that we have left the of the goal that you were more or less attempting to accomplish in terms of like, you know, raising up you know defining some problem statements sort of identifying where a couple of these are, in other words, you are either after a different persona or after different use cases for those same personas in your, how can we help, or, or what, what would be ideal as either an outcome or sort of a set of work or Let's try that without mute. So that's that's a good question I don't have a straight answer for you right now. I just wanted to basically, I guess, terrify you about the kind of networking we're doing so that, you know, really to say that it doesn't matter what you do with service measures, it isn't going to solve this problem, because it's a weird and different problem it demands efficiency, among other things. And, and it's annoyingly low level and not really relevant to, again, most people's experience of what they're doing with networking so I just want to basically raise the fact that there is a weirdness going on here. You know, the networking see I don't want to hide it from you want to basically lay it on the table. Now, in terms of solving it. I'm working on that. I'm working with Taylor on that we have the CNF working group. It doesn't currently belong to a SIG. It's not obvious that it necessarily wants to belong to the network SIG, because it's not about networking in Kubernetes. It's about building a network function. And what Kubernetes needs to provide to do that is only one part of the problem and it's also a little bit. I'm trying to figure out at the moment it's how do I make my, how do I make my network functions cloud native, what is cloud native what is an application so it goes on. So it's a little bit cross platform. Really, I just want to basically lay the problem out. So you can see we've got some work to do here. I want to figure out how to get the right people in the room to talk about that because again, talking to people who operate hypothetical network functions is well and good but it isn't going to resolve the networking needs of Kubernetes to talk to them because they're not the ones necessarily writing those applications. I think this will be an ongoing conversation while we figure out the structure of this basically. So I think that's one of the points and I'm to be candid, I'm confused as to like CNF conformance working group, like, it is all about I wait like, for my part, I don't need any additional work. So you're happy not to or not happy to have the working group here or not. So much more about networking and even deeper sort of lower layer service providers centric networking than the CIG, CIG app or the other and granted it's a cross function. Yeah, well that's it. There are other parts here, things like for instance CIG app delivery could potentially be interesting but on a fairly cursory read of what they're doing. Then what they're doing is talking about applications that are delivered, including the Kubernetes right Kubernetes is a part of the application. And again, that turns out to be a use case distinction between what service providers would like to do. And how most people use Kubernetes most people are interested in running one app on their Kubernetes. That's fine right controlled by the app team because it's helping them. Service providers want to basically make this an independent component that they look after and they run multiple CNFs on it. So if we are doing something different than then the familiar pattern that's well studied. I think it's important that we get people who are thinking to do that to recognize the distinction between what they want to do and what everything that has been done before because the immediate conclusion they come to is, this is the best work. And all I have to do is go and pick and choose the components off the shelf and figure out which ones make it work. And that's sometimes a flawed assumption. Other comments on problem statements that Ian is documented. I think by the way I'm Frederick and Ian both of you guys like the way that you you characterize the differences of these layers is, is nuanced, and I thought, I think very accurate, like, and the nuances are quite meaningful. There's, it's a deep, it's a big old and this is what why this is why earlier when we're opening the call and I was talking about the service mesh working group and some of the things that it's doing. The where I was trying to express a feeling of potential guilt was to like go deep and focus on one of the advancing one of those initiatives using this hour, because networking as a thing is fast, broad. And so, but you might view this as for networking as a whole, how do we make the important things easy and the hard things possible. And arguably the CNA might makes a lot of important things very easy and service measures make even more important and comparatively complex things easy. But the CNI as it stands denies the possibility to make the hard things possible that so there's a little bit of something that needs to be broadened out in that direction. And since, again, nearly 100% of Kubernetes users have no interest in doing hard things. It's completely understandable that's where we are. But unfortunately, we're awkward. Yeah. Let me ask you this. Right, there's a number of related problem statements that that I would probably add to this list that are from a layer seven centric perspective where we're saying where I would I would say that the application and its needs are under characterised in in layer seven to be able to bring forth to facilitate for those needs a bit more programmatically or a bit more automatically and things like not exactly CNAB, but things like open application model OEM. I am lost with your terms but where are you going with this. But there's a lack of the lack of application lack of a complete definition of what a workload is what it is what it needs what it depends on doesn't need multiple paths out like, and if there are, why would it prefer one versus the next sort of we consider like things about affinity or anti affinity and yeah, you've got it's sort of interesting. I think what's happened here is that we don't really talk at the level of a workload. We talk at, you know, again if I'm Netflix right. And sure this is not how they run on many levels but if I'm basically running a and as a service offering with Netflix be it Salesforce be anyone else right. It isn't a matter of running multiple applications I'm running literally one application of the components that I'm trying divide up and give to people to manage them separately, but I only have one application I don't really have to think at the application level because it is my literal job description. When you start asking yourself well how what happens if I'm running many of these things simultaneously not microservices but actual self contained applications. If you run it in one cloud, then you actually don't have a description of what an application is and what it touches and what else it might want to communicate with, because a lot of these things like policy of how microservices communicate and the security you put in place there, and the metrics of how microservices communicate that are relevant to the person running the application and maintaining it are also interesting to the guy who's using multiple applications because now they want to know how these are interacting with each other, but actually the internal communications the application is a black box right. It's happening. Get stats out. They mean nothing to me because I don't know what conversations this thing has internally. So, we haven't got that higher level of abstraction today, certainly. There's, there's a part of what needs to be considered in the space as well is how do you provide enough information for these infrastructure components to be properly scheduled by by the orchestrator. So, right now, it's, it's, it's a black box but even the parameters on on what the system supports is is still a black box and not well, not well abstracted. So in other words, to, if I want to run a bump into wire firewall, this thing doesn't have an IP address it doesn't care about load balancers or anything like that. But when I, when I install it into Kubernetes, I need to know, okay, well what, what protocol is this thing going to use is it an Ethernet based on was it IP is it something else. So what is its payload. And the second thing is how do I actually communicate with it like do I speak with it over a kernel interface is there a shared memory thing I'm going to drop in is or a device I'm going to drop in that it knows how to communicate and so it needs to be able to describe its capabilities to the northbound so they can get scheduled in now whether it's Kubernetes or something else I in the ideal world shouldn't matter. But, you know, there's right now there's not a good description or definition of what these things look like in a way that we can meaningfully consume them. Despite the fact we have all these different efforts to find okay well what is the infrastructure for this like this core problems still still remains not untouched but but insult. It's some of these things. Again, if you consider effectively you've got a at the lowest level of this networking you've got abroad, but dangerous level of abilities, you can do whatever you like, almost with the networking. But most people will shoot themselves in the foot if they try and join in at that level. And at the highest level, you've got something which is narrow and prescriptive, but generally doing exactly what you want to do with no additional, no additional craft, like, again, service measures trying to deliver particularly specific functionality that health applications. It's possible that if we stuck that lower level in there we would suddenly find more flexibility at the higher levels as well. I'm not saying it's necessarily exactly related but the one that always comes to me is the fact that envoy is effectively running privilege because it's doing dirty tricks with the, with the networking that aren't strictly permitted by the CNI and argue would be a related question of well what networking is the CNI itself consuming. If you want to see an eye plug in how does it attach to the outside world is there a positive potential for for more things there so it's possible that what we're doing here would bring value. It's not the first place I'm trying to go, but I'm certainly not saying this is all on a on another avenue all by itself and it doesn't tie back in. By the way, the OAM looking at the networking portion. I think is binding on the wrong thing because you're binding on a subnet for your application and you pass in a subnet ID for for how to connect it in. And this, this is, they should have nothing to do with the application itself the application should not ask I need to be on this specific subnet in order to in order to communicate that that's something else. There is a place where you need to describe these these properties, but it should not be in the application definition itself. This this is one of those things where we always like to talk about immutable infrastructure and if networking is infrastructure then indeed it's immutable. But if you're a part of the networking then it's not infrastructure is just, you know, more functionality existing elsewhere and in no sense is it immutable. So when we talk about network functions since they're going down to the lowest level, then they start choosing their addressing. They start choosing their MAC addresses sometimes they certainly start choosing protocols that are not IP. And that's where, you know, a lot of these standards like oh well you're an application of course you'll want to submit and figure out what to do with it. That isn't actually how this works once you've got to that level. There's also the question do I even want to submit like if you look at Calico Calico networking is not subnet based it's actually everything is a slash 32. So, the concept of a subnet in Calico is not to say it's completely meaningless but it's, it's not what you would expect. Sounds like you've got another meeting to go to Frederick if you don't like what they've written in this OEM spec. Yeah, I do have, well, I'd say I have to jump onto another call right now. So, not an OEM one, but possibly I can only see sounds so many though so I'll see what I can do about that. Yeah, yeah there's, if anyone is interested in the link to that call. There's a link to the call to get the overview of OEM. I don't know that it, I'm not suggesting that it captures or provides a facility to capture all of this. It did look at first blush fairly like it had considered for quite a bit. Yeah, if you look at it from something like like build pack like saying what do you want to connect to well there's a database that I'm going to call that I'm going to give a name but that name is not a is not a. It's a top level request it's not it's not the actual binding it's like I need to connect to my Postgres service, whether it actually renders to an internal Postgres or some external one, a hosted in cloud version, you don't care about the protocol is Postgres and you need to communicate to something that speaks that. And so, having a having a system where you can describe I need to have a relationship with a service within the spec I think would be would be excellent. And how the system renders that into it. It's the same advice that I gave to originally they call it the multi interface working group they changed it to to. I love it now but I never never plumbing. I think it's what they call it the working group within within Kubernetes. But the initial version of it was that they wanted to provide multiple interfaces and it's like who actually cares about multiple interfaces as a user I don't care about it about multiple interfaces I care about. I need a fast I need fast access to my, to my storage. And I want to describe that I have that I need this faster access somehow, and how it gets rendered into the system I don't care if it's all over the same interface whether it's a different one. If they, if they found a way to make carrier pigeons work in a, in a ethical way, then I'm all in. And so it's, it's not. It's not a, like, you know, you want to be careful not to bind to things are too low in this scenario and I think that's, it's a common practice to do so but there's, there's consequences down the line as you start to, as you start to try to scale these things up. And so I think describing what they need to connect to, and the type of things that they need would be okay. But not necessarily describing down to like I need the specific submit ID or I need these particular, because those that now you're making, now you're making assumptions about the implementation that may not be accurate. Clear separation. Sounds like we will, we will not. We will meet next year, I think is our next point of collaboration. It's been a great, great call. Any, any final requests or call to actions that people had had before we close out. Thank you Frederick. Thank you. I think we may want to catch up a couple of. I'll be working this week probably middle of next week. Ian and Mike Blake very thanks for coming guys. Welcome. Yeah, thank you. Have a happy new year. And he's happy holidays. Bye guys.