 Okay, I think we can probably get started here so thanks everybody for joining today. Thanks for the comment in the chat. This is the telco version. So this is the, I guess, like second kickoff meeting if we can say that of the CNF working group. And the notes on top. And the agenda for today, first I want to say thanks to anybody that's contributing to the discussion right now in the Slack channel I think it's been like quite a lively discussion and I'd like to kind of address some of those points today in the agenda. Quickly going over that we'll just do like a quick overview of what we're doing, talking about like how we think that the group can work. And then kind of addressing some of those discussion points that have been brought up in the Slack channels, especially around, like, what is our audience and how do we incorporate them into what we're doing so we're really focused on like the outcomes of this group. So if we can get started is there anybody else that would like to add something to the agenda. Okay, hearing none, I think we can get started. So, and also please feel free before any other meetings to add things to the agenda, whatever you want. And so the first thing I'm going to point out for those who weren't able to attend the like official kickoff meeting at the moment is there's kind of like two different things you can do here. One is, you can rewatch it on the CNCF YouTube. And the second thing is we also put together a short overview deck of the working group. Now this will be a living document it'll be updated but the link will stay the same so if you want to pass us off to other internal colleagues and chat about it, or provide comments like on the deck. So please feel free to do that and use that as the way to pitch it to other people that may be interested in the group. Now I know a lot of people on the on the call today weren't, or sorry, we're at the kickoff meeting so I won't go through the overview deck, because it's a lot of over and over then. So the current focus is on is on the best practices and requirements for telecom targeted like workloads and I guess the first thing that ties a little bit into what we're talking about before is like what should this group kind of focus on in terms of the benefits of like CNS. Thank you Victor for creating like this first pull request in terms of the charter. And so I guess I want to like maybe jump into the discussion here is what do people think is does anybody have any other ideas of like what should be included in the charter did anybody not have a chance to create the polar cross that they wanted to. Maybe one question of wookie speaking from Deutsche Telecom. Does the group think it would make sense to to address a bit the cloud native network function. Architectural best practice or architectural best practices from the cloud native point of view in our work. Because if we say just only conformance, then we are having a set of definitions what to test conformance against. But we do not give a prescription what would be maybe a better practice for a cloud native applications to be developed architected or so on. So I guess the idea would be like blueprint or sort of reference architecture of cloud native network functions might go beyond the scope I know but I'm just throwing it here for discussion. This Taylor I think it's at least going to come up in discussions. So I would say it's not that we should avoid discussing infrastructure and and the architecture and everything else. It's going to come up so it would be is it in scope. But at some point to actually try to test and talk about the best practices that would be a little bit different, but I would say it's going to be talked about. As we're talking about the applications running on the infrastructure at a minimum. I guess if I if I could jump in here too for a second. I know one thing that CNTT, and I guess the new and new kid under a lot of networking is talking about like building the like an actual like reference architecture, based on Kubernetes so maybe could you tell how you see the group of the work that you're suggesting in this group as different from that effort because we don't want to like duplicate efforts here. We see that there's something that they're missing. Hey Bill, this is Andrew from Red Hat. Just, just a quick one, I guess on sort of conformance and reference architectures. I think, you know, one. One of the complexities of dealing with telco is that they have many, many constraints around, you know how they produce architectures and actually as we think about the types of applications that will go from, you know, more traditional into sort of CNTF kind of environment. Some of those restraints are going to come with them. Some of these applications do not naturally fit into this environment. Now that doesn't mean they can't be adapted to do so. I think there are limitations for some of them. But, you know, I, I, I guess. Just trying to use my words carefully. Is sort of strict reference architectures, I think are difficult to do when it comes down to telco, you know, the idea that, you know, without a specification. So I've used this example before but you know you take meh for example right it's clear specifications around protocols around interfaces files etc. Without specifications reference architectures are rather hard. And, you know, one of the wonderful things about Kubernetes is it's a framework rather than, you know, a true set of specifications. And I think it's difficult at that point to, to get conformance against very strict reference architectures I think is like being able to provide, you know, guidance I think is very good, you know, in sense of a group that has, you know, good skills, good knowledge and a good goal around what's a good way of doing things from Kubernetes point of view. But I think that has to be taken in context with the application approach because ultimately is the applications here that we're really trying to support rather than just a Kubernetes platform. So there's just a couple of my thoughts. Yeah, absolutely. And maybe I could address like the last part specifically because I think this has been a point of confusion for like quite a few people in terms of conformance versus best practices and this also came up in the conversation in the Slack with the comments from Ian that right cloud native isn't exactly like a thing it's it's not it's not a reference architecture we're trying to match exactly it's more of a spectrum that we're trying to see where we land upon and kind of like drive towards in this case like being more cloud native and so the goal is working group is to help people understand where they are on that spectrum and what would be the benefits of becoming like more cloud native and so I think we might want to move a little bit away from saying we're talking about conformance specifically because I know it's quite a loaded term in the talk industry and maybe move more towards talking about like best practices. Hey, Bill. Yeah, I would very much agree with that statement. Best practices is definitely a key here I think. Yeah, great. So maybe as like an action point I had like before the next meeting. I could kind of go through what we have so far and kind of like update some of the documentation to make sure that we're talking about like best practices rather than conformance, unless anybody else wants to raise their hand and say like this is something that they want to work on. On that we have updated a lot of the existing docs to no longer talk about requirements. So that would be a related thing to conformance. So it's unlike some tests that are just going to be pass fell. This one, you're, you're really looking at that spectrum that Bill's talking about. So any type of phrasing or wording that you see that looks like you're locked in versus it's just helping you to know how well you're doing. And I think this would tie into something Gurgay had talked about in Slack too, and maybe Daniel as well. If you're looking at requirements for yourself as a service provider and saying we need to have some type of integration or performance or something else and there's a gap or some capability that's not there on a Kubernetes base deployment. Then by all means you're saying this part of the solution is not going to be as cloud native that's fine. It's that's not the point the point would be on the parts that that you are there and for whatever you're trying to gain that you can assess that where it falls short then you're going to use other measurements. So this, this won't be what we're coming up with these best practices, you could think of them as maybe more all the car. You're going to go through and someone may gay, say, we're doing great on security for how you're going to handle security on a Kubernetes base platform but we need to do something else here if we want to advance to the benefits and then something else you may say we need to not do this at all because of our needs. Yeah, so I think that maybe kind of like a good like transition into how we're thinking about like structuring the work right now that people can understand that it's kind of more a la carte. And so the process for defining I guess it's not requirement best practices. And so Taylor of I, Taylor and I have created like a template, similar to the Kubernetes enhancement proposal the cat process used in Kubernetes, which is similar to the process used in Boston Python to And, I guess, as we, as we found out the hardest thing in computer science is always naming. So I think Taylor maybe the first thing that we need to do, based off this discussion is actually maybe rename the templates because it's not specifically conformance that we're talking about best practices. Now if anybody has a better name here, Taylor and I tried to come up with a name, but didn't have an idea. And so we want to create kind of like a template to help structure the work and the discussion here that we're doing. So I think, Taylor, one of the things that we need to do is remove conformance from this and maybe move it to like best practice. So if anybody's familiar with the work in Kubernetes, a lot of it now is structured through Kubernetes enhancements, and it's kind of a template to structure like what we're going to be talking about and this is kind of how we see people like working in this work group to propose like cloud native best practices that they think should be And so, similar to that, there's kind of a release sign off checklist. Summary motivation goals non goals, the actual proposal with user stories notes reference constraints references alternatives, the testing plan the scoring and the implementation history. So I'll dive kind of eating to like each of these more specifically but if anybody has any questions or feedback or comments about this process this is kind of like our first like idea of how we should do it so it's still, I think like a little bit work in progress to be great to have feedback of what people are thinking to Can I ask a couple questions real quick. Yeah, we dive in. I'm kind of confused now because it seems like the scope of this is shifted over the weekend. So like a couple things if we're not doing requirements, then what makes our best practices any more than just our opinions written down on paper, like how what is the determining factor to say that this is a good practice that you should be doing B2 if we're just now doing a collection of best practices. How is that any different than you know, Amazon or Google's cloud native best practices that I can look up right now. Like how is this going to be relevant to CNS. What are we going to focus on to say this is how you should incorporate network functions in a cloud native way like I'm not worried that we're starting to water things down because people get offended when you use words like conformance and they might not be conformant, but you know we have like 5000 reference architectures floating around and people forget that the first word is reference so I can go and reference something and do an architecture off of it but it's probably not going to be the same so how do I apply best practices to CNTT's RE2, you know how do I know that this best practice actually maps to a requirement somewhere I should be doing it in the first place. I'll second that I have I sort of my hackles went up a little bit when I heard the term best practices, because that is a very very squishy thing. And I think that perhaps what's getting a little bit lost here is that from the point of view of we can do things. Rather than thinking perhaps more in terms of the thing that we want to ultimately do, and what we need in order to get there. So, there are lots of different answers to that second question different people have different answers. And it might be worthwhile collecting those with some specificity as to what people are actually trying to achieve what problems they're trying to solve by participating in this group. And that will allow us then to figure out what exactly it is that that our group is about. And we've gone back and forth on having hard requirements, and that could go, whether you have some type of scoring system that doesn't that does pass fell or you could still have a gradient, but these are hard requirements. And we're expecting everyone to follow them. I think in the end, and whatever you put if someone says it's not valuable they're not going to do it. So, if you look more at, I guess what CNCF does in general with options it gives you a lot of options the landscape is to help you go in and find different things for categories that you may need. And it's, I wouldn't say that the point with this is to throw out every best practice about any applications running so if, if, if we relook at this, the idea was, if you're going to run networking or a telecom specific what would you take this and then run them on a Kubernetes based platform and get and make it as efficient as possible, leveraging capabilities and Kubernetes and everything else. So that you really need to look at specific applications, what are real real world applications, and then talk about their features and how you may best run those. Let's not have a you in that sentence which is the you that cares that a CNF is, you know, running properly assuming that it's a bunch of microservices with a mutable infrastructure who cares most about that. Because I don't see how that matters terribly much to a service provider who's buying an application from somebody else that it's made from a bunch of microservices. If it's made from a bunch of microservices or not if it delivers the service and it's maintainable and it's operable and so on and so forth, then we're good. But there are a lot of questions about actually can it deliver that service today. And what is that service and is there any framework about what it looks like those are best practices I think be very valuable, but I am concerned that if we skip past that level of requirements the level of requirements that actually matter that it's doing a job that we all recognize, and we jump to it doesn't matter whether or not it does anything useful it just matters that runs as a bunch of microservices then we've, we've, we've like a lot of projects at the moment we've basically just a pretended that we all understand what the requirements are and skip straight to design again. I guess the difficulty with that would be is, you know, how do we determine that it's useful. And so, rather than using we, you know, I'll use we. So how does anybody determine it's useful. Well, maybe subjective based on the environment that it's going to go in. I think there's a couple of things that aren't subjective though, like uptime. If we do something that reduces uptime, it should be next. You know, I mean, there are some core things that I don't think a single one of us would dispute as a core requirement. And I see some really fancy technology that comes out from here and there, and it'll like fundamentally violate one of these core requirements for just networking in general, like junior, you know, network plus type stuff. So questions never get asked of, you know, okay, I did all of this but now my network is way less stable than it was before, should I have done this. I could give you better, well, I'll say better examples different examples right. One is that we are looking for a CNF that can do, at least in many cases fast networking and other cases if you like clean networking weird and networking that Kubernetes familiar with. A requirement of a CNF is that it uses a set of interfaces that it would expect to find in the platform to deliver those that functionality. And without documenting what that kind of functionality is we aren't going to get those interfaces that's one example. We expect, I think everybody would agree with me that CNFs are mostly provided by vendors and that run on a platform that isn't provided by the vendor. So some sort of understanding between vendor and platform would be necessary for that to work. We expect that a CNF is deployable. Now, the application deployment stuff that I've seen in the community largely assumes that Kubernetes actually a part of the application but that's not the model we're trying to follow here. So what a deployable application is what an application is and certainly what a CNF is not well defined in this particular environment. And I said this on the channel and I'll say it again that there's more than one audience here we tend to focus on the telco itself as the only people that matter here. But they're just one of about four audiences that you can reasonably consider while deciding what a CNF is and what it's good for because yeah we need to make it so that they can operate it and that it effectively builds a be a mobile or a cable or whatever network for them. But on the other hand we also need to make sure that an application team independent of that telco because this is not quite the DevOps environment we're thinking of can be supportable that the platform can't be broken. These things right different audiences need different things and they all need to be satisfied for a successful system. I would agree with what you're saying I think as always the devil's in the details is now I mean what's only what we found from doing doing integration and I'll not name any vendors is you know the less complex the application you know sort of as you head from web service down to complex network function is the more complex they get the more integration work is required and it's required all the various levels all the way down to to the hardware right down to we found everything from you know microcode to bios is to all sorts of things that need tweaking. And that I think that's that's the challenge really is where where do we draw the line in terms of saying okay. We're not going to be able to be completely prescriptive about types of hardware or in exactly how it's going to be configured because we'll never have all the hardware in the world test. And we're never going to be able to go and do a we can't have any more layers of abstraction in my opinion I mean the whole point of Kubernetes and others was it's a layer of abstraction anyway we don't want to go to lowest common denominator in terms of abstraction. So it. It's difficult to know where to draw the line I mean one of the things I was considering was if it's as this is an inclusive group. Is it is in a vendor and platforms best interest to to contribute how one would normally configure something now. You know whether whether any particular vendor that sits on top of this is is happy with that secret source being known is the discussion left to have another time. But you know I think that that's one way to do it is make this a group where references are provided as well as we have a framework in which we can receive them. But my last point and I'll shut up is, you know, one of the difficulties is CNI, you know, we, I don't believe we should prescribe a CNI, I mean there are going to be many and they're probably going to be more. And they serve different purposes so that that does complicate things as well. That last point, I think, perhaps the problem you might argue with the CNI is that we've never. I'll take the CNT as an example and therefore probably make a lot of enemies in the process but you know the CNT is busy defining a lot of ways in which you would put a reference architect together, but I don't think it's ever defined what that architecture actually needs to accomplish. You know the requirements came in basically by seeing what OpenStack did and listing them as requirements rather than by saying what does a CNF need to do why don't we ask some CNF designers what it needs to do and listing the requirements. I think you'll get a different answer from whomever you'd ask. On the other hand, I don't think you'll get anywhere until you ask. So, you know, they're the people that care about this stuff to be perfectly honest. They're the ones that need to produce working product if it's a vendor environment. You know, the application designer is an audience here. Yes. One aspect I was in the beginning throwing this best practices or reference architecture in the room. But I really have a same sentiment regarding the reference architectures and these kind of standardization work on paper because it's never going to cover all the use cases. What I meant actually is I fully subscribing to that idea of certain level of conformance testing of the hard, let's say, facts that could be created against some definition, because it's something to that will drive potentially the change in the environment. So what we are facing, and I can also bring a concrete examples, and maybe we can start with the pain points identification addressing the concrete use cases. So we are, as a company, we build our own platform based on upstream so we don't get a vendor to build the Kubernetes based platform. We essentially manage Kubernetes internally using upstream and the ecosystems around it. And we face I mentioned that in the in the first session, we face the issue that when an application vendor comes to us, like, for example, 5G packet core, whatever vendor you take. They have a specific certifications with some commercial platforms, let's say on based on the partnerships but nobody has covering everything. And then we are in the in the position, okay, how do we prove that our Kubernetes is really capable of running that application should be vice versa that application should actually have a conformance test to confirm that it can run in a generically enough Kubernetes and this is what I meant about can we set the set of or can we think of set of assumptions blue even derive them from the best practices to actually make this cloud native network function definition. So what cloud native function can do and what it cannot do if it claims to be a cloud native one example and I think it's not for this session but for for further work. One example is this networking approach. It's a one to one copied from the virtualized world. And the network function skips normally like this big network function skips normally entire Kubernetes networking for whatever and goes out through this Multinic setup. Is this a practice that we want to follow or we want to advocate more for like let's use the Kubernetes networking for that and let's make it say usable adopted and so on. So many other things also kernel parameters bios and stuff that that could be taken into account as a part of current practice and then qualify it as a pattern or nothing button. Something like that. Yeah. I like the conformance very much. I agree but I would just say that each one of those things serves a purpose and the purpose that it's serving. Again, largely there are say two purposes here actually what one is does it allow the CNF to run. Can we bring this down to a checklist of items that the CNF consumes this way that the platform provides it this way because there's a contract between CNF and platform of this is the level of behavior I'm looking for from the platform for this CNF to provide the SLA that it's offering. The other one is where you come into this which is as a network architect you want to be able to link your CNF to the wider network. And that being the case then there's a separate an independent set of problems that says does the networking that we have allow you to properly why this into the external network because again, you know Kubernetes. And, and I'm not going to judge multis and things here but the CNI definition as Kubernetes has it assumes there's a single point of egress for packets from the cloud and that is clearly not what we're trying to accomplish. If we're trying to do complex wiring of network function level traffic into the world. Yeah, but I would touch that I would touch that as well. Because why I mean with a complex environment and networking environment that's definitely one part of the of the blame is on the operators who grown that and then who are running that in the production. And it's something also to be revised in the cloud native way is that actually the best, best pattern to follow, or is it maybe like the is there a better pattern to follow like recommendation that comes out of group or whatever Grimium, and there are some current practices which are maybe not so good button to follow this is what I'm interested in if we can induce some some change into how we produce the things beyond just Kubernetes itself. I think I think that I can understand where you're coming from I think but I mean my background is in networking okay so just to be transparent is, you know, the difficulty with the networking pieces if we try to. I think if we try to make a conformance for more simplistic networking for example in in Kubernetes and obviously there's a good reasons for wanting to do that because it's easier. You know, and probably is less less problematic for everything from coding to troubleshooting the real problem is that as I think is that as Kubernetes is becoming infrastructure in that sense you look at how it's going to get deployed for ran for example it's essentially a networking infrastructure and it's a platform infrastructure it's an orchestrator. The difficulty is that networking goes back 2030 years, and a lot of this stuff is set up in that way for really good reasons. So you'd actually have to go back to the whole networking piece that has nothing to do with Kubernetes and start trying to change that now I'm not suggesting everything should stay the same forever. But that's that that is a very big place to start, and a lot of these applications have multiple interfaces for for many many reasons. I mean yeah tunneling and everything could be made a lot more straightforward and there are things we could do in that area, actually for service providers and others to make things more telco friendly with actually less protocols frankly on the networking side, but that doesn't fit all telcos either. Not all of them will want to run the particular protocols I'm thinking of and not all of them do and never will never religious objection to pretty much. So I think this is where the networking piece is complex because people, you know their networks are different for many many reasons, and it's, I gotta be honest as someone who worked at the itf quite a lot I don't see we're ever going to harmonize that from a Kubernetes point of view. I want to zoom in a little bit here and underscore this point, we're kind of dancing around it and I think it's a big one. We all know that vanilla Kubernetes is insufficient. Right. The difference between the vanilla Kubernetes and the actual Kubernetes distributions that could be used for CNF is big. It's, it's exactly that gap that has to be with networking and not only that there are other things having to do with orchestration and multi cluster. I know some of you are part of the multi cluster sig there's a lot of work going on there and there are a lot of different solutions to that as well. So here's the thing you know if there's going to be a best practice that says well. Some of us are encouraging for example the use of multis that was that was discussed here is this is a practical good solution for a CNF. Now, if somebody's creating a CNF and the platform does not have good support for multis then there's a huge gap there already. And as some of you know, even multis isn't sufficient, right. I was just going to say maybe the point here is not whether multis is sufficient or not but what your scoring system is. What do you mean by scoring. What makes it sufficient. Oh, okay. So, so you know it, it does the job in terms of that that it allows you to connect to exactly as Andrew said these networks that already exist as as other networks in the operating system. It does it does that job. If I'm a CNF builder, as opposed to a network architect, does it do that job. And actually when it comes to connecting to the network I would personally debate that it does that job as well as it could. Absolutely I think things going on in the network service mesh group are also very good critique. The point is, you know, if we as a CNF work group are talking about best practices is using multis a best practice or not you know here, we're moving beyond vanilla that is and maybe, maybe we have to to an extent because this is where I think a CN, somebody who's building a CNF will need some help you know what, what can they do and what can they target. And if they target multiple technologies, you know, that's where we get into the conformance thing right is this work group going to say that the best practice for example is to use multis and what would be that the implications of that. And this connects a little bit I know in an earlier telco user group meeting, I talked about creating another work group having to do with multis part two to kind of fix some of the orchestration issues having to do with multis. My initial question though, like, comes up you say that like multis is sufficient but like, what are we basing sufficiency on. I'm just going to keep hammering the requirements thing online and you guys can tell me to shut up but like, yeah, but like, I don't like in the CNT reference architecture one of the debates we had early on was you know they put a requirement done that says I need to see and I multiplexer. Well is that a requirement or is that an implementation slash solution to meet an actual, you know functional or non functional requirement of multiple interfaces or packet treatment or whatever right like, like I said, even if all we do is prescribe best practices. What are we judging the quote unquote best part on to like define that. And that's the right now right I'm trying to keep it high level it was just an example but, but this is. This is this is sort of from Jennifer I could try to interject as to what is the best practice like for instance, a simplification of the configuration could be one, one characteristics of the best practice in order for it for a CNF to connect to multiple networks, using multis can be a very complex configuration. Okay, so now that the key is CNF needs to connect to multiple networks, so take 5G deployment 5G ran deployment. When you're running a DU functionality or UPF functionality it needs to connect to G node be it needs to connect to AMF function it needs to connect to SMF function. These are all CNFs or or or Kubernetes applications running at different parts of the cloud. Right, so the requirement is these guys need to be able to communicate they need to be able to discover each other they need to be able to connect with each other. Now, trying to bring in and saying oh multis is the best practice why is it the best practice it's not it's it could be a fairly complex thing to manage that, or even to configure that. So, so why even go there as a part of the best practice to say use multis or denim or or whatever practices. Why not just stop it there and say that the platform must provide an ability for a CNF to be able to connect to multiple networks and stop there. And, and, and in terms of in terms of the best practices you simply state that it should be very simple to configure and it should be automatically so that you can auto edit. So, otherwise, plumbing all these pieces is a pain in the neck, even if you use multis or use any other multiplexer, don't even why even go to the multiplexers why multiplexers. Right, so if I think I agree, this is kind of what I was alluding to before is that, you know, as it's a sort of a plug in architecture on one level is the people who provide these technologies so let's you know we're talking about multis so let's say multis is, you know, a sort of a reference implementation but best practice implementation from the point of view of multis to cope with is the type of configuration we recommend in the following use cases. I mean that that's, that's kind of what I was alluding to before is taking, I agree with you it's like taking it from the application point of view where do you stop it's like well, kind of stop with the applications integration at some point and if you have methods of doing multiple interfaces or you know whatever else you want to do with it then then people can contribute. This is best practice for that particular technology because there's, there's going to be more of these. No, but, but, but, but I mean, if I could just pause the conversation for a minute here. I know we've heard a lot from the vendors and I want to thank all of you for your perspective. I want to turn it over I know we have quite a few service providers on the call to so I'd like to hear from them too so I know we have digital calm Swiss calm bell Canada charter. Sorry if I'm missing anybody but does anybody from one of the service providers, you know we've had quite a few different conversations want to speak up and provide their input to. This is Ruben from she's come speaking. So I'm fairly new to that group. So bear with me please. I was just listening the thing on on requirements or my view is that conformance without having a shared understanding of what are the assumptions is really difficult, at least for me right. This would be one thing right without necessarily going into the details if the design space is infinite because anybody can do whatever he wants. That's going to be very very difficult. I hope I could express myself properly here for me this would be one input right I mean right now with conformance I would not even know where to start. I would not even know where to prioritize the design space. Does that make sense. Or do I need to try it differently. It made sense to me. And then the other thing I can say I mean, I think there was this debate about requirements, or how precise we want to be. I think at the current time when I see the status of the technology and how fast it might be moving I'm not sure whether we can. I'm not sure how much we can look into the crystal ball and really make very very strong requirement right but as I said, at least from my point of view, at least trying to restrict the design space would be useful or at least giving good, you know, good directions. That that that's all that for the moment. Thanks. And so if you're saying if we could restrict the design space, I guess, like what would be helpful to you as an operator. At least, so my two cents if I come back on the on the whole networking part or I would say the more. I mean we have this discussion around vanilla Kubernetes not being enough right. I mean, here at least we could try to make it a bit more. I think we should try to make it a bit more. A bit more precise right. Because again, this is sort of it's true everybody understands that. But there are so many ways to solve it right. Now, that might be clear for all of you. And maybe just me not knowing the overall landscape enough that could be. But that could be a place to start right. Yeah. Does anybody else want to speak to that point. More generally I would say an effort to map terminology and language to what you're going to find in Kubernetes. And right now I'm going to keep leaning into Kubernetes and sort of general cloud native. It's fine to do this and we want the principles. And when we talk about something we can actually measure, we'll have to get more precise and talk about each implementation so starting with Kubernetes. And if we take language and the talk of space and try to map it. So, one of the examples would be what is a Kubernetes work like what there's actually definitions around that. And then you can map those out and say, well, how does this relate to the talk of work. So you could talk about the a network function and say, well, then what would that be as a something running on Kubernetes and and start looking at how that ties in. So then you can actually get down to a specific application say this telco application, which would be a Kubernetes workload could be deployed in these many different ways so what are the best practices on those things. And then it has various features that it already potentially already exists because it's already been implemented. So how would you harness those or use those they may tie into using CNIs or maybe or some other networking or maybe it's totally different features like storage. How are you storing your data is are you handling data on Kubernetes in a way that it can help you do recovery and and everything else. Or is Kubernetes going to get in your way because you were you've implemented handling state in a way that you're going to fight Kubernetes. So I think there's, once you look at a specific application you can actually break it down, and there are existing best practices and maybe multiple choices on on each feature. And probably starting with the ones that a real application and breaking down the features and actually sing, rather than say let's let's pick one particular thing that has a gap on Kubernetes and figure that out let's start looking at best practices and a real application and and see what comes out. I'm also adding I know there's quite a bit going on in the chat to in terms of conversations. Thanks for everybody that's adding there, like one of the points. One of the things that is that and if he didn't work out somewhere between working working out very well some people call it feeling like our goal is to help how can we make this transition better, and that's what this working group is trying to figure out and so what are the outcomes were we're trying to have. And what are the operators that would like to kind of provide their input or opinion on where we should go. Dan from Bell, so I've been vocal around the fact that I don't think like a reference architecture that everybody's going to be applying to will work. I think there's too many fluidity in that ecosystem to think that you're going to be able to align on a single platform that agrees on it. I do agree with Ian's comments around the contracts. I think that's the, I think that's the one we need to look at more finding ways to abstract as much as possible the CNF requirements from the platform. This way you're able to be connected decompose it differently, and you can have like fluidity in that space as well. Aside from that yet think from an operator perspective cloud native is not that we just want to have a package CNF or package container workload is we want to be able to leverage the capabilities of the that the cloud can offer us to do those services. So rather than have for example complex control and sync mechanisms between functions can we leverage gremlin capabilities. And then we leverage any cast services or capabilities that we see in cloud to offer some of our workloads rather than having complex orchestration systems to tie those together. Those are the benefits we want to get out of this and I've been vocal about the fact that I would like that working group to help us find new use cases are new ways of building functions in a network that doesn't become like a big clutch big fat container with six functions because we don't know how to integrate them together and make them like a more integrated into a simpler way that's from our perspective that's the way to go. Whether I end up with my own, if we're able to have that abstracted concept that we can have that fluidity in the in the past, the single way of doing it. And still now we have that same challenge. We try to get to perfection too fast in those kind of framework. If we aim for perfection and that kind of in the same ecosystem for CNS. Well we'll end up again eight years later, not having a real like industry approved way of doing those functions and something that has happened unique or knows are some new ways of doing things with asics. And the end will have the same results. I think the use cases and reasoning behind is all part of the proposals that at least the process that's there right now. And says here's a best practice that they think should be, or their offering is a good thing to follow. Then one of the things in the those proposals is to say why. You could talk about the capability to it allows you to stitch these different pieces together and lowers maybe the efforts for the developers, as well as the operation side, whether that's collaboration between vendors that are helping to support service providers options. But they know that the applications are following some best practice that they've already are familiar with say Kubernetes feature that helps to provide some something for each of those applications that fit together. And then if you look at some of the comments in that chat about like own app and OSM and stuff, you could say a an application that's helping to provide maybe enhance enhancements are leveraging the capabilities of Kubernetes for some type of business view on the orchestration is likely to tie right into the monitoring and status and other capabilities of Kubernetes but let Kubernetes do a lot of the, the legwork behind so you're, you're not trying to force things to certain way but just nudge and guide things in a certain direction. So I would say it's going to be a tie in with the applications that are run well on Kubernetes and allow you to monitor and use the status that they provide across the platform and then mainly let it be automated. I think we have a, I agree a bit Taylor I think we have a kind of a short term problem and a long term dream that we need to fix. So right now, most of the implications of CNS we see coming are kind of trying to grab the same things we were doing in VNF the same capability the same requirements the same infrastructure requirements we have and you need to put it together. SRUV new map ending all those things need to come up because that's the way we use from existing code the VNF the VNF vendors come from existing codes to what it's going to be happening right now in CNS what I'm looking at that some point is, Well, is it required. So I understand that some protocols are missing in community so it's not was not built for some of some of them that's okay multi interface. Do we need multi interface right now we do because the code we get comes with it, but at some point if I may have going to build a CG not function in two years do I still need multi interface one for management and something and do I still need that or can we look beyond that because you don't think about this but adding that multi interface means you need to find different ways of doing network segmentation in your cluster which is oh by the way still not there. So, you're, that's the trick that to get all those things working because we try to integrate them as they weren't in VNF at the end we're still eight years and it's still going to be other other things not working. And that's where I'm afraid of it the time is going to take to align on this over time, rather than look at the kid this is the ones we get for the next 18 months but if somebody is building a new CNF in 2022. Why can't you look at a different way of building it and this is why I'm looking from the cloud communities, different ways of doing this, and getting experience from that that ecosystem to look at a we can do it differently. And that's why I'm, I'm hoping from that effort will help out. I wouldn't, I wouldn't, you know, some of the some especially around the multi interface piece, I wouldn't have any of us take this too lightly in terms of being able to change that I mean some some of the separations that are required by networks are regulatory. I mean, they're not even in the purview of somebody who says as a developer I wish to do it a different way. I mean, you know some of these problems with telco are profound. And I don't think that should be underestimated in terms of how we wish to design CNCS or CNFs. It built we're almost at the top of the hour. I just want to throw in one more proposed outcome from a provider. And this is going to be way less sexy than a lot of the stuff that we're talking about but I've been pushing kates out into our production network for like the last couple years, and I think the vast majority of us on the provider side are probably people like me who work on like our development and office at the CTO side but like engaging with operations. And doing things at telco scale, you know, without crazy things like pneuma pinning and SRV but like, do I give every single CNF its own cluster. And at what point do I start having, you know, an almost equal amount of redundant master and Etsy D nodes that I do worker nodes. How do I scale this at the edge. How do I manage, you know, multi tendency right like if I don't give every single one. And then this comes to Ian's point about, you know, a single egress right you. The problem with these reference architectures right is it's looking at a single piece of a stack and then all of a sudden I go one layer out my network and I have this aggregate firewall. I've opened up one port on this firewall for a subnet that all of these, you know, things in a shared cluster are now piggybacking off each other and there's just all these weird operational challenges. You know, for running, not like even the most basic network like control plane function stuff and just regular web scale stuff like we don't tend to run like two to three enterprise applications we run literally thousands of applications for lots of different people, video applications. Right. I'm really hoping one of the outcomes of this isn't just the sexy stuff or how do I get like a packet core to work efficiently in case but like, how do I run like 5000 clusters with like 5000 different firewall rules, you know, in operations. Well, thank you for that. I appreciate everyone's discussion I think it's been just as lively as the kickoff meeting I think it's great that there's so many different opinions. Everybody can make this meeting should also be great to move some of this discussion to get hub so if you have an idea, or if there's someplace you'd want this group to go, I'd encourage you strongly to create a PR against the repo and then we can also have the discussion there. I think both the at the meetings and online through GitHub we can help kind of move this forward. So, with that I really appreciate the discussion for today and I hope you will. Continue to contribute to this group. Thanks for joining today. Thank you, Bill.