 Good morning, Oliver. Good morning, Lucina. Good morning, Victor. Hi, Lucina. Good morning. Good morning. Good morning, Taylor. Good morning. No worries. I think we're just going to get started. Are you available to lead today? Yeah. No worries. We're just getting going. Cool. Does anyone have anything to add other than maybe going over the core question, looking at the latest. Best practice. I just just put like a one topic that. All right. My attention, but yeah. Confidential computing. I wanted to raise the. Topic of the. The co-chairs. And the voting. I put something in slightly of the week. I'm going to struggle to make this time. Regularly. Due to a new project commitment. And it's been over a year since. Since we. Victor and you and I were voted as. The co-chairs and I think that term was supposed to be a year. Put up another. I think we tried to do it. Something happened, but we can. Go ahead. Okay. Cool. Okay. Is there anything to add to the events. Before we move on to. What are your thoughts on the full request. Well, basically. Just you remove the. The open source event, right? Like. The last week we have the. The one regional summit and the open source on it. Yeah. And I was at. It's not there. Okay. Yeah. Yeah. It's on the list last week, but not this week. Did you, did you attend that? Yeah. Unfortunately, I couldn't attend the one summit, but I had the opportunity to attend the. The rest of the open source summit. Which was great. And that's why I read about my. My topic in the, in the agenda, but yeah, we can talk later about that. I haven't seen any other new, new learning regarding. Okay. So what was the, what was the open source summit like generally was a much talker stuff there. Or talk it wasn't. It was good. I mean, I didn't find anything. Like different, like. Most of the regular. Topics were, I guess, folks in security probably. In general. But yeah, regular attendance, nothing, nothing huge, nothing lower. Yeah, I guess as, as, as usual. Any other event updates? Trying to think if there are any CFPs to close soon. I think. Yeah. Okay. You call Europe CFPs are open. One summit. North America CFPs are open. Well, maybe a regular regional event probably in, there would be a key CD in Mexico. Has been confirmed yet. But yes, it's just more regional. Okay. What's that I have the information I will, I will chair it. Yeah, cool. Thanks. Okay. No open pool requests. So yeah, the. Single concern for container one was. Merged, which is good. And a bunch of things. Was there, was there an issue or a PR you wanted to. Review in particular Taylor. You know, no PRs. And I guess that's there. I don't think we had a. CNF working group. With the. The best practice that was merged. So maybe just going over. I'm not going to look at it. Would be the next thing. Okay. Since we didn't have anything actually on a call. Gotcha. Okay. So is this one. Number five. So do you, do you, or. So I know you and Victor and Oliver did most of the work. Together on this. One of you want to run through it. Yeah, I think so we had. Yeah, he was the first one to get it. And then we were, Oliver and I that were going back and forth. And trying to get a large amount of the content. And then we kept getting input from. Yeah, there's a good number of. Comments and suggest edits that ended up. Pushing it. Over the finish line in the board. Anyways. take a stab at summarizing what we've done? Well, I don't know, like, so, yeah, basically, do you want to describe, like, provide summary about what is these best practices for, or? Yeah, just summarize it, since, I mean, we all talked individually, but we didn't have anything on a working group call, so just what's the best practice in the end, and any highlights on, I think there were some notes, caveats, and then just some notes and stuff that we ended up merging in at the end. And then maybe we look at what's the next best practice for? Got it. Okay. Well, just in summary, the best practice is trying to suggest to you, to define, like, putting just a single process type or container in terms, like, a design of the CNF application, and it's highlighting the benefits. We have detected multiple benefits in different areas, like, for example, the way to upgrade the services, like, in terms of security, we're reducing the tax surface, and controlling the observability, and so on. Like, there are things, political things, as always, like, and also, like, again, like, another thing, we're not trying to define exactly the way to define things, like, we left some room for architectures, architects to define things, but general is that that's, that's the best practice, like, suggest to have, like, a single process type per container. There is also a good references at the bottom where you can also get on deep, like, providing, especially the benefits in terms of performance, like, what else, how others in the industry are using the same things. And if I remember, like, two weeks ago, someone was also using some of the information in LinkedIn for providing similar things, right? I don't remember who was, like, referring that one in one of the articles, but I don't know, Oliver, do you want to add something? Not so much about the best practice, I think you covered that pretty well. Maybe just a reflection on how we worked. I think, you know, we, I don't know if we can replicate that. I mean, it would be ideal. I think I know we tend sometimes to be more than three people in this call. And I think, you know, if we could figure a way to scale that a little bit. For me personally, I'm a little bit maybe less, less experienced in writing these best practices. So I found it very useful to work together in, you know, having a couple sessions. I think we had generally one session per week where we tried to, you know, in addition to the Monday call, which I think is, you know, we cover, we cover more things. And I find it, you know, people need to have an interest, obviously. So sometimes people are here to listen. But if they're interested in driving one of these forward, I think doing that alone or just doing that sort of asynchronous can sometimes be a bit of a struggle. I found us having lots of conversations. A little bit of it was word smithing. But I think it, you know, and it also helped to, you know, allow some learning of areas that you may not know that well and sort of then from there be able to contribute. So hopefully we can do something similar as we, you know, on the next one, try to find a way to say, you know, there's some working sessions. And if you want to get involved, you know, that might be a way for us to get some, get more momentum, just a thought. Oh, yeah, that's true. It's a good point, I guess. Yeah, that covers a lot of good sessions, working sessions that we had. And once that we create like a good draft and we publish, like, it was nice to receive a lot of feedback from others, like, for example, Jeff, just give us some additional things. I think part of it is just familiarity with the process. Like you were saying, Oliver, that you weren't as familiar with writing a best practice, but I think it's just partly the format in general, as far as being more formal and everything. Unless you're doing that type of documentation, you may be familiar with an area of the content that's actually trying to format it. And, you know, we did this on purpose at one point, trying to cover different areas. It is more formal than some write-ups. We look like 12-factor instead doesn't look as formal as that. But the points are there. We were trying to cover different areas where we've had questions in the past, in the talk of music group, different white papers. So trying to cover all those areas, if we decide that the format isn't for us, that's fine. But I think the content that we cover is good. If other folks are interested, though, in trying to get specific best practice or passion about our using and work, trying to solve problems, then any of us that have already gone through this can try to help once it gets going the momentum. So that's, I think, what we need to do next. We have this one related to, it's more general practice, I'd say, but it's the principle of a single concern for a container. Trying to, it's related to microservices, it's not just that. So there's a microservice architectural staff in there. And then that principle for single concern is, it could be applied in other areas. We're applying it to containers, the idea of having a focused area so your team can work on it and all the other things. So we have that sort of thing. And then there's other practices that might be more like specific. When we do the no root and the processes in a container, that one's like a more specific. And, you know, we can go either way. But does anyone have any specific ideas that they want to work on? And we do have open issues. So we have open issues. We have the discussion board area where there's a lot of different topics. Some of the things are more multiple best practices. And I hold discussion, which is fine. So if there's just an area that you care about versus a practice, that's fine. We don't have to pick the practice today. But if we know an area to focus on. So now that I am seeing what Tom is sharing, maybe I think that we can close the best practice. I mean, the best practice, which is referring about the single concern that is not closed yet. You see what I mean? Because I guess the criteria that we have was based on the, or at least the criteria that we use for creating the next PR was based on the number of discussion. And that's why we pick the single concern. Yeah. Now that it is on, yeah, it's time to choose another one. So yeah, I think the answer to your question is yes. So regarding the topic, Taylor, well, probably I can share what I heard nicely about the topic that I bring into the agenda. At least for me, during the open source summit, there were several sessions talking about confidential computing. Maybe it's not a new concept because yeah, it's regular practice or like regular technologies that have been existed for a while, like Intel SGX technology from others. So, but yeah, I just captured my attention, like multiple people talking about confidential computing, like sharing like benefits, cross cons, technologies, projects, at least, at least from my point of view, my main takeaway of the conference was a little bit around confidential computing. I think people were categorizing confidential computing in three different pillars. Obviously, the first one is storing the information, like encrypting hard disk. The second thing is about the data, how the data is in transit, like using secure protocols to change information. And the last thing was more referring to the runtime, like things like how can do confidential computing in the CPU and the different technologies that they can use. So, my point or at least the point that I'm trying to bring to the table is like, should we have any best practice in confidential computing or like do we have anything related with confidential computing? Maybe I can raise the topic in the discussion or things like that. I don't know if there should be a best practice for that. But yeah, just one topic that I brought my attention during the open source on it that I just wanted to share. I just included like a regular wiki entry. But I guess there are plenty of more useful things or like information about this particular topic. Because yeah, as I mentioned, the concept has been for a while, but it was interesting that people were like multiple tracks or session were talking about the same topic. So, do you reckon there's any best practices from a CNF point of view when you know we could create an issue for user's reference within an issue that we've got already? Okay, yeah, but before creating like any issue or just moving forward, do you think that it could be a good candidate or like it's a good idea or like are you already heard about anything about confidential computing at all? It's not something I've read into an awful lot, but I think security is a key normal functional requirement of technology in general, but certainly in telcos. I think it would be good to include some security best practices. I don't know, I'm a bit nervous about doing that because there are a ton of other security best practice sources elsewhere. I don't know if there's something that's kind of specific to CNFs or type of applications that we're doing where confidential computing could be something we recommend or some of the techniques. Yeah, because I understand about the topic regarding the techniques. There were like different technologies where like for example, some of them were like securing just a container, others were expanding the scope like not only the container, also the whole bot and others the worker node and the last thing was another techniques where projects was considered like securing all the cluster. Of course, there are a few open source projects like they are trying to tackle this particular problem. Yeah, what about use cases? Were there use cases brought up or do you have specific use cases for networking telecom related? Well, the only thing that came to my mind is like for example, maybe one of the core network components could be, I don't know, a UPF or one of them which requires some security addition into the, I think the way that they call this like I'm not sure it's secure, runtime secure, like when they are like loading things, registry or things in the CPU, they were ensuring that those that information is not accessible from upside. But in these terms, in this particular case, it's security at the CPU level, not like so I'm not sure if one of the core network components require this level of security. But yeah, I can double check like maybe it's a good question. The diagram to me feels weird where it says, if you use a confidential computing library, then you're safe all the way. But otherwise, I mean, it's talking about trust boundaries, I guess, but at some point you got to pass it up the stack to get it like some information is passed up. I guess it's saying that you're going to try to keep everything at the application layer, the application process at some point has to access. I don't know, it just kind of feels weird. Let's see, like when was this edited and who is, wow, there's a lot of references to this. Yeah, that's why I guess it's not a new topic, maybe just like a So all of the providers are it's just talking about hardware providers. I don't know, I have a bias, I guess to feel like is this just promoted by all these companies that are trying to sell their chips versus why aren't we just saying zero trust like zero trust is about trying to it's to me, zero trust is really just the concept that some security people have been pushing for a long time. Don't trust anything at any, at any part of the pipeline or any, anywhere you are, you're all you should always whether in development, you know, it's or testing or anywhere else, you're going to always try to have security built in. If you have it baked in at all layers, then you're going to be better off than saying we got it covered at this one spot. I haven't looked at this in a while. I know some about the trust is competing but I know that there was in the last several years there's been like exploits on some of the early ones like Intel was you know one of the first doing the hardware and if you gained access and I think there was even like some security bugs that went into manufacturing, if I'm recalling correctly, but the like where people exploited in the manufacturing process, but there was some other stuff that made it out into production and when you get access then you can you get everything. So just saying that it's covered is I don't know I have a bias against it. I think it makes people feel like we're good now, we don't have to think about it instead of thinking you should do this all the way through the whole stack. Security through the whole stack, you know, automation through the whole stack, development all the way to production, you should in my mind it should be a holistic approach for everything. Okay. I would like an examination how confidential computing and zero trust and stuff like maybe a comparison of that would be good. Yeah, especially to find that use case like that. Is there a use case for that? Yeah, that makes sense. I could definitely see it as part of it. It seems like it would be one part and you have the use case and you say we're using you know, different practices from confidential computing along with other things. Yeah, the question is in different order like maybe confidential computing is the solution, but there's a solution for what? Right. Well, that's that's one option. So I didn't realize the confidential computing consortium is a Linux foundation. It's related to Linux foundation. I don't know how it's related. Well, one of the solutions is data containers. It seems to be offering like a part of one of the solutions, but yeah. I dropped a link into the notes. Oh, nice. Yep, to that one. You can look at the members. So it looks like cloud providers are listed along with the folks like ARM, Intel, Red Hats there, Microsoft, a bunch of security companies that I'm seeing in the second level. They're a coops. Wow. Accenture, they're a premier member at the top. They probably have a lot of the government contracts that are going to be more. If you click on about, then you get a member. So Accenture probably has a lot of the government contracts, which are going to end up using more of this sort of type of security, just have it built in, including systems that have no network, no external network. Meta has a lot of government contracts. Yeah. Looking at the projects that are listed, it, from what you were saying earlier, it feels like they're part of the zero trust idea. So how can you put things in place to trust certain parts of the stack? You know, like attestation of the hardware, for example. That's making more sense. Yeah. She get the Mexico KCD in Veracruz. That would be good. A funny name for the project. Yeah. Nice place there. Okay. So do we want to do anything as in have an action or actions about this? Probably. I can try to find like a use case. I can rule something with later, and just in case that you want to consider that. Yeah, I think it'll be good because I guess it's kind of tricky because things like hardware attestation, you know, it's relevant to telco. It's part of, for example, in the UK, the New Telecom Security Act. But it's hard to, in the short period of time we've had today, me to work out what we might be able to test against a CNF or a Kubernetes platform layer, you know, at that kind of level of the stack. So I think if we were testing hardware attestation we'd probably be trying to load some unsigned something or other onto a server and testing that it's rejected, which is a bit below the Kubernetes layer perhaps. Okay. What about some other ideas for areas like we've been talking about deployment automation? Yeah. You know, this could be stuff with projects like Nefio, Flux, CD, and Argo CD are being used for automation in networking telcom stuff. Butch of Telcom, they're using Flux with GitOps patterns. I think Barange might be using Nefio, I can't recall. So it seems like there should be something around automation that would be of interest. Does anyone have anything specific or an area that we could focus in on? Oh, yeah. That's a good topic. Like deployment is could be one option, like defining best practice for deploying CNF. And the other thing, I don't know if it could be part of that best practice, but also the way to package things. I'm not talking just like putting in a container. Because there is like a good discussion about using Helm charts versus other things, like especially KPD is like the kind of a new approach with Google is pushing too hard to have it. Obviously, there are pros and cons with the existing solutions like Helm or Q2Mize. But I guess also it would be a good opportunity to define new ways to package applications. Probably the question is like should Helm is still a good way to package or like what is the best practice to package an application? How to offer that CNF to customers and consume the best way? Yeah, I think that would be a good one to cover. Because we could then either include in that or link to something around customer resources and operators and that framework and that concept. Because Helm can play a part in that. But it could be part of a kind of wider maturity scale we look at. I think that would be good. Yeah. Another hot topic about different communities trying to talk all the same thing is about defining the hardware requirements and all that on and between trying to put in something in the CSR packages. SILVA project as well has a proposal to define hardware related things in the CNF. And eventually the FIO is trying to do similar things like defining what are the hardware or like the software dependencies as well of the CNF application. So I don't think that we have a best practice in that area but at least it's a topic which different communities are trying to address from their own perspective. So that's more around the packaging rather than the automation method? Yeah, that's right. Should we start a couple of discussions on GitHub and see what the uptake is or maybe just create the issues and I don't know what the preferences around how to gather opinion about what to focus on first? Yeah, I guess it's a good idea and that's where we can sense the the the the community is more intense. Has more interest? Yeah, we'll start a discussion on GitHub and then post it in Slack and elsewhere. Try and gather some opinions and thoughts and and then maybe create an issue and start PR for something that's most popular. Is there any challenge Tom that you're seeing at Vodafone where if people were following an area like principles it doesn't have to be specific practices or specific practices either one that you would find helpful or is there a challenge that y'all are having that maybe we could dig into and maybe we end up finding practices that help either way? Yeah, so I think the main challenge is we have we we shared in Amsterdam along the lines of there's a there's a kind of different set of delays between a community Kubernetes release and then so a community's Kubernetes release happens and then there's a different timeline between a platform vendor making that available in a version of their product and the CNF vendor having tested their applications against the new versions of the APIs and the new features in that new version of Kubernetes and then onwardly the new version of the product from the platform vendor and that's one of the bigger challenges you've got the other challenges are around kind of the automated testing of validating that new stack if you like so CNF version a running on platform version b which uses Kubernetes version z and you know cni option n and how do we how do we kind of get to a point where we can automate the testing of that validation without being too opinionated about what each of those version numbers are or particular choices of subcomponents of a platform so if someone needs to use Calico for the cni or someone wants to use Antrea or someone wants to use OVN or someone wants to use Flannel we can still test and validate that that that feels to me to be one of the kind of the main challenges there's a bunch of other challenges that we've got but many of those are organizational and you know modernization things I think that the thing that introduces the most delay between you know starting a thing to launch something and it being launched is it's kind of testing and validating that complex set of options you know features and versions of features and so on I'm a bit nervous about trying to be too opinionated on what those subcomponents should be and rather make sure that the framework for testing is flexible enough to allow you know the each each telco and each service provided to decide what works for them so integration testing and like I guess individual testing you need both for those different options so as if you have something like a Calico feature for some type of network setup and you need to make sure that that keeps working and then you need to make sure that it works with the new version of a cnf so there's seems like there's both types of automated testing the individual whatever that's going to be unit unit api whatever and then some type of integration type testing and then you know yeah for all options it's it seems like there could be so we don't have to talk like opinionated you must run calico feature x because it made it did something nice or whatever but maybe there's something around um being uh supporting that automation automation of testing so if you there's something around like how things fit together as far as like the api apis um being exposed and um providing on the integration testing I mean there's if if you require a someone to come in and manually run through like large datasets and they have their own special tools and that's the only way to test um that would be a problem so there's got to be something around that where we can talk about um providing like some capabilities so something beyond just your liveness and readiness pings so some you start doing something else exposing something else so um I'm not coming up with like a specific best practice right now tom but I think there's something there where we could look in like how do you make this possible for testing and then maybe maybe there's something about just the testing in general I mean like there's it's not saying testing is the best practice but something with relating it to um your test cases and stuff we should be able to run those in the pipeline or you should be providing test results so if you're doing like a complex pipeline where passing between teams instead of you know saying here's the operations team with their pipeline we take it we onboard it but you may have like development to production teams and now you you have to say exposing the results well there's some companies that are looking at taking from the development from an external org so now you need to have did you do you have test are you doing sign off so if you're saying an an image has been signed off as secure so there's those sort of tests can you also sign off here's the test results and then as a consumer you say those test results are covering what we want and it's signed so we can move it forward I don't I don't it seems like there's something that we could pull out of this practices yeah I think there are um I'm sort of thinking back to I think I shared it and it's in the manifesto thing that we're working on elsewhere but those cnf automation use cases and um that they're kind of always the starting point for us about what's important and how do we how do we improve what we're doing we want kind of automate these lifecycle management operations and then I guess it's trying to delve into each of those and think what is it you know what characteristics of a cnf all the way it's designed or built or operated are things that become a best practice that then mean that we can automate those lifecycle operations and I'm not sure I've properly done that that next step of the full process yet um but yeah okay well I think what was the first um challenge I I didn't get it typed into this uh before you're done I was hearing it but it would kind of go on since then the first challenge before the automated testing I'd like to summarize that one so it's the it's the um the difference between uh the matrix of kubernetes versions that a cnf vendor supports for a given cnf version and the matrix of kubernetes versions that a platform vendor is offering at any given in time at any given point in time from the upstream release that makes sense so you know one month after a kubernetes release I'd expect neither a platform vendor or cnf vendor to have updated anything 12 months after kubernetes release you know a platform some platform vendors are just releasing a product that includes that version whereas the cnf vendors are thinking well we wouldn't need to start removing that version now so it's a it's that you know there are sets of kubernetes versions that each vendor supports or provides and trying to align that set of versions up is challenging yeah I wonder if this is why some csp's are starting to move towards uh move back towards I think would be a way of saying it bare metal deployments uh running their own kubernetes version so that they can run more recently production supported upstream releases of kubernetes and get the benefits there and then have the cnf's running on that yeah maybe it's it's one of the classic trade-offs you know if you've got the engineering skills and resources to be able to do that then that's one of the benefits you'll realize but if your general mode of operations is to buy products from vendors then you may not have that that engineering skill was based to be able to sort of run your own kubernetes platform not not in the short term you know yeah it seems like with some of the it seems like the csp's many of them would be moving towards teams have the capability to support it if you're looking at building out the expertise training internal operations teams for supporting multicloud so if you're looking at um you know amazon and azure microsoft azure and whatever else and you're running you're trying to run on multiple clouds then you're going to have to have the expertise for monitoring and stuff like that and unless you're dedicating which some of them are like as i've seen some csp's that are more in one but otherwise if you're doing multicloud you're going to have to be building out the team that has that like operational understanding for managing like the life cycle and if you're looking towards that uh go ahead vector no no the thing that i was saying is um what the at least the use case that i have heard is um when you have for example so for your core networks and you are using different components of different vendors and those vendors have different requirements so if you're using amf from one vendor and the sm smf are from other one so eventually you can have that situation when you have some level incapability even if they're exchanging messages using the the standards and all the protocols and following that see uh um combines and everything like that so so having those requirements how can you deal with that and especially now that you have like a Kubernetes and up cycles um that also increases the chances to to to run in those particular i mean i guess it's not mainly by um technical limitations of the group it's mostly compatibility between between different cnf it seems like uh let's see um possible issues so compatibility of requirements between different cnfs so it seems like that's one that you're pointing out so that seems like an area also where we could look at best practices so target how do we increase compatibility yeah it is one of the things that it is one of the things that in afio they're going to work on that in this particular second release like having uh one of the components with free pi gc and the rest of the components with open open a year open um oh yeah i um because it's open air interface i guess that's the name of the project i think this is it's good to delve into i think the challenge is going to be testing for that compatibility you know one instance that sticks out in my mind is uh you know two different products from the same vendor you know this is something we've seen in the past where um one of the products uses multist to provide multiple interfaces and therefore um because because those secondary interfaces aren't kind of first-class citizens and Kubernetes we can't use network policies and therefore we've got to use um some kind of fireballing technique outside the Kubernetes cluster to protect access in and out of those secondary interfaces whereas another product unit in the same company would would achieve the communications flow via a different rate and and sometimes the requirement is a you know internal CSP security requirement that drives one or other of those behaviors and so you know some of the challenges we've seen aren't necessarily compatibility in the kind of um sense of the word of meaning that you know CNS can't run on one or other of the platforms it's that we're getting inconsistent um I guess inconsistent requirements per CNF if you like so it's it's challenging to test that sort of thing um in an in an automated fashion because it's it's probably quite specific to the security requirements of a given CSP or machine that particular example awesome um I wrote down this in the notes um this seems like areas to dig into there's some deployment stuff I'm actually I'll just put that right in there niff I know um I think that's also here um there's probably some get-up stuff matrix supported I don't know exactly but I think there's probably get-ups okay those are like relating it to challenges are good um Victor if you think of more challenges and stuff to dig into the use cases so maybe Tom if you could uh talk and um talk with some folks and we get some specific use cases on either of these areas written up then maybe that would be something to do um yeah all right we made it yeah thanks all thanks everyone okay cheers see you next week cheers bye thank you bye