 Hello, good morning. Hey, how's it going? It's all right. You went to Monday. Everything's busy before the next QCon. Yeah, I don't doubt it. No, I didn't get funding, so I won't be attending this year. Sadly, it's going to be my first one in forever that I didn't get to go to. Oh, that's too bad. It was a ton third. I'll post the link. We'll give it a few minutes for folks to arrive. I've seen a lot of posts about the Dell Toko lab and various things happening there. Are you engaged in that directly? I'm engaged in it indirectly. I could chat offline about it a little bit. Okay. I have been marshaling some support internally, though, around specifically cloud native endeavors and telco. We're starting to see some CNFs that are actually, I would say, as close to cloud native as I've seen so far. And it definitely breaks a lot of people's brains when trying to design infrastructure for like an actual CNF versus, you know, an 8GIG monolith where you do a one to one box to application ratio. That's cool. It seems like something to dig into and talk about more. Where is that? Where can they run? Or where are you seeing them run? I'm seeing the CNFs. Yeah. I mean, so this is mostly a packet core that I've been working with. But they basically completely rewrote their whole platform. Since this is recorded and I don't speak for them, I got to be semi careful. But I would love to get you guys in contact with them. But, you know, they've got a 5GSA core, an LTE core, all runs in a single platform. Everything's deployed with Helm charts. They rewrote everything, you know, and go. They've really put in the work like they're getting to the point now where they want to have horizontally scaling UPFs, which is obviously like the Holy Grail of cloud native and telco is how do I horizontally scale my data plane. There are a lot of really interesting things, though, when you talk about like what it means from a session manager standpoint of how do I migrate an IP, things like that. So, you know, there's still a lot of interesting challenges to overcome, but they've really put the work in to like rebuild their core like from the ground up to run in a cloud. It's just, it's tough trying to get legacy people to kind of get pick up what we're putting down right so I've designed a couple of environments and network topologies with this partner for some customers and, you know, just the fact that you start with a minimum of three nodes because you're going to allow the scheduler both the hypervisor layer and the case layer to do its thing like everybody wants to talk scaling like yeah but like how many more boxes do I add to get like two more p gateways and like it doesn't apply right like you just have total capacity, you need to like monitor and like the overall consumption on your cloud and what you're willing to allow for oversubscription failure rates and then build from there so it's been an interesting talking point and everybody's a lot more friendly with Jeffrey now internally because they're like hey how do we design this to like actually work. It's cool. I mean, they're really, in my opinion one of the leaders in this just at least from a, you know, total refactoring standpoint. It's not just a good containers with all the functionality shoved into one pod. And I've seen it mostly in VMware I've seen it in AWS. I've seen some really cool hybrid architectures. I was trying to write it down just as a skip at high level. The what do we mean. Yeah, I think saying this like across multiple pods versus in a single pod. And just the notion that you, you know, if you're deploying cloud native software there's an expectation that there's a cloud to consume, right, like, I get people asking like how would I do a single node deployment. And, I mean, the answer is it's technically feasible but it's a really terrible idea right like you at that point have to shift to something like microkates, or K3S or something because you don't have that CD, you know, federated. But it just it's it's funny because like now that there's actual software that's coming out that's fairly cloud native. A lot of people's brains are breaking just trying to figure out like, okay, but like how do I still do things the way I've always done them like I want to deploy, you know, two boxes at this edge location like probably not a great idea if you're deploying this in kates. And to be honest with the packet core, I don't know why you would deploy it that way anyways, even for like private LTE private 5g where you're doing like small localized deployments. The ran is a lot tougher though right because you're probably not going to build three to four node cloud stacks out of the cell site but it's a different different challenge I feel like the packet core is a great use case for diving into these and learning these challenges because it's got lots of control plane which is cloud native friendly. It does have some gateways in there that are not so cloud native friendly so kind of gives you like a healthy balance of different challenges to overcome and deployment. So you mentioned helm charts what about actually like getting how do you the helm chart doesn't actually send it you have to have somebody how are you getting deployments and sending stuff out. And that's going to be individual, you know, customer dependent. Like everybody's going to have their own delivery methodologies right so you get all your images, you get all your charts you get all your manifests. But because it's because it's cloud native quote unquote, it has almost no opinion on the cloud, other than, you know, baseline requirements. So how you build your cloud, whether you're consuming a third party cloud, what kind of pipeline you have etc. That's a case by case basis, you know, I could go in there and manually, you know, type like CLI commands to apply everything if I wanted to obviously not the best idea. But, you know, we've been helping some people, you know, some of our workflow at the Dell side so we have like bare metal orchestrator from the telco group, we actually do use hotel for this type of stuff this part of hotel I can talk about where we basically help customers build pipelines, and they can like link labs. But in this form you know I don't want to turn this into a Dell sales pitch but I've seen everything in anything like I've seen the absolute worst implementations you can possibly imagine and I've seen some of the coolest implementations imaginable. It just depends on where they are in their journey. And one of the challenges with like moving to cloud native software is if you're a very legacy network shop. You typically don't have any teams or processes in place to do things like continuous delivery. And then you go back to people just run in cube cuddle commands. Are you running into folks that are doing stuff like with, I'm just going to say get ops patterns and stuff like that. Absolutely. I've seen some that are like really advanced so like instead of using you know like some vendor provided like workflow engine. They're orchestrator of orchestrators, you know, they're running something like our go, and they just come up with like, you know, actual pipelines to handle all their charts handle all their manifests and they use all of like they basically use get is kind of like their state machine to control everything. See other people push it in through controllers because they really like the like, you know, API gateway style of things. I've certainly seen like seen a couple of providers in Canada actually that surprised me at how far along they were in get ops. Obviously you got looking crew over at Deutsche telecom. They're always doing cool stuff but um, you know, my old alma mater charter, we were doing a lot of stuff in that vein, using a lot of the stuff. So, I would say get ops is probably the most impactful thing that providers can do, like even more so than deploying a better version of a packet core or ran or this and that like actually changing how they do operations. Because I would argue you could use get ops for even all your legacy stuff. You know, your big iron type, you know, deployments if you manage all of your config to manifest and you run it through get and you get real, you know, processes in place gates checks balances. I don't I feel like there's like this notion that get ops only applies in a hyperscaler and Kubernetes when I feel like it's just a paradigm that is just good governance for managing things and you could do it across any of your environments if you're willing to put the effort in. I also find the smaller providers tend to be more get ops savvy just because they don't have as many levels of bureaucracy to overcome not as many committees to fight through not as much shared responsibility like one team will kind of own engineering through operations cradle to grave and so they can be a little bit more daring in what they attempt. Hello, everyone else. Hey, Victor, what are you seeing, as far as like production deployments. I mean, I know that you're heavily involved on the nephio project. I just put that with get ops operator patterns. You know, it's, but if you look at the underlying KPT is, I think, heavily about get ops in my mind, without that word being maybe directly used but it becomes the same thing. But what do you say in like production. Well, just realize that, for example, who has this DNA product, which is basically nephio production rate. So that's what, yeah, which is following exactly the same like the get ops practices and behind teams is obviously using KPT, which is, you can think you can connect with blocks or articles, but they are trying to promote like the use of conflict thing, which is the KPT reconciliation system. So, so yeah, I have seen that particular trend like especially like the justification is mostly because they want to try to scale out. At least get ops, you have, you have to have it played. You know, if anyone besides Google is doing anything with nephio production. It's even see this is this like a forked version similar to, you know, Kubernetes is was something before it was Kubernetes. And they have something. I heard like a few weeks ago, other ways for trying to contribute to nephio, hopefully they have like a similar nephio vitro. But yeah, today we can see anything. Jeffrey, are y'all using operators for management with the network functions. Do you know that y'all are doing the packet core stuff. So, I mean, it depends, because we come, you know, pretty wide open as far as stack we have some tools that we use operators in for sure. Like, for some of our internal products around like automation, etc, like the ability to like, you know, declare different types inside of Kate so that you can manage certain components the same way you would other things. But it's really mostly around like infrastructure orchestration and just being able to leverage the Kate's API at that layer. You know, as far as like, what we're doing from like a man or an SMO standpoint, this and that I mean, I think there's some stuff being released but we're mostly providing like cloud infra. And then we're pretty agnostic at the moment, you know, if you want to bring VMware red hat, rancher, whatever. And then we integrate different, you know, CNF vendors operators but as far as like our own DU and CU software that's coming out I couldn't speak to that like, you know, in a it's too early I don't think we're even talking everything we're doing other than that we have a ran but you know whether or not we're leveraging operators in there. I don't know. I can tell you that the packet core vendor I'm working with. They're just straight manifests and Helm charts, etc. Like, they're basically just using everything, you know that comes from Kate's they're not trying to like modify the API. I would say that I think that the operator craze is kind of calm down a little bit. I don't know Victor if you're seeing the same thing. I mean, there's still out there but I feel like, instead of like everybody saying I need an operator for everything. The hype is kind of calm down and they're being employed a little less frequently and typically in areas that make sense versus just trying to brute force things for the sake of forcing a paradigm that probably shouldn't be there. I mean, for example, at least from from probably like, I mean, right in an operator, we, it's quite easy. I mean, anyone can do it, but I'm writing a good operator with all the best practices and following and make it more long term like, it's like a chicken thing like that. That's, that's the major challenge for their other problem that I have here. And at least it is something that this in the FIO community. So hot topic is about the hem charts. I mean, for small CNF and things that doesn't require too much parametration customization. I think it's like ham charts is like a nice thing that you are when you have to consider like a multiple scenarios like when you have to expose a lot of different customizations and different parameters and all these things. And that's when ham charts, like a lacking of like a because you have quite bigger values, but Chamo, and when you have to expose everything and you have complex customization functions and yeah, so, so, I mean, with the operators, but mostly like, especially the way to package an unauthorized CNF seems like ham charts is like a lacking of some of the features or at least many, many people are talking about that particular method. I've seen a couple of ways that I mean, I think that we probably tried to make home charts do more than they were ever intended, which is where people ran in trouble right like you said for like small modest deployments they're great but you know what they're originally doing is deploy application X Y or Z. And when you have a multi tiered application it gets more complicated because you get into like, order of operations challenges. I've seen people just struggle and then build super convoluted operators because helm doesn't give them quite what they need. Other two approaches I've seen though is I've seen several people. It's weird that I haven't seen any like wide scale adoption of one of these types of tools, you know, outside of private implementations but you see these people who basically build like a helm aggregator if you will or something like something that manages multiple charts. And you can build like a chart of charts if you will is like what I've heard some people call it, but honestly the more savvy, you know groups that I've seen. They're just, you know, goes back to the get ops question is they either have some kind of comprehensive workflow engine, or you know they've got some type of, you know, advanced level of knowledge of something like Argo, etc. They're like they're building real pipelines, and they just build their deployment into the pipeline. And so that's how they're handling, you know, successive charts multiple charts charts that have dependencies on stuff you know so if I like to play chart a, you know, maybe I can't deploy chart B until a is finished like that's where like helm kind of falls down a little bit. But I've seen a lot of different approaches to, you know, dealing with it. I think my favorite approach is probably the get off style approach of just building full deployment models in my pipeline and deploying charts when they're needed versus the chart of charts just because you still sometimes get into these weird race conditions. But I don't know, I feel like operators are like one of those deals where you give someone a lot of rope to hang themselves like you were saying Victor like you can write one in an afternoon. But it's probably going to be garbage right. I've seen those some like the ones that I've seen the operators I really like are the ones that make small modifications to do what Kate's does best instead of trying to like redefine Kate's in a single, you know, operator where like, you know, I want to run a server, the same way that I run a pod, or a, you know, a vm in the case of Qvert. So like, they write an operator that lets you deploy it to declare type server, right. And, you know, assuming they write it well, etc. So now you have bare metal infrastructure that is, you know, manage the same way as your other primitives inside of Kate's. But when I start seeing people basically try to like write an NFL, or a vnfm and an operator. That's when I think things start going super sideways on people. Would you say operators with a single concern or less with single concern, less concerns. Yeah, I mean, I don't know at what point, you know, like one, like one concern per operator like I don't know if it's necessarily the same I do 100% agree with them kiss. I just I feel like the best operators I've seen are the things that like take what Kubernetes already does best, and then try to like, you know, apply that to a small subset of things that you want to like include in a deployment right. I mean at the end of the day, you know, Kate's is kind of like the fanciest rest API on the planet so if I'm doing those types of things like right operations, where I want to push a manifest etc. Other things I'd like to include in a manifest and be able to define and you know track state on and ensure that like my declaration is being upheld. I think I've seen some really cool operators that you know, follow the kiss principle, and they do like great things right like a small little things that you know vastly enhance the capability of what Kate's is doing from a deployment standpoint. And then I've seen other people you know, follow the yellow brick road into Oz and just get themselves in a lot of trouble. Like they basically trying to write an entire orchestrator in an operator, just so they, you know, their orchestrators API is you know the case API instead of a custom API like I don't know. When you start trying to force Kate's to do things that it wasn't never intended to do that tends to break things in my opinion but there's also way better software developers than me out there so they might do cool things who knows. And regarding the similar trend or at least what I have her is, well, in order to keep like a, like a simple, it's important to have like a kind of a note. Okay, because I mean we have like an operator that you gave us, like, open air or CNF SDK or things like that, where you can reuse most of the things and a facilitated development or like this provide like a library where you can reuse. Some functions or certain functions that are very common on CNF that you can reuse, especially for because if you have like a single operator for CNF, eventually you are polite for the frame like a lot of operators and. And that could be also an I'm air, especially for operators, but for people who have to implement those things. So, yeah, I think that has to be about between the number of things that you, you want to deploy like especially for your CNF, I mean you have quite a whole pack of things, or you can reuse things and which using that kind of SDK or library or things like that. In order to achieve like what I free that like a simple as possible. Seems like there's an opportunity to maybe examine and talk about operator usage at nephew is, I mean, say it's all about operators and they're trying to figure out. Best practices around that and I do think there's probably some people that are thinking they could do this. Do everything they would do and that's a mano and. Folks talked about how do you do layers to do Yang and other stuff. So there's probably going to be, I would just say there's going to be some that are maybe too much. But not everybody's doing that, and it's in the nephew some and it seemed like some of would be around this. A simple purpose or single focus or breaking things up, you may have multiple operators but they're all specific and why are using it. Best practices around that Victor we've been talking about it for a while but just that nephew and deployment but maybe the focus should be operators. Which isn't just nephew operators but it would be, you know, what are what's happening there what are the best practices that could be used there. I don't know if. Actually, I do think that Dosh chef from that should tell calm. I think they do have operators now that I'm thinking about it in the Dosh chef, which uses flux and a bunch of other things. It would be interesting to see who's doing it with, you know, more cloud native and flexible, automotive automation. Management deployments. What are they doing are they using operators. Try to figure out some best practices around that. There's those Jeffrey. I mean it seems like it's something that I don't think it's going away. I think it's maybe not talked about as much. So, sorry, I missed the first but what's going away or not going away. I think that the buzz around operators has slowed down but the adoption of is, I think, becoming more standard people are going to use operators. So maybe helping to try to talk about best practices and don't overuse them or don't add too much to them. Yeah, I'd say a lot of people, you know, who were just consumers probably using operators without even realizing it, which is probably a controversial statement because you should know it's in your infrastructure. But yeah, when I say the buzz died down, I mean, like, I feel like we've kind of gotten over the hype pump where everybody was just trying to release an operator as fast as they could to show, you know, that they had one and this and that, you know, really that bullet you have highlighted. All the neps are like, you know, we're going to do X, Y and Z with an operator. Now it just seems like it's, we're getting more to the keep it simple stupid, you know, portion where people are realizing they need X functionality and they package that cleanly and neatly to achieve a goal. And then they move on. I think you know your point about multiple operators. I think my real question is is, you know, as we get more and more mature solid operators that are actually, you know, worthwhile. That are doing good things for us. You know what happens when you try to like manage all of them if you start to get like conflicts and what they're trying to achieve in the API. I haven't really personally like I thought that was going to be a much bigger concern. When, you know, like two years ago, especially when everybody was talking about these big super complex operators. But now that most people are coming in with like limited concerns and just, you know, this operator achieves this goal. I'm less concerned, but the more and more you add obviously the more and more risk you have for some kind of conflict if you know, two different operators are competing for the same resources or something. Yeah, the other thing that I was thinking, I mean, we talked about this a little bit in the last week. I think the major change for it happening right now I mean, probably like, like the larger for some years ago, most of the efforts for operator were like, try to expose as much as possible. I mean, adding all the capabilities and try to offer all these capabilities to developers and try to sell those ability to them. But I guess now I have seen like a kind of interesting trend where who is dictating the capabilities of the infrastructure is mostly the application. So let's say that you have like a CNF, which requires certain given a conversion plus certain things. So, so that that's going to say to the infrastructure, I need these things. I don't really care if you offer a free or like multiple or things that I don't need it like that. I think that that's I guess that the major switch that is happening. Because now, especially what when you have people consuming public cloud providers. They don't really care too much about the infrastructure like so, so, so developing and packaging your application setting. Well, my application only needs these certain pictures and that's fine. I think it's a little bit particular change that I have noticed like I don't know, like, it's interesting trend. But, but for me it's like a new way to deploy things, not to make too much emphasis too much emphasis in infrastructure like that you still have to do to support these things but mostly who is driving the conversation who is dictating who has to offer what is the application and not is what the infrastructure is capable for. I don't know if I can say correctly, but I feel like that's that's a little bit changed but where has been noted especially in the way to deploy things. Did that I was trying to connect how does that relate to operators that just challenge in general for the requirements. In that case, like, yeah, basically, two prayers. I guess it's the same thing like if you are a TN offender. So, and obviously you have to pay like your own prayer or you have to, but I guess the major thing is like who has to the TN offender has to obviously to what are the things that they needed and probably they're going multiple places where or like, I have seen it for example in on app or like by your or others, trying to collect all these requirements like a hardware requirements or like offer requirements. So, and try to enforce the skin offenders to specify which are the needs like which are the things that you need for your CNF. So, or that could be like your services. And so, whatever you have it, plus operators. In other words, for example, things like an okay where where they were trying to develop like a platform like a reference for supporting all the people combinations. I think that's good but I think some special limitations in that approach. You don't want some single management thing that controls everything and well, if it's not like built in I guess. I'm trying to think of this so they, if you you want to be able to share your requirements and versions and stuff. And you don't want to have everyone require their own Kubernetes version. Single nodes and that's kind of related to what you're talking about before Jeffrey. No, no, well, probably. Yeah, I'm just trying to the problem in general, Victor, it's like a trying to solve it there's problems both directions. It's a hard problem to solve. While you definitely want, while you want the application developers who know the application best, they know what they need. You want them to be able to drive that. At the same time, you need to have some, you know, compatibility and stuff between them. I think that the major changes, for example, in the past, they have to be aware of what the infrastructure was capable of, like, I mean, okay, maybe I can improve the infrastructure in this particular way or using SRIOV or things like that, but I need to be in constant communication with the operators. See if or could even preserve to support those things. And I will say those things. So I think that was in the past where they have to have that conversation as they will probably, I will need this particular library or this particular general version because probably the app that I developing is going to require all these things. I think now that the limitation has been broken. So because they can, they can access to the private or even with workloads easily like I have seen that particular trend, which in the development, it's like, I guess, removing those barriers. So now the operator can, I mean, the developer, the CNF developer can say, well, I need this particular capabilities and I don't care if for current persons supported or because I have that freedom to use them. So eventually, I delegate that and I'm sure that the operators have to mean that the infrastructure operators are going to solve a particular issue, but I don't really care in my development to have those limitations. Hopefully that's actually with more the difference. Yeah, that makes sense. Jeffery got anything. I'd have to noodle on it a little more I think. Like when we start seeing need for compatibility matrix. Victor's earlier comment about, you know, kind of having like a framework or something, you know, like common so that people, I don't know. It just it starts to sound like Yang and toast go again where, you know, we have like these defined specifications and I really don't know. Like I said, I have to noodle on it. I don't know if the answer is just everybody brings their own operator. And the thing is right is, you've got like nephew, you know, it's network plumbing. It's not a CNF right. You've got, you know, companies like Dell and others, you know, Cisco, etc, who build operators to manage infrastructure. I don't think that like it's easier to come up with like some standard for like, you know, operators look like this so they're all interoperable. And conversely, though, I don't have a good, you know, answer right now for, you know, how do we deal with the version and compatibility issues, you know, like operators, like you said, I think they've come a long way and they're super cool. I don't have a great answer at the moment on how I think I would tackle dealing with them at scale. But, but at least all of our, although we are a great get out to probably the only, the only solution or the key part that we have to go. Yeah, I agree. Like, you know, the more and more I'd like dive into the get ups, you know, abyss deeper I go like, I mean, really all it is, you know, there's the second half of the catchy phrase ops is really what it is is just doing ops better. Right. So if I have a lot of Helm charts, you know, how do I operationalize those Helm charts to do something good. If I have different operators, you know, how am I testing to make sure that I, you know, don't have the conflict. That's mentioned in the first bullet. How do I deploy them like, I don't know. The only problem with saying like get ups is the answer is is get ups requires lots and lots of work and care and feed so it's like, like telling someone that you know, oh yeah writing applications is easy here's the job of programming language good luck. We always come back to this like circular thing of, and I haven't found a way to break out of the circle yet of, what's the right amount of like, you know conformance or consistency so that we can. operationalize things at scale, which is a requirement for really running big networks versus not breaking, you know, the openness that makes software to find anything a good thing in the first place. No operator wants to run 15 different sdns in their network, but nobody also wants a single opera, you know sdn that doesn't really do anything it's really supposed to I don't know, just kind of babbling now, like you said, I need to noodle on this I don't really have a great thing. Other than that I think the challenges you've listed are pretty accurate. All right. Let's see. So, looking back to see enough working group nominations are open right now for self nominating. And for those that haven't heard that it was at the nephew summit mentioned, and it's more of it'll be at the telco day. There is a merger of efforts happening between CNCF and elephant. And especially around certification testing. So, what's going to happen to this working group and the efforts here. What do you want to happen. We need to communicate that. That'll be part of the decisions between CNCF and elephant. Think about it. If you want to post comments directly posted here, you can reach out to me, add stuff in. All right. So, no, CNF working group during coupon this meeting at least. I don't think we should have it. Victor. Yes, no for canceling that one. It is the next week or like after I think that's fine. We can cancel. Yeah, we'll be there. All right. I'm reached out to me previously. So, yeah, Jeffrey. I'll see you know, y'all are on with everybody. Yeah, I mean, my schedule is so hectic that basing any cancellation or keeping based on my vote probably doesn't hold a lot of weight, but. All right. The cash. But if this keep going, then yeah, I would cancel. All right, cancel. Okay, what about this one during the elephant dev and testing form, Victor. November 13. Do you know if it's going to be hybrid or. I don't know. I only just saw. Like, registration in person, but I'm not sure if they're going to offer. They're to lax it. So, I mean, I'm not planning to go to the best. But I know that there will be some topics about the file. Right. As long as they provide like a. There's all. Well, there's some in person. I don't know if it's there's any virtual. There's probably. There's still open for proposing topics. Wow. They're November 1st. You got to do them on site, no remote presentations. All right. All right, anything else. All right. Thank you everyone. Have a good day and good week. Thank you. Cheers, everyone. Cheers. Bye.