 Looking a little light today. We'll give it to about five after and then we'll kick things off All right, it's five after we ready to get going sounds good If you haven't had a chance, please add yourself to the attendees and if there's anything you want to add to the agenda Go ahead and throw it in there Typical, you know Upcoming events plug Obviously KubeCon is coming up next month There's going to be the CNF working group intro slash deep dive session. I don't know if anybody else has any other Sessions that are going to be relevant to the CNF space that they think we should list here or if anybody in our group is presenting but please feel free to add with a link to your case and then I Was kind of hoping that instead of slogging through pull requests today We would continue the discussion from last week around use cases and best practices Maybe specifically focusing on delivery and platform Try to capture some notes and then maybe we could open up some issues or discussions After this and then start getting these populated in the repository Well, yeah, I'm agree with that like but there is just one PR. It's just it's small fix of typos and Turnery, I don't know you went to quick look of that one. It's Make up the number I Didn't initial glance at this. I don't think there's probably anything first shattering Probably have to go through a few more times. I mean, I looked in everything Small grammatical, so I'm good to approve it when I reviewed it this morning. They'll probably need to be a few more passes At over it to just in general, but if other people want to Approve it and we hit the magic five before the end of this call will merge it. I think for spelling. It's just one Yeah, we can merge this one Nothing but spelling there's no Yeah, I guess For some reason I thought I read like a grammar fix or something or like the word I think is the word list makes it feel like it was dictionary or something, but it's it's all just related to grammar and Spelling and everything else linting Victor, thanks for diving in and cleaning that up for us All right time Do we want to take a peek real quick before diving in new ones look at No root One thing that could happen on this one would be relating it to the existing use cases. It doesn't have to be a new use case So and that's that one section is really the only place that was left in this Use cases and user stories and if we Have something Relating it to the existing ones that would work as well We have the a longer least privilege Use case that we've been right or writing at that one's not complete So there's really two paths to get this one done finish the least privilege add in that new user story Use case or Relate this to existing ones even something as small as the discussion Forum item that Frederick added for bumping the wire firewalls That could be used something very minimal. It's just we want to provide this section and the best practices just for providing context I've gone through this one a few times So basically just need to read you now that most of the comments have been resolved and there's still a couple of Ianisms from like a Writing style standpoint. I think that we might want to tweak a tiny bit, but okay. Well, I mean We can dive into this one Taylor if you want to just do it or we could kind of do some brainstorming to generate some issues slash discussions and get and then try to get people to kind of work asynchronously on Generating some of these best practices and use cases and stuff We don't have to do a I guess a live working session Unless that's what people want to do. Okay. Well, then I Kind of just want to read through it again without People staring at my screen while I share so if I'm Mean and I think we could maybe have some use cases that pop up from this But if we talked about The onboarding and lifecycle management use cases I'd be curious around best practices people think of delivery I would specifically like to talk about air gap installations and Specifically like what is air gap in general mean beyond just delivery, but also like, you know, a lot of Third parties want to provide like their monitoring platform via a SAS model Doesn't necessarily work. Obviously if there's no direct access to the internet Or you know Thinking I said the box like if you're a big big, you know customer of a public cloud and you have things like a direct connector You know a VPN to like a private VPC or something like that Could you do the SAS offering in a private VPC versus a vendor VPC? What would that look like like I don't know if people have thoughts, but I can tell you that I have to go through and constantly figure out how to air gap a lot of things and so I Don't want to just put CICD as a best practice. I'd rather like kind of talk through the nuance with the group here around Like some of the things I know that I do and that others I talk to do to get artifacts checked in Scanned made available through eternal repositories You know figuring out the right balance of being quick enough but safe I'll pause there if anybody just has any high-level thoughts before I just start putting stuff into the Google doc So you want to brainstorm on potential best practices to explore on these topics? Yeah, I've been thinking that a lot of people are kind of hesitant to dive into get in like 99% of all activity happens on these Monday calls So I'd kind of like to do some brainstorming and then like a few of us go in open up some discussions or some issues in the note section Maybe you know mark things as like good first, you know attempts or whatever especially Once we get past cube con if we're lucky and we you know get some new interested parties in they could potentially go in and Find something that they could pick up. That's you know seems obtainable to them and put in a pull request. I Don't know if that makes sense or if you have like different thoughts Yeah makes sense to me The I have questions about the air gap, but just trying to see what we're going to focus on If we're going to do brainstorming it's good to somewhat have some rules around it that idea of putting Being able to put forward ideas and not worry about critiquing at the start and then we Can come back through after we have a essentially a dump. So whatever people want to look out. So someone may say CICD that's fine. And then you say well what within CICD is a good practice And you're if there's a specific challenge on the delivery or whatever then that can help guide but the idea primarily is to let Someone put something forward without critiquing during that then we come back there and you can add to any ideas as well So if someone has an idea anything, oh, what about this This with that one and just keep adding And it makes sense to me I mean We get like this small one hour a week with a lot of smart people on it and I'd like to get away from us, you know looking at voting rules and Glossary definitions and instead kind of just start talking about like Extrapolating like what people are doing what developers are, you know Developing what challenges they're having and getting their software out to customers like what providers are doing what the CNCF Thanks of all of us crazies trying to like Break all the plumbing inside of Kubernetes And then like I said, I think then if we go in and also pre-populate some stuff and get It might be easier for people who You know are introverted or you know, don't have the confidence in the subject matter to just like completely wing it on their own that like if there's things that they could go in and Grab and try to tackle then we might get a little bit more asynchronous work and then have more things to discuss on these Monday calls And the other important thing to cover is also describing the problem is saying right like I mean, how can we Propose something or at least it's something that I like it from the last published White paper like having very clear Which is a problem and it's on that Propulsion ideas and yes, it's the best way to tackle those things So for example in this case we can describe What what could be the problems Or that we are phrasing, right? hmm One question I had and I think the least privilege one demonstrates it is what about the things we can't do today because as an example with least privilege, right least privilege would imply that you can You don't need privilege containers with any level of privilege For doing day-to-day jobs that we're going to need to do for NFE, but we've already discussed that there are places where You're going to be asked for cap sis net and cap net admin, whichever one it is And we don't really have an option. So what do we do about places where you know, we would be saying well Kubernetes needs to be better for this to be ideally working Yeah, that was my take as well. I think Okay, that's an open question Oh, we can dive in. I'm just capturing some notes and you know, I'll be honest. I'm you know, bill and taylor This is probably someplace too where we would lean on you guys to be ambassadors, right? I mean, I know that sometimes we have unsensible requests that the upstream community is just going to balk at because they like to keep things Simple they like the way that kates works now. So as coming in and saying we want to fundamentally change things Probably not going to go that far. But if we have like, you know, small sensible requests How do we uh From this this working group like push that like versus trying to infiltrate a bunch of other tags and you know Yeah I think the most important thing here is to have a good reason for asking in the first place, right to say there is truly something we can't do Because coming to someone saying we want something to change and then turning around saying but you know I don't understand why that would be necessary. Why you can't just So so we need to have a rationale Particularly since you know, we're working in a specialist subject and it's It has its weird requirements I agree with what you're saying and I said having been able to write up. What is the challenge? What is the challenge? And what we've looked at And what's available within the ecosystem and that we're not finding any anything there That meets the challenge And then if we have ideas then we can point to some of those but I would focus more on a high level idea for potential solving the problem versus an implementation specific Yeah, I certainly wouldn't want to jump in saying we should write precisely this code and absolutely not You should write precisely this code because you know, that isn't about to happen It's open source because we write what we need. We don't just expect other people to do our work for us so it's You know again as an example multiple namespaces There's a multi solution that allows you to get a certain way with that but You know since you are going to want to reprogram a namespace. There's any so much so far that gets you before these privileges broken And it becomes, you know, almost impossible after that anyway, yeah, so Fine my my proposal as much as there is one is really just that We shouldn't rule out possibilities because kubernetes doesn't do it today we should just highlight where There is a problem with the way that You know with what we're asking for and what kubernetes provides that looks like an extension is required Um, whether that extension is changing code exists or writing an entirely new plugin Which obviously kubernetes is quite happy to let you do then That's a solution specific detail Jeffrey, um, I have A question around I guess the air gap stuff. I I don't think you're writing other things in there. That's fine um The air gap I'm having a hard time thinking about the relationship of air gap with cloud native um, it's to me it keeps feeling like air gap is another Feature that may be needed by some users But it doesn't seem Like an air air gap has a direct relationship with being cloud native So then I'm wondering is there a way to do air gap Implement air gap solutions in a cloud native way. I guess would be the way to look at it But I'm wondering what you're thinking about when you think about air gap Do you just think of it as a required feature? Or do you actually think of it as something that Is a cloud native thing? A challenge or what? um I don't know. It's I think you ask like three different providers. You'll get different answers. Um, so for me, it's an ironclad requirement from my folks in production Right, like We do not in certain environments A lot of direct internet access In some cases not even through a proxy or anything else, right? for safety reasons like this gets into um All right, let me think of how I want to like phrase this because I've got a lot jumbled in my mind So hey, I don't think that like whether or not you air gap something or don't air gap It makes it cloud native or not cloud native. So Um, I think that this whole concept of making it quote-unquote cloud native via air gap or non air gap Doesn't quite click in my brain. Um Additionally, you could argue that if you do a bunch of preceding steps, right? You've minimized the risk and maybe you don't need to be air gapped. Um, assuming that you know exactly what's going into your Infrastructure that it's versioned correctly that it's been scanned before it ever sees anything. Um but the fact of the matter is is um There's in lots and lots of stuff hidden underneath the covers where you'll have like You know repos, you know that are listed in charts. You'll have hidden curl commands and automation You'll have you know Just things that go out and until you've actually tried to go and install something in a world where there is absolutely no internet And see what breaks. Um There's probably things that you haven't caught and this like opens up attack vectors. Um I've seen it a bunch both in my time at a service provider and on the vendor side helping lots of other service providers, but um Really what it boils down to is, you know, one of our first use cases was cnf onboarding and then cnf lifecycle management We keep talking about the contract between cnf developer and cnf operator. Um One of the biggest challenges you'll have is getting your software Into their network and into their data centers and into their private vpcs and other public cloud environments, so You know, we could argue that it's not a best practice to do air gap to installs. Um Then we would have to figure out like What we say to service providers who are like, well, then I guess I can't do cloud native But like I said, I don't think that being air gaps makes you more or less cloud native It just is a means of how you are managing your dependencies and packages and providing software to environments that maybe don't have direct internet access It seems like it may limit your choices and implementation um, if if you said you had like say a distributed pipelines uh, which you went over and there's a Use case that you've talked about but if you had something like that coming in and thinking of um But so okay real quick though I'm gonna disagree with you for a second because I would argue that the distributed pipeline Is a best practice that allows me to do air gap without breaking everything Like if I have the corresponding web hooks pulling from upstream pipelines But the initiation of my pipeline is to pull down all dependencies into a private repository Get them scanned and then make them ready for the next part of the distribution in the pipeline I think I could still pull this off. Um And I'm speaking somewhat from firsthand experience. I can say that it's insanely insanely challenging to get the initial framework in place But there are a way for you to get everything you need into your own repositories To watch things dynamically upstream and then ensure that stuff is saved and versioned based on the versioning that you're doing within your infrastructure And if you look at books use case right around the lifecycle management piece um You know In my opinion a bad practice to have colon latest when you build things right like just always pull the latest You need to be able to like roll back roll forward You need to be able to do root cause analysis based on my current version that you're on And I I do think that there's best practices that make it so that you can do air gaps Well, not necessarily like preventing yourself from being able to use a bunch of stuff upstream all right Perhaps the question there is Where's the break right? Does it need to be between your ci and delivery or does it need to be between Your ci and the network, you know, you pull the container down you feed it into your offline your disconnected ci um Without judgment. I'm not saying there's a right answer to that. There's probably a it's probably an opinion thing Well in different service providers will have different levels of restriction, right? Um like some Maybe they don't wow just like, you know publicly, you know, routable ip's but they have proxies. I mean And I'll like to this exact point though one of the things, you know, is this bullet right here is What would a gate look like and I mean? Probably not getting down into granular but like let's say I have You know open source repos vendor repos, etc Like what does the first gate look like to pull in their artifacts? Like what initial ci am I doing? You know, how does that roll out? Um What is the this is once again getting to books use case of life cycle management If I've got different dependencies and packages changing asynchronously How am I like orchestrating all that chaos and ensuring that you know The contracts that I've made with various entities, whether it's at the platform layer the application layer or the network layer are being maintained Like I'm not proposing any right or wrong answers here. I'm just saying, you know We keep saying like what are the challenges? I can just tell you this is a challenge that I face every day of my life is figuring out this Yeah, well me too. I mean, I'm trying to make sure that build comes out the same result every time When there's always the temptation of any developer to go and pull a random package from a random source on the internet Supply change attacks like that are very very easy to end up with if you're not careful Which isn't quite the same as offline install, but it's there for very similar reasons Um, but that's a thing where I wonder whether we could find the best practice document of someone else's and refer to it um, you know not say subject to our own supply chain attack someone to use changes the source document, but you you know, so um uh that A ci's conclusion should not change just because it's a wednesday effectively if I test the same piece of software Something that claims to be the same piece of software. I get same results Um, Jeffrey, it seems like we could go down two different paths with this one is how to best practices on and and ways of Deploying into an air gapped environment supporting air gapped and the other would be Looking at what are the challenges on why air gapped was even needed in the first place and On that side, there's probably best practices And alternatives to solve the problems the air gapped is there to solve with regards to security and that could open a whole another area for us to Write up the use cases and best practices Doesn't mean everyone then doesn't use it immediately or anything, but it's I'm mainly just saying a different area to explore No, it makes sense to me And it sounds like you have a lot of um Inside on why it's there. Is that something that you'd be willing to work on? Uh, sure. I mean, so the first one is um You know security, right? Um, specifically minimizing a major attack vector and Once this isn't going to be cnf specific, but we can talk, you know, at least at like high levels is Obviously, there's the notion of if i'm exposed to the internet then that means that the internet can attack me So now in addition to the inside insider threat, you have, you know, legitimate like vulnerabilities to the outside world um But once again inverting that argument is If you are air gapped it prevents um A certain level of insider threat. I mean, you're still going to have to worry about east west, you know type of tax horizontal issues but like Let's say Let's look at the target breach that was in kubernetes, right? They had a bunch of kubernetes clusters. I don't This is public, right? I'm not going to get in trouble for talking on this recording about a case study. I read. Am I because I mentioned a Let's scratch out the uh The company's name. Um, anyways There was an issue where they pulled down a helm chart. Um, the helm chart had certain things in it that, you know Spun up certain services and expose certain ports. Um, because these were these clusters did have, you know, internet facing interfaces. Um They, you know, eventually Got caught up in a scan A malicious entity like knew that there was this now backdoor to go in and they went in and, you know, wreaked havoc. Um so then once again like If you don't need to provide a public service You know, should you be exposing it to the internet at all and I would argue that The majority of the time you expose it to the internet has nothing to do with like speed velocity agility. It has everything to do with convenience. Um It takes a lot of work to get, you know, your distributed pipelines in place to manage your artifacts correctly. Um, and it's just kind of a pain, right? Like If you're just building stuff in like the development phase You don't necessarily want to uh Have to first pull in artifacts to a private repository, you know, get them checked scanned. Um But there's a big difference between, you know, building on your laptop and building in your actual lab dev environment, right? What amount of risk are you willing to expose? So I mean That's one thing The security and Specifically exposure to the internet. So it sounds like if for that one I'm being able to open ports publicly and other things The first level would be a proxy server And you can't open anything directly. You have to use a proxy. You can do pull only Maybe no push or anything else. No no capability to open ports Okay, so before we start like And I know we're not supposed to critique but I just want to put out the baseline stuff because Okay, go ahead A little acknowledgement that there are remediations and stuff But once again anytime and this is why I think security was kind of a tricky one to start with Everything in the security world, right is around like risk analysis risk mitigation like um You know, it just depends like what level of risk is your c iso willing to take if they are super super risk adverse Um, they might not be willing to do any of this and then like I said, um This once again now gets into the least privilege conversation pretty interesting how all this stuff ties together Right is like if I come in with a privilege container that says hey, mr. Proxy. I'm allowed to do this Um, you know, what happens if I pull in a malicious image that then As long as it can phone home to the internet, uh bad things happen to you. Um, so but I mean I do think though everything you just said taylor is is great. Um, you know, like there are definitely ways where you don't have to go full air gap um I'm just saying though if you do right like And keep in mind too when it comes to service providers banks and medical Um, sometimes we have regulation so then it depends on what country we're in and like what restrictions they've put on You from a networking perspective like there could be some type of legal restriction that prevents you um Right, so like if the NSA for instance came in and started hanging out with us or the department of air force Which is doing all kinds of cool things with containers Um, they will not ever in a million years allow you to pull straight from the internet into any other private data centers, right? they're Yeah, I think we have to remember that security like test is an endless tax, right Nothing can be proven secure lots of things can be proven insecure Nothing can be proven working, but you can prove something doesn't work So the the question is always what mitigations if you've got what are they mitigating? and What's the likely head that somebody's going to do them? You're arguing that Disconnecting your actual active running network of software Disconnecting its management side from the internet so that it's not randomly communicating as For instance an iot device in your house might do and you don't know what it's sharing You don't know that it's not just pulling down random software to run Is a very strong mitigation for any attack on the management side of the network. It's not perfect proof, right? There's always malicious actors possible in your company, but on the other hand it is a strong mitigation for a lot of attack factors Now that's a hundred percent correct. And once again, I'm I'm just presenting a challenge I have to work in an air-gapped world. Um, I'm not saying that that is a best practice or that everybody should air gap You know, it's just a reality that some of us face. So like You know, there's original question of like, well, what does that mean for a distributed pipeline? So I'm just kind of like can we keep adding I'd like to keep adding any of these Reasons behind it. Yes. Another thing is um consistency and um Resilient consistency and resiliency, right? So like one thing too about using um private repos and not allowing things to go upstream to repositories that you don't control is You get to ensure Exactly what packages are available In your environments, right? So like if I and this gets really important when we start talking about like networking equipment, right is um you know, typically They're very very big complex pieces of software, right? that have tons and tons of features like you look at like A router like all the things that it can do if it's like, you know, a high performing high-end router from one of the major vendors And typically there is like a certification process, right? Like that um something like the way that you know, bgp Entries are added or withdrawn from Your routing table things like that like sometimes they tweak it because the rfcs don't just explicitly say this is how you will write Your code it says this is the behavior that you should like present and then it's kind of on the developers to um achieve that so like Once again, like if you have a version that you have like run through all of your tests Um and eventually long term. Yes distributed pipeline should hopefully mitigate this but like You know, I've got four major vendors that I'm you know, doing bgp peering across or you know I'm as a major service provider in a pop and I'm peering with one of my other major service providers like And we have an agreement that I'm on this version of you know, I don't know since he ends the call I'll say asr code right How do I ensure that it's always on the package that I want? Well, I control what packages are available and once again not saying this is right or wrong It's just a means of mitigating risk. It's a means of ensuring Consistency and where it becomes important right is if you have thousands of routers or thousands of cnfs, right? It's pretty easy to make sure everything's on the same page when you're operating in the tens and even the low hundreds Once you're starting to scale big big big things to the thousands and like that kind of stuff then um, you know ensuring consistency through control mechanisms Is important and then like I said, you know, here's another bullet here is um policy as a means of building trust You could potentially tackle this a different way than air gaps, right? You could do it So where you write policy that enforces things So i'm not once again trying to prescribe any type of implementation I'm just talking about some of the things I get um and one of the things is um You know if I control the packages then I kind of you know have a little bit more control of my destiny And I have a very like high level of confidence what has or hasn't been deployed in my environments It's it's um There are a couple of things there I don't think there's any firstly policy is fine and good, but it's easy to have policy mistakes as well It doesn't solve any particular problem completely cutting things something off from potentially contacting unsupervised sources of whatever variety is 100 percent certain Whereas policy is not because again, there could be mistakes, but um, there are Yeah, I mean it's the reason So I was having a conversation last week with somebody about Trusting software and one of the things um, funny you should mention asr's because it does this right asr's won't run software That hasn't been signed by Cisco Um, they can recognize it. We have a I think it's a shared key arrangement, but honestly, I don't know The the the point is that in order to ensure that this thing is running exactly what it's supposed to be running Which is better for all concerned, you know, both us and the operator We make sure it will only run things that we intend it to run And that's why I stuck the next line in there, which is effectively saying how do we have end-to-end chain of trust end-to-end A guarantee that what you are trying to run is exactly what your cnf author intended you to run um If isn't the same solution, right? Because that software could then try and reach out the internet and pull extra pieces of code So you can't prove it's not proof alone that you're getting what you're looking for But it's another form of proof that says I am running the thing that I intended to run and also the thing that I ran through ci and so on and so forth I'm not just running something that bob in You know in whatever department decided I would run today because he hacked it around a bit and he thinks it's cool He has stronger mitigation features for some purposes because it actually stops internal actors from Changing your software as well as you know pulling the wrong thing off of the internet So that's another one that Serves a similar kind of purpose I mean what we're ultimately getting to is you run what you intended to run, right? That's the use case in it in its Uh in the shortest form. It's true for It's partly true for airgapping that you can't go you can't have software randomly fetching further instructions from the internet It's certainly true for end-to-end chain of trust um, the other thing about Cutting something off the internet is independently it cuts off an attack vector and people like that as well for obvious reasons But um, those are the two purposes. I think it's trying to get to Perhaps what we should have done is realized that Cutting something off from the internet airgapping is not a problem, right? It's not a problem. We're trying to solve It's actually a solution that we're trying to find a reason for Well, it's I'm going to shut up after this because I kind of want some other opinions See if anybody else has the oath, but you're right I mean, this is a solution and in some cases depending on like what the The compliance or regulations are it might be a forced solution. So then You know, what are the challenges? for Like how do you like mitigate this like and I mean, like I said, there are best practices where like hey maybe like in certain like in your dev environment where it's like You know the east west type of tax are minimized and you're willing to like risk it and you know The splash domain would be minimal. You put the proxy in there. I don't know. There's a million things, right? But um, I'm just kind of curious to you know Alexis Victor others if you any of you deal with like airgapped environments or worlds like Do you have to deal with it when you're handing off software running software, etc? Yes, definitely as soon as we hit production a lot of the Publication we're deploying our gap meaning they didn't have any internet connectivity Um The way I've seen this being solved is really to you know, you mentioned the CI city But again, I'm going to talk about a solution necessarily explaining the problem We had the product that the reason for the No connectivity in product for these applications was solely because you know security purposes So mainly the the main bullet point here the first one um And and and the way to to work on it was basically to have a registry in a production environment and a registry Uh, you know in a pre-prod environment and and the only You know bridge. I would say between the two was The fact that we could push to that prod registry from that, uh, you know pre-prod registry And so just push the artifact that we would have vetted As part of all the testing and all the Did they build done prior and and as for the network so Of course some of the practices In this application is don't don't have any curl in them I actually need to pull things well We would overwrite all the information to point to to the registry that we own that registry wouldn't be a proxied registry It wouldn't proxy anything it would be fed with the artifact that we would care about and um And and that's what I've seen from for an application standpoint Now what while we're talking was thinking about uh Network function applications My my my I don't I can't recall How much internet connectivity some of them really required Um, I didn't work with a lot of them, but some of the project that I worked with um I didn't face that our gap issue for these network functions Well, I can bring up one thing right when we talk about it in the cnf context specifically is um, you know licensing um A lot of uh vendors have moved to like smart licensing servers And I've seen different ways of handling in the air gap world, but like, you know the default behavior that a lot would like is Uh, you phone home and you know hand a token a key or whatever say i'm licensed I've got this functionality yada yada and um Suddenly we say but we're going to stick this in an air-gapped environment until like Ian's point What we're really saying is that the management plane is going to be completely cut off from the internet. So then Um, how does something like smart licensing work, you know Once more there are ways to accomplish it, but i'm not saying I know the right away wrong way No, I agree and that phone home is a great example. I've lived through it the solution that was provided to us at the time was um A self-hosted basically, uh phone home server So the network element would phone home to something that is internal and that internal piece would go Out of the internet to to register to the vendor provided license Uh solution So again, it's a tiered approach. It's It's it's it's one that I've seen But i'm not being very practical here It's I mean, it's just a tactical answers to some of the points you had I My point was I I don't have much to comment on the air gap specifically approach I understand the concerns That that it's trying to address and we do see actually a lot of Uh customers that is looking into our gap for for the run for instance deploying run applications Um closer to the edge But but I don't I'm not armed with enough material right now to provide any insightful You know comments in the discussion Honestly, I think you are, you know because you've effectively come up with an example that demonstrates that this happens Um, which is you know the point it's important that we actually have concrete examples rather than just working hypotheses Sure. Well the example I can tell you the use case Well, I won't tell too many details, but it's just the the vlns that we have in the data center that enable Good connectivity from uh from the customer, you know set up box or or router home to to to to the private csp network That the use case that I've lived that even this requires to be You know provided a license exactly what you said jeff and so We had a capability in the software that we've chosen for it That has a self-hosted um Go home server And uh, and that one would proxy basically all the licensing Um, but that's specific to the license Again for a network function Unlived experience actually just one question there. Was that a Thing where at the late in the late in the day you went to your vendor and they sort of turned a bit pale When you explained that their licensing system was a problem and they needed to rethink it Or was it well thought out from the beginning We we came with that ask after one of the top tier one, uh us uh telco So when we came in the code was there for that telco and we could just have it My experience has been everything from like they're like hey, this is a really great idea to You know, absolutely not, you know, um And I can tell you like we talked about um, what challenges did we have in the vnf slash in a v space? I can tell you this is one. Um, I've run out of licenses And i'm in an air gap world So I can't just like suddenly like spin them up on demand I have to go through like some cumbersome manual po process And like Virtual firewalls don't do me a whole lot of good if I can't just spin them up on demand Like at that point I have a virtual firewall that from an operations and supply chain standpoint Basically deploys the exact same way that a physical firewall does. Um so you know and We've used some methods like and this is what i'm actually hoping for right is when we talk about a best practice I would I would propose even though I said I wasn't going to do this in this phase of the conversation Like a best practice would be like a true-up model where like they have some type of you know in-house monitoring server that is sitting there and it's just calculating You know, mr. Jeffrey here's total consumption of a cnf and then right I take the bill once a month want supporter or whatever and I say this is how much I consume and they say this is how much you owe me All right, and then we can have a long conversation on how that gets into some difficulties I mean, I'm not saying they're not insoluble But please bear in mind the most important thing you could do right now is write down why that works better than the other solutions you've worked with because You know, again, that's a solution. The problem statement is critical Yeah, I wanted to add is yeah, no, but what I wanted to add was another example that I've lived through Where we had to be creative basically a ucp solution that we're selling to enterprises And we were also selling of course vnf that you could put on that vcp. So the customer could order firewall or other type of vnf that we would pre integrate upper in the chain to manage licenses for these elements we we came up with, you know license inventory so we would We would get in a pool of licenses and as soon as we do an instantiation we get a license from it market as used consume But but again, that's There's a lot of work Up hand and it's not answering all of the points that you raised here But it's another example where it's completely air gapped Um, because it's quote unquote human that feeling the inventory of the license are available to have that license pool and the orchestration which is Whenever spin up the vnf in that ucp, but it could be a different use case Uh, but that's what I've seen Well for the packet management, what I was suggesting is don't don't actually have a Packet management that is a proxy rather have a packet management that is completely Isolated and you push it and you push the package within that package management manager Sure, sure. You know I was like that wasn't in reference like you just as you were talking it made me think of things and I was just typing down like tangential notes So when you're saying push it, do you mean a package image or image repository as well would be part of the package management system Yeah, it could be anything like container image could be java artifact, you know, maven repository So artifact repository Thank you or repositories Um Well, one or the other I was I have one technology in mind, but it could be any any technology that provides artifact management basically Artifactory harbor, there's a bunch, right? And you can there's some that combine multiple things and then there's others that split it up. So um, so We want something like artifact scanning And so that's part of the CI CD, right? So before you push things in that Artifactory or that Nexus or whatever is the technology then your pipeline would But that's part more of the development pipeline Whenever you're you're you're producing your artifacts. So yeah, you would artifact scan the artifact or scan the result images for vulnerability and I need that scan you could You know, we talked about policy at some point, but you could inject policies to check for instance Hey, I don't want any artifact or container image that has curl in it because you know curl, you know It's not going to work in in the Are not supposed to work in the in the secure environment Well, it does work. It just doesn't work to the internet, you know, there are reasons why you might want curl I mean, it's a perfectly reasonable policy if that's the one you choose, but I mean, it's got its uses in different sort of Ways you you'd be hard pushed to find something that didn't have hdp client code in it somewhere But you what you're looking for is that it has no means to attempt an internet connection Correct. Like basically And I since we're getting into this and I mean some of this is like artifact scanning I don't know if that would go in package management or in, you know, best practices for gates But once again, this is the type of stuff you have to figure out, right? Like If you get a bunch of helm charts or a knit containers that do have curls inside of them and they're pointing to Repos, um, even like your kates platform, right? Like if it thinks it can just go straight upstream to pull a new version of cubelet, um You have to go in and basically make sure that everything has been cleaned up and it is pointing to these private repositories and then You know, you have to have some type of other automation That is watching the upstream repositories and pulling those in and making them available. I mean, it's doable It's just a lot of work and I mean I think actually it's not doable. You're trying to prove a negative Is this ever going to go to the internet to do something? Well, I can check that for every circumstance I've tested it isn't doing that But I can't tell you that it's never going to try doing that because I haven't found every particular way of, you know poking it So, you know, if you're proving a negative you can't I'm not trying to prove anything. I'm saying that I can set up things to where I pull dependencies to a private repository and then As I'm going through all of my testing and validation I'm ensuring that everything that I do points to the private repository Will I catch everything? No, but then it'll break and I'll go in and say Oh, this broke because it was trying to like curl to this URL that is out on the internet Like yeah, I'm not trying to prove or disprove anything. I'm just saying like you can Do this and I have to see how much I'm allowed to share what we're doing internally because I don't want to get myself in trouble but um But like there are ways to do this and you could argue that doing it air-gapped in the beginning was not The right solution to start with and we could have those talks But um, but I'm just saying like what is the best practice for smart licensing in an air-gap world because That's just a reality that a lot of us service providers have to live with and so I'm hoping smarter people than me can help me in this group Propose some of these ways to like overcome these challenges. Do you think a best practice would be? Splitting the dependency Not having any I guess it's like a full Reference where it it hardcodes dependency should not have the repository hardcoded with the artifact So that you can Save the artifacts to your own private registry do scanning separate from the image version That much is absolutely true, right? It's never going to be where the developer put it in the first place It's always going to be on a different repository. So absolutely repository URLs have to be flexible So that so that could be maybe a best practice like they I don't know How often it is but there may be helm charts out there I've definitely seen it another code that reference the entire Repository as the URL and I'm using url in general may not be htp But they designate the entire thing So then those helm charts would not work. You would have to do something like Cat a proxy caching or a fake it with dns and That's right. Yeah, right So if if we could say if we could find some stuff it seems like there's some stuff around here that we could push on Push forward as best practices on how you should write your helm charts and I just realized we are over time Yeah, I have to go Taylor. Um, those are Those in the notes because those are key points I want to talk on later is like additional stuff like when you reference an entire repository If you pull all of helm you get things like bitcoin mining charts and things like that So there's a lot to unpack and it's also more reasons why we air gap But I'm now a couple minutes late to my next call. So I will chat with you all on slack in next week Thanks, everyone. See y'all next monday day all