 Hey, how are you all doing? I guess we'll wait a minute or two to let the people stroll in. I actually got a whole bunch of responses to this. So yeah, let's see. 19 people, so they were going to show up to this. So pretty excited about that. How's everyone doing today? I'm awesome. How are you? I'm busy. But a good kind of busy. I haven't even eaten breakfast yet, so that's my kind of busy. Brian, it's Potski Day here in Chicago. It's a Tuesday, man. I don't even know what those words mean. They're like magical pastries that all the Polish Yeah. Like I'd never heard of them till I moved up here, so don't feel bad. I'm so jealous I moved from Chicago to Texas two years ago. Yeah, my spouse just got back from Kroger, and they're like putting together like 12 packs for people. It's pretty wild. All right. So we got, we got, we got some, it's 12 people here now, so that is over half the people that said they were going to come. So let's get this kicked off. So I am Brian Lyles, and I see that my, one of my other co-chairs is here. Alice, how you doing? And what we're, the goal here today is to, let's create a work group for air-gapped workloads. And I only have a couple things to say. One is I'm not gonna leave this. We're going to find someone else to leave this. And that's actually, that's like one of the biggest things like, you know, okay, I got a lot of other things going on right now. But I am super interested in making sure this gets going. So we want to find out we need to find someone to lead this effort. And then two, we need to figure out, all right, you know, people are interested in air-gapped, but we need to figure out what we're going to do. This is a work group. So hopefully there will be work coming out of this. So we need to figure out what that work will be. And we need to track it and then make sure that it gets reported back. So everyone, and that's the goodness that will come out of this. So I'm actually, I don't have much more to say in this. I am interested in what comes out of this. But like I said, I am here as a participant. And we need to work on figuring out how we're going to get someone to to lead this effort. Alice, do you have any suggestions or? So boring you got. I'm not even talking to you. Nice. You know, so I think it really depends on the people here. So I agree. We see ourselves more as the facilitators to enable this and to communicate it with TOC. So I think it's a work group. I think actually people working on air-gapped environments and have already good understanding of what is needed and what should be done would be good candidates. Yeah, as someone whose organization is just starting to head into this this quarter and next quarter and the year to come, I definitely need to hear from others with experience in this area, especially around the context of CNCF tooling and everything between. So I'm definitely looking for somebody with with some prior knowledge to lead this. All right. And here's the best part. As with, I mean, this comes with a little bit of time. And it doesn't have to be one person. What I'm looking for is maybe one or two or possibly three people to help lead this up. I know, put everyone on the spot. Everyone wants to come and get all the goodness. All right. I mean, if not, you know, then either Alice or I will have to at least take it from now. And then we will work on getting someone else. But I don't want that to stop us from one of the actually one of the bigger things is on the million list, a little bit ago, Alice actually talked about doing this whole work group. And we got a lot of good feedback and people who want to be interested or who wants to be involved. Maybe we should go around. And while we do this, I will get some notes started. What what are we looking for here? Because I because I think that I mean, what I think of is air gaps. And just in my situation is, I have a Kubernetes cluster that is not connected to anything. And you know, that's very vague, because I'm thinking of it in a very vague way. What do I expect to work? What do I expect not to work? And what could be approved? Yeah, just that we don't have like that long moment of silence. There was also the outcome of of KubeCon where we had this discussion that was exactly also the idea for a working group is to have actually something to work on. And that was the big question. Okay, I like like this nice internet connected Kubernetes environments. But what if I have to ship a Kubernetes, and also my applications, and I'm not connected to the internet, I'm not connected to a public registry, I really have to run everything on the spot. How do I do this, which I think would be a good target for the working group, figuring out a best practice on how we do this from tools and what people are using. Yeah, don't talk too fast. I'm creating this document. I can. But I think we're the same page. It's really have a Kubernetes environment. It's not connected to the internet. You have to manage obviously Kubernetes as infrastructure, plus also the employer applications there. So I guess I can chime in. So we've been running air gaps for quite a while now. And one of the biggest issues we run into, and one of the things I would love to find is good tooling for determining even the artifacts I need to provide, especially with the explosion of operators, it's getting more and more difficult to determine the artifacts, the Docker images that we need to actually serve up. So it's a lot of testing in an online mode to figure out, Oh, these are the actual artifacts being pulled in. So I think that would be a good, or what I would hope to get out of this is tooling to help me, I guess better determine that. And go next. We're running Kubernetes in like a commercial environment. So it's on AWS, but we're also running it in fairly regulated and constrained AWS Gov environments. So they're not like truly air gap, but for all ints, extensive purposes, they are because we have to go through really, really rigorous process to move things from commercial into that environment. In terms of like moving images into private registries through a process, how we get any kind of operational control in how we upgrade components for Kubernetes, how we treat the life cycle of applications. So I'm super interested in this space still just trying to figure out how we best manage, we have a whole bunch of things that we've written, a whole bunch of things that we're using to do that, but how we actually make a best practice out of that. All right. One second here. I'm gonna, or never mind, go ahead. So, hey everybody, I'm Chris Short. I work at Red Hat. We do a lot of disconnected and air gap installs for like US government type folks like on-premises. So they have, and like just talking to these customers, like I feel their pain. I was in the intelligence community and communications community for the Air Force and previous life. So I completely understand what they're going through and trying to have these completely disjointed and disconnected environments, but also like you have these disconnected environments where there might not be, but like one internet terminal for like an entire facility. So like getting that piece figured out and is like encapsulated as possible, I think is good here. I know at Red Hat, we have a lot of documentation on this and like working with like Jeremy and others, I think we can come up with a really good best practice in that regard. All right. Thank you, Chris. I put up a document. I'm actually unsure how to share this thing. I can't share it to a mailing list, so I just posted a link and we'll need to figure out the sharing for this, but at least we have a document now. Or did you post this link? Yeah, where do you put this link? Oh, I put it in, I'll post it here. I put it in Slack, but I'll put it here too. There you go. Jeremy, record. Anything from you? I can go next. This is Ashish. So we operate on a heavily regulated environment where we have Kubernetes clusters running in, but as some of the previous people mentioned, like everything is proxied through like upstream proxies for us to do installations, which makes it extremely difficult for us to package and ship an application because if you have a dependency which is not available through an existing proxy, it probably takes weeks for us to wire up all those informations, proxies, firewalls, and all those things in order to get it deployed. The second part of the trouble that we have is when we ship some of those things, we know that, you know, we tested it on our side and we validated it, but from an end user standpoint, they may be looking for some attestation or some sort of validation on those particular images. So how do we validate something that I checked in my environment on that specific deployment side and then confirm that, you know, this is same as what Ashish shipped? So those are two of the challenges that we have with the air gap, I mean, primary challenges, I would say. Yeah. So I'll go back. I think I mentioned that we're doing things, we run clusters in the commercial environment and in the kind of more restricted government environment, we definitely have that attestation need. We have to verify that like the things we're shipping across this gap are what we tested and ran in the commercial environment and we also have to have like a receipt. So it's got a transaction list for all the, for everything we've put through this kind of gateway or diode thing, we have to have a list of the transaction IDs associated with each one of those things. And like one of the problems we've run into is that we've either missed those transaction IDs or somebody has pushed twice, because there is some automation that kind of drives through this process, but it's kind of broken. And then getting on the other side, we've missed a certain image or we don't have a good, right now, we don't have a good story for like, what's the manifest for everything that makes up this application? What's the contents of, you know, running service X, it may have four or five containers. How do we make sure we get all of those things across at the same time? We're going through this like FedRAMP high audit right now. And part of that has been fixing up images and making sure that we don't have vulnerabilities and certain things. And we missed certain versions of containers that ran in that application. So we went through this fire drill this weekend, like, oh, crap, we have vulnerabilities. So what happened? And it turns out we didn't actually ship across version X or version X plus one of certain certain things, because we don't have a unified manifest or definition of what that application is. Yeah, I think these, what we should really try to get into this document is like really all those questions and problems we have in a semi structured format. Another thing where I heard is we have, I think even different definitions of air gap that don't want it like to be a definition working group, but we have totally disconnected, we have regulated via proxies and we have, we can download it from the internet, but then have to transfer it over. And otherwise it makes a lot of difference for the different use cases. I think it would be good to start like for the air gap environment that everybody uses, what does your environment actually look like? And then we have specific use cases. Obviously we have to deploy Kubernetes there somehow, which is ideally something we can more easily handle than how do we ship an application, which problems that we run into. I think if you have to full list there, then we can more or less, as it's a working group, like really work from top to bottom, where do we run into where we have input? Because eventually the outcome of this should then also be a report back to the TUC and maybe a report back to other working groups, where we've, for example, or other projects, or the Kubernetes project if we need something. Right. Can we take it? Go ahead. Could we take an iterative approach to this? Because I feel like the disconnected install, like or disconnected environment kind of goes the gamut, right? Like you have some level of disconnection and then it goes to like full disconnection, right? Like there's a spectrum here. So I feel like you can iteratively do this, like peeling away, peeling away things that you don't have access to, right? Like if you don't have access to X, you need to do Y. If you don't have access to Z, you need to do W, for example, right? Like so forth so on. Yeah. I kind of like that approach of having more like a cookbook of like how you address certain restrictions. Do you have, do you have the full like DOD, intelligence community, burn everything to a USB or to a CD and like send that on an airplane somewhere or do I? People laugh, but that happens. Yeah. It does. Yeah. That was my previous life a long time ago. Deploying stuff to other countries is not easy when you have no network connectivity. Right. So what I'm hearing, Jeremy, and maybe this is distilling down on what some of the artifacts that might be created are is maybe it does start off as a cookbook. And, and you know, with like the old O'Reilly style, I think they did a great good job with that, you know, identify situation, you know, give some words there. And, and then, you know, if we get that right, then the next step could be actually implementing some of those ideas, automating some of those ideas and making that available as tools. So that's a, that's it. I think that's a very sane way for us to see at least my point of view. And it feels like there's a couple of different ways to tackle this as well in terms of like part of the concern, at least my part of the concern is really the Kubernetes installation part being totally air-gapped, right. And then there's also all right, here's the stuff that user X is running on top that has to also be, you know, coming from internal registries or whatever, right. And so at least for me, the, I'm far more concerned with the installation part, because as you know, Talos is, we just have users that are, you know, have European government clients essentially that are trying to do totally offline installs, right. So if we can get that working, you know, that's hands off for me at that point. But yeah, I mean, yeah, there's definitely kind of two parts to it, it seems like. That's definitely like, I think the best way to focus on this is multi-directional, right. Like come from it from both angles, complete disconnected, and then some level of disconnection. Yeah, our groups have that. Yeah, exactly. Our, for instance, our concerns aren't necessarily around the bootstrapping of the environment. Like we're in a, we're working on a lower level of FedRAMP right now. So like an initial bootstrapping of the Kubernetes cluster and some of the infrastructure like in Madrid, etc. isn't, isn't in scope, but the day-to-day operationalizing of deploying to that target is more in scope for us right now. So. Yeah. And so I think we're kind of saying the same thing. I really just sort of hearing some reoccurring patterns. And, you know, what do we actually have power to change or do, you know, as a CNCF working group. And I would love to see the outcome of this be almost a set of guidelines for, if you're a CNCF governed project or product, here's how you do some of the air gap installations. As you start to peel away those layers, now it doesn't have to be your first how-to, hello world, read me doc. But as you start looking at some of the CNCF products, it'd be great if that was almost a requirement or a friendly nudge to cover the air gap installs as part of being a CNCF umbrella piece. You know, I think one of my biggest struggles, and I think Spencer might have touched on this, Ryan, I'm not sure, is images are everywhere. And a lot of times the manifests are pulling from a specific publicly available registry. Now there's tools out there, Helm customized things like that that solve it. But it'd be great if this came packaged in with the delivery mechanism of said application. You know, here's how you deploy it using it from a different image repository using whatever tool they want or some form of example that can be built on top of. So the deliverable becomes more of a friendly nudge or guidelines around what CNCF government projects look like and you know, what would be best practices for air gap installations, whatever air gap means to you, you know, to whatever flavor that means. And I think that those are actionable things that you can take away and apply to the community in our control, I think. Yeah. Oh, good, Chris. So John Roach in chat, I don't think he's on voice. He mentioned that it might be easier to start at for this group, like how to install Kate's in a disconnected environment, right? Like just get it up and running. What level of disconnectedness and like that would be the output initially of us, like doing this work, and then we can iterate on top of that. Just wanted to make sure he was heard. That's it. I know. I mean, that's that's a good point. And you can't do everything at once. And is there consensus? Oh, is there consensus around that? I mean, is anyone object to that approach to begin with? I think it's a good idea to start with Kubernetes first, because it doesn't make sense to talk about the running applications and something that they put in the first place. So yeah, so I think that is a, I mean, I'm just trying to, I'm trying to whittle down to something where, hey, this is what we can, we can actually rally around and still looking for, I will ping again, but still looking for someone else to run this meeting and then decide, how often would you want to meet? Is it we're meeting today on this Tuesday? And I just picked it because it worked for me to tell you all the truth. It's just, it's time worked for me. But going forward, what's the cadence that you would like to meet at? I'd probably suggest something every couple of weeks, just because of there's a lot of meetings out there. And then start figuring out how the coordination is going to work. Is it going to happen? Is it happening in a repo? Or is it a doc? Is it a Google doc? It just gets started with. I think that's the lowest amount of friction for me. I would just say, ask my Google doc and start typing in there. And then the group, whoever it is, can start working on that. Yeah, so I've just... Hey, Brian, did you see that Jeremy just volunteered his tribute to Hopefully? All right. Now we have two people. All right. I think that's, you know, that's a, that's a good start. I will not press on that anymore. So thank you, Jeremy. And thank you, Ryan, who pinged me separately. Unless there's any dissension here, there's no dissension. We will move on with this. All right. Gabbled. Let's move on. And we'll just do one more thing that we could trigger, Brian. We should... We know we have, actually, because we talked a lot about fan ramp, we have a lot of end users in the end user community in the CNCF from government organizations. And they might just not know that we're working about this right now. So I'm like the U.S. Air Force and a couple of others who are actively... I can reach out to Nicholas, Shillian, if you want. You might know them as well. So they just know this exists. This was one group. The other one that initially started this discussion was the telco working group simply because they deployed to telco hardware. So we can just ping these two. So if you already know people do it, but we can also rock it via Amy, hey, can you connect us to the right people that this actually exists? Yeah, I think that's a really good idea. Have Amy connect us to as many people in the end user community that might have good input for this? I think there's probably a lot of people that are doing this stuff that don't necessarily participate in meetings like this all the time or watch mailing lists. And it would be good to kind of reach out and proactively try to get them involved. Oh yeah, definitely. And we also have the federal reserve here. So that's already great. So there's also a research users group for CNCF, which is largely academics that are doing HPC type things. And a lot of them also have some form of air gap disconnected, which I think would be good to reach out to them. Yeah. So I think this is something where we can help also as the state chairs to help with this, with if you have been working on that document to connect to the right people and get them involved here. All right, sounds good. All right. Let's see here. So I guess between Ryan and Jeremy, now this is your show. It's no longer, you know, I can not say anything else for the rest of this call. As you all can see, I'm a professional delegator. This is my job to delegate and the CNCF and being aware is all I do. So what we'll do, and actually, I don't know, unless you all want to talk more today, I'll be here from the entire time. But this is really, this is Jeremy and Ryan. This is your show now. And Allison and I are here. And Harry are here to support you in whatever way we can. Awesome. So I think I like the approach of starting with how we deploy Kubernetes in an air gap environment. We should just start a Google doc and we can start kind of capturing ideas in there. And then once we have some content, we can add some structure to it and refactor it a little bit. Is everybody in the the SIG app delivery channel on Slack? Is that the best place to communicate? I see Chris just joined. I always hesitate saying like Slack is the primary way of communicating. Do we want another mailing list, the regular? Or do we want the regular mailing list? I mean, it's not that much regular one. Yeah, let's just use the regular one. And because it's not much traffic on that list. Now we'll use it. So actually, there's more people subscribed than we know about because we really there's way more people usually replying to an email that we see either on Slack or in any of the meetings. Yes. The mailing list is the best starting point. So this is the first thing we want to work on. You will see that you will get great more engagement out of this. All right. So I don't want to put Ryan and Jeremy on the spot. So we need to give them a chance to get themselves together. So anything else that anyone wanted to bring up that we did not bring up? At least get started. Is anybody just maybe this is a little early for this, but has anybody discussed how CNAB might fit into this picture? Or is I mean, that's pretty big subject, obviously, but we're thinking very, we're leaning very heavily into CNAB and thinking in that direction. And one of the things we're trying to keep in mind is that we might be able to use lean on this back to help us realize what one manifest where artifact might look like to be shipped in an air gap manner. So I'm sure that was something worth bringing up. So Carolyn and I both work on CNAB and we were the authors of a tool called Porter that does CNAB stuff. I think definitely it has a place in the, as a solution that people can take a look at, whether it's appropriate for every situation or not, I think has some debate still. I think we learned some things regarding air gaps from working on CNAB. We get into the security and attestation piece of that. I think we kind of settled on three definitions of what air gap meant in that situation. Whether it's you have no connectivity at all, whether you have some limited connectivity, and a lot of that drove the tooling piece. Notary, for example, had some ties, some need to have references back to original signing keys and things. How do you ship those across a completely air gap network and not an air gap network? I think we found that from just the manifest piece and what the bundle looks like. I think that's a pretty decent stab at listing out images that you need. So I think it's definitely a thing to look at and has opportunity to address some of your needs. I'm not trying to sell CNAB as the solution. There's going to be a lot of tools in this space. I think it's like CNAB might be a great solution. Don't get me wrong, but I think it'd be more important to focus on not necessarily a high level, but more process and more expectations and not necessarily tools, if that makes sense. So Jeremy knows this already because he saw this two weeks ago. I looked at what CNAB did and I said, this is some good ideas, but I wanted to solve the problem of image relocation just in general. Assuming you have a cluster, it's there by magic, and you had a set of manifest and you want to just take those manifest, figure out what images were involved, package that thing up in the tarball, ship it over USB, CD, DPD, does not matter. And then when you expand it on the other side, you could automatically get it. You could automatically rewrite your YAML even before it was hit into the cluster, and it would fix all the images. And I'm calling this thing chief, and I could talk about it because it's almost done being open source at VMware. I don't understand the problem, so when I don't understand problems, I write code, and I will be sharing this with this group. And it was just to solve that problem. I have manifest, and they talk to images, and I want to move them somewhere else. But that led into a thousand other problems where you realize where 99% of everything is in a pod spec template. You can find images, but people use things like for arguments, they actually just hard code the image right in the argument. And now you get to the place where if we had a great convention for specifying images in a config map, then any tool could guess. And actually, funny enough, on our VMware Slack, I threw that bomb to our K-Native team this morning, and now it's spawned off this huge amount of meetings. And, yes, goodness what happened there. But I would like to talk about this because we can't be the only people that are having this problem. Everyone is going to run into this. And I would like to see other solutions. Mine's a POC. I just wrote it in a weekend to say, hey, I think this could happen. But I would like to see not only tools in this, but exploration of conventions too, because there's things that we can do. And then ultimately, I mean, ultimately, we wished that Kubernetes could support this. Like, Kubernetes actually, if Kubernetes supported image relocation out of the box, we wouldn't even be having these conversations like that. But what will we actually say, and how could we actually spearhead that, or tell or work with someone to spearhead that effort? It will take two years. I swear it will take two years to get it done. But let's see if we could get that done too. And that's a weird place for this group, because we're CNCF and not Kubernetes. But at least we can formulate a good want in that space. Like, this is where we would like to end up. Yeah, I think that's a great idea. Exactly. Like, coalescing on a standard first, or at least a convention first, I think is a good first step for getting the idea to spread as well. So yeah, I like this. So one of the main issues we ran into with the image thing that you were discussing, Brian, and I don't know, I guess the full implementation, but when you do that, sometimes you lose some of the signing and you can't validate that it came from upstream. And so what we've actually done, which we've found quite a lot of success with, we've switched to the container DCRI, and they allow much better proxying to upstream. So we just proxy every single possible upstream URL into our own, and then we validate the keys. It's not great from a signing perspective, but it's allowed us to not have to worry about re-imaging and taking images to meet our internal registries. It's far better than the Docker's approach. So that's cool. Actually, it's cool to do that. Yeah. And I think you should have a list of everything that people already worked on and did like as a multi-collection of things we should eventually be discussing. I think we have a lot of, I think if you go over that recording again, you will find a lot of things that we eventually want to discuss. I think it's good that we have a point to start somewhere, but because also it's like significant work is eventually, well, there's a dedicated session at KubeCon on air-gapped environments. We do have a session on app delivery work, and it would be good to dedicate at least some parts. Okay, this is what we're going to do around air-gapped there. Yeah, that would be great. And then there's great exposure in video, so it would be good to get that message out. And keep in mind, I scheduled this for 50 minutes because I was not sure what we were going to talk about. So you don't feel that you have to expand it to the 50 minutes. But Jeremy and Ryan, if you need some help getting things going, Alison, Harry, and myself are here to help you. And you can definitely lean on us for whatever else you need to. And then we're still trying to figure out how we work with TLC2. So this will be learning for everybody. And you also have, if you have questions and we want to bring them up, there is a meeting that Brian, Harry, and I have like on Mondays where we discuss more of the organizational cherish type of stuff. And usually we don't have that much to discuss right now because most of our work is actually going into the working groups. So if you need some discussions around bootstrapping for this more organizational point, I think we can also invite you to these meetings as well. Okay. All right. All right. I think then we're done. I think nobody else is meeting like today. No, I mean, let's, yeah, I think we're done. But if anyone else has something to say, send it to the money list. There we go. And I just want to say thank you, Ryan. Thank you, Jeremy. Thank you for letting us take your time for no pet. Thank you for putting this together. We thought we were the only air gap people for a long time. So Oh, no, right. Real cool then. All right, everyone, we have a meeting. Was it next week? For the main? It'd be nice if we could report. We'll definitely report on the status of this. Yeah, cool. We put it on the agenda and we have Jeremy or Ryan to talk about what you're going to work on. So it's in the hour. Yep. Great. Alrighty. See y'all next week. Yeah. Thank you, everyone. Thank you. Bye. Thank you, everyone.