 Okay and that's two o'clock. How's it going? My name is Jeff McCoy, I am for Central Command, I am stationed in Silicon Valley right now, perpetually TDY to Colorado Springs. So you are here if you aren't sure this is for Kubernetes, Kubernetes being built for space and for air operations. So if you're in the wrong room, this would be a good time to. Can you close the door? Yes, it was not Wi-Fi. Okay, before we get started real quick, who in here has any idea what Kubernetes is at all? You've heard Kubernetes before, you've seen KADS. Okay, who in here can explain Kubernetes to your co-worker? Who can explain it to your grandmother? Okay, cool. Good audience. I just want to make sure we have the right audience here so I know who I'm talking to. So like I said, I am currently stationed in Silicon Valley. I am TDY right now in Colorado and I was one of the original Kesslerun engineers, so one of the first two guys on Kesslerun. If you've heard of that, it's agile software for air operations based out of Boston now. After a Kesslerun, I moved on to DIU where I stood up another organization called Rogue Squadron that does COUNTY-RAS and then from there we've set up two more, what we're calling camps that serve camp and space camp. So what is space camp? Space camp is essentially, I think, Kesslerun for space. Essentially what it is is we're trying to build agile software for the National Space Defense Center and the C-SPOCK, so Vandenberg and Colorado Springs. We're basically building apps and I continue to model much like Kesslerun did but for space. So I'm the CTO of space camp and surf camp, but I'm also the CTO of level up. You may not have heard of level up yet. Level up is part of unified platform and have the best rep, but we are rebranding it and we're trying to take the good things that Kesslerun did and the good things that space camp did, the continuous ATO, the continuous delivery model and rebrand that and build it out for other organizations. There are actually 34 software factories right now across the DoD, so Kesslerun is one of 34. Now Kesslerun's a pretty large one obviously, there's 650 people, but just so you know there's actually a lot of this activity happening across the DoD right now. The DoD is trying to figure out how do we move faster, how do we deliver value to the warfighter faster and one of the challenges I had in particular was how do we do that in a cost-effective model. So Kesslerun is a fantastic organization. You've probably been to some talks already about Kesslerun. They also have one minor caveat that is they have to use PCF right now. They're moving to other platforms. PCF is very expensive. It's a fantastic platform. It does a lot for you. It really provides the Heroku-Luck experience, what's called the 12-factor wrap. They've built it to just push code and it magically is built up. It's awesome, but the caveat is it takes a lot overhead in the classified environment. It's not as easy as just push a button and you have software working for you when you're talking about classified air-gapped operations. So one of the things that space camp would look at was how do we do what PCF did for us, that amazing push-butt experience, but can we look at other technologies to do that and specifically how do we do with microservices? Microservices, if you're not familiar, it's just a concept of distilling down functionality into small chunks. And that's complicated to orchestrate those together. So if you look at the history of Kubernetes, Kubernetes came out at Google. It was a technology called Borg. Borg is used today for what they call a plant and cell computer. It deploys 8 trillion, I'm sorry, 8 billion containers a week, that's 660 containers a second, and they're doing this at little global scale. Now if you look at the J-Wix problem, the TS SAP, the SIPR SAP, the SIPR-REL problem, we're not talking about billions of operations a second, but we're talking about needing high availability. I can't just have software doesn't work if wars are counting on it. I can't have software doesn't work if space control is counting on it. So when we look at Kubernetes, we don't look at it for the, we're going to serve 10 million customers a second. We don't have 10 million customers a second. What we're looking at is can we guarantee transactions? Can we guarantee horizontal scale? Can I not have to worry about the infrastructure I'm deploying to? So when we chose Kubernetes, we looked at it because it seemed like the right solution, though we're not solving the million ops a second problem, we are solving the high availability, the scalability problem. And so for that, we chose Kubernetes, but Kubernetes itself is complicated. It's not trivial. If you've tried to learn Kubernetes, if you try to spell Kubernetes, you will know that it is not something you can just flip a switch and learn tomorrow. It has taken us a lot of blood, sweat, and tears to comprehend what it means to deploy to Kubernetes. So we started Space Camp officially in January, our first line of code was January 7th. We deployed first to Docker Swarm, which many of you probably have heard of Docker Swarm before too. It's like baby's first Kubernetes thing before your Kubernetes. So we did that in March, realized that it wasn't scalable, it wasn't maintainable. Yes, it was simple, but it wasn't reliable either. So then we looked at Kubernetes and it took us about two months to really figure out just the very first concept of how do you deploy Kubernetes on the high side? Because if I go to Jwix right now, I'd go type in a command to install tools to install Kubernetes, it'll work. But I try to spin up Kubernetes and it doesn't work. It doesn't work because Kubernetes packaging expects you to talk to GCR.io. So an interesting fact about all of Google Compute, you can have on-prem Google Compute that still has to tie back to the cloud computer, right? They call it their machine or their planet scale computer. So even if you're installing on Jwix, you think it's working and then it just starts breaking because it can't finish the installation. Why? Because the packaging that Google's built and the IBM and Lyft and these other organizations that are built assumes you have internet connection at some point. So it doesn't work for us on Jwix or SIP or SAP or anywhere. So we step back and said, OK, what do we need to do to make this actually work? Now, I'm going to attempt to do a demo on this guy right here. It's worked before, but I did blow up stuff earlier and I just got a keyboard in here a few minutes ago, so it should be interesting. But what we found was that there's a lot of little steps you have to do to make Kubernetes work on the high side. So we started toying with building them out in a bunch of bash scripts, a bunch of shell scripts. It was about 25 scripts we created to make the process work reliably and stream it. But I didn't want to just deploy Kubernetes. That's actually not the hardest part of Kubernetes. The hardest part of Kubernetes is the day two management, which is how do I upgrade those applications? How do I change those applications? How do I maintain the state of those applications? How do I recover from disaster with those applications? So there's a lot of tooling out there in the Kubernetes ecosystem that allows us to do this, but it doesn't work well on the high side. So after a couple months of playing with a bunch of bash scripts, we decided we wanted to kind of codify this a little better so we could test it. And so we had to learn the language. So we chose go lang. Why did we choose go? Well, go can deploy anywhere. So if you're familiar with the Docker Scratches, that's essentially an OS with no OS on Docker. Go runs there. And so our thinking was let's build it with something that can go anywhere so that we can just give it a brand new computer and we can put all of our tooling on it, all of our application state, all of our systems in a declarative fashion. Now when I say declarative, this is important because I was in San Francisco about two years ago for Kessel Run and we had a situation come up with one of the first apps, Jigsaw. We couldn't understand the state of the application. Now we're in San Francisco. There's no access to SIPR. The apps on SIPR, our operators are getting mad and we had to do something. So literally I had to take a flight, same day to Qatar to go address an issue because I could talk to the operators. I'm talking to the sys admin guys. They couldn't tell me the state of the app. They couldn't tell me what version they deployed. They couldn't tell me what was being used. So I had no idea as an engineer what was actually on the high side. So our fix was to get three plane tickets and go hop to the Middle East for a day to go fix it. That's a terrible solution just in case you're curious. Also very exhausting. So we thought about that. That stuck with me, right? So PCF is amazing. It's a great technology. Class and environments are a challenge. How do we do declarative? That's the real key. There's a concept to this, very important. Imperative is a state of, I have a GUI. I'm an operator. I'm going to go change something, right? That creates a new state. Not your data, not your application, but the system administrator. I'm going to change a configuration. That changes state of the application. If it dies, the sysadmin answer is, well, you restore it and you back up, restore it. The declarative answer, the Kubernetes answer is, you don't need to do that. You've already declared it and if it dies, just recreate it from the previously declared state. Why does this matter? So if you look at any infrastructure as code solution, which is codifying how you declare your infrastructure into code, you can then version control that. So now I can say three months ago, we had this orchestration, this configuration. I can now roll that back to that point in time into exactly where I'm at. So we took this concept of infrastructure as code and we kind of rolled it into what we're calling platform as code and essentially it's the same concept except that we're committing to and to get all of our manifest declarations. Now here's why this is really cool. So if I'm a developer and I want to see the state of my applications, we have one that's called Astro. It's a MIT Lincoln Labs refactor. It's a rewrite of one of their PhD apps that's turned into an actual use system on the ops 4 at NSDC. We're making it professional now. It's a bunch of microservices right now. They have to work together. They have to talk to Kafka, which is an event stream system. It's kind of messy to deal with. It's complicated. On top of that, we wanted to add security so we added Istio. If I tell a developer, go start up your microservices to go test this, they're going to say how. Their job is to write code, not to orchestrate microservices. They don't care about that stuff. They just want to write code. So I don't want to have to hire somebody that both is an expert in code and infrastructure. That's too tedious and too expensive. So what we did was we took those manifests that were deploying in production, the declarative state of infrastructure, and we pushed it all the way back, called shift left, all the way back to the developer box. So what this means is when I'm a developer and I sit down on my terminal, I can spin up the environment literally identical to the production environment. All the controls, all the restrictions, all the ingress controls, all the authentication pieces, every piece of state that's derived from those manifests I can also do now for the developer. And you may be thinking, okay, what do you do? That's not a big deal. This is actually enormous. What this means now, and this actually happened last week, if I have something that's dependent on something else, we call it chain dependencies, and it breaks something, usually you don't see that to your last leg integration test or last mile pre-deployed production. So you put it up on Jwix, put your test, and then it breaks. We now see that as a developer's writing code. We just took out hours' worth of testing and automation and shifted all the way back to the developer's box. So the minute that he made that line change that created that dependency that caused a cascade failure, he sees it in his box. So one of the things about Agile is you're talking about reducing feedback loops, right? So you go to the user, you get feedback, you reduce the feedback loop. That's how we deliver value faster. What we're saying with Kubernetes is we're doing the same thing for the developer with their software they're building. So their feedback loop is being reduced, it's being tightened, so that they're seeing faster the things they are doing that are good or bad. Now, yes, we have tests. Yes, we have automation tests, integration tests, unit tests, but they don't cover everything, right? Don't believe the Kool-Aid. You can write really, really bad tests that actually do nothing. I've seen tests in Kelsil Run. I've written tests in Kelsil Run that were a dumpster fire. They just didn't work. Like, they passed, but they didn't actually test the behavior we're looking to test. Now, the Air Force is starting to figure this out, and we're starting to look at ATHLETECH and the test organization to say, hey, help us write tests better. So we're trying to work through that now, but for a long time, developers could write bad tests and it would just fly under the radar until it's time to fix something else. And then you're cussing the person behind you who made all those bad tests because you have to go back and fix them. I've literally seen two lines of code affect 40 tests. That's a terrible, terrible idea. It's called tightly coupled testing. So we have those tests, but what this does for this manifest does for us, it allows us to declare and predict state in a way that we can reproduce over and over. So, on this nook right here, now, this is actually kind of just the GWIS demo thing at first, but it's actually very relevant right now because part of Level F's organization is they're onboarding, and you may have seen this on LinkedIn, they're onboarding the F-16. So I had the F-16 PMO out at Space Camp last week, and they're out again this week actually testing out some of their physical modules that are on the jet to see how we can put communities on them. It's actually kind of hard because those modules are very, very, very old. They're very underpowered. PowerPC, ARM, AMD64, it's a mix of architectures, and there's not like one easy solution. And so I talked to Nicholas Shalon, Air Force CSO, about this problem because he's the one who sent them to us. And I said, hey, Mr. Shalon, here's the problem set, here's what he got, and his answer was just make him buy a new hardware. And I'm like, I wish. It actually costs a lot of money to put them on a thousand jets. So we're working through these problems. The right answer is buy a new hardware because what they're trying to do is almost impossible. But this actually is about as powerful as what we're going to get on the jet times two. So F-16's pretty old. The stuff is, I'm kind of shocked at the age, actually, but learning to work in this constrained environment allows us to do what's called computed edge. So one of the things, and I'll talk to this in a second. It's just G with stuff. One of the things that computed edge does for us, it allows us to push some of the compute to the last mile. So who's used SIPRNET before in here? Anyone use SIPRNET? Who's loved the experience? Yeah, right? We know latency is terrible. We know it's unreliable. J-Wix is the same thing. SAP is sometimes worse. So we understand that things aren't great there, right? So computed edge gives us the ability to push from that compute to the last mile. Kubernetes allows us to do this very trivially. This is not some rocket science for Kubernetes. Deploying to an edge compute device, say this guy right here, is actually not the hardest thing in the world. Deploying to large scale clusters is also just as easy. Now, we've actually found that building clusters is the simple part. Creating clusters is not what makes this whole problem interesting. What makes this problem interesting is how you manage that after the fact. All right. So right now, as soon as I meet my phone, what we did here, we issued one command, and I'm really sorry this is hard to read. Let's see if I can get this up a little bit more. So Minion is just a Go CLI we built. Fun fact, it was actually not supposed to be that Minion. It was supposed to be the fish Minion from Megamind, but we had a very dedicated officer create a 3D printed version of that, so we had to rebrand it. So it actually was a fish before. But essentially what this is, it's a Go CLI. So you go through here and you can look at what the different options are and what it is. It's not super interesting. I won't bore you with all the details there, but what it's doing, or what it just did, was it used QBabman under the hood to generate a cluster. Not like I said before, that's not that interesting. It's just a command and it creates it. What's interesting there is how we're doing that. So one of the things we do is we go through all the declared manifest. It's called YAML, it's just a plain text format. We absorb all of the images from it. We find them all, we re-label them, and then push them back into what we're calling a transient registry. What this means is we can lift and shift our entire cluster state from place to place so we can do it incrementally. And that's valuable because now we're declarative from in class to class or from SIP or to Jwix or whatever your environment is, and we're able to do it predictably. So over time, we're not just taking whole sets of images up and deploying them, we're taking pieces of images up. So the example I give is if you have 10 apps in production and say all those 10, only two changed. So the normal model would be you would take those two apps up and you would redeploy them. These could be big apps. And so one of the problems is data guards right now that's low to high guards are pretty limited in what they can do. So we're talking burning disks and all that mess. What we've done is we're not actually taking the entire app up, we're taking the slice that changed. So Docker splits things up into layers or chunks. We'll actually take that last model of changes and bring it up. And here's the alternative. I was talking to a large defense industrialist contractor a few weeks ago about how they're doing this for a space program. I won't name names. They're taking 10 apps from secret to SAP. And the way they do it is they purchase a hard drive and they load 30 gigs of data onto it and they take it to SAP and they reload that on there and then they destroy the hard drive. Re-deployment, this is how they're doing Kubernetes. So the alternative is you take physical disks, you make them burner disks if that's a thing and you bring them up the entire thing. We're not even taking images, we're only taking the things that changed. How do we know it's still safe? We're using Git to do it. So we're using a thing called Git bundle which allows you to take increments across an air gap. If you Google it, it actually says sneaker net and the word and why they did it. I think they did it from research laboratories. They do it for government, but it works really well for us as it turns out. So we use Git bundle. So what we did here while I was talking, which is just a command. I mean, if you're really curious, it's essentially just... It's hard to read, but that little line there says create the cluster, allow it to be scheduled for master, use this Wi-Fi IP. I actually can't pull it up on here because they blocked that. They don't allow them to talk, which is kind of sad. But you can see... Actually, first, I'll give you this. All right, so that's kind of hard to read, but you can start to see some stuff spinning up there. All that is is just some of the base Kubernetes things, the images that it needs to deploy, and it's deploying them from that local registry. So it actually is synchronized at local registry and it's deploying it from that, which here isn't as cool because, well, I mean, we have Wi-Fi, but when you're deployed to Jwix or SIPR, I don't know if you can read it, but it says master45000, really small text. Sorry, I can't zoom a raw Linux box, but that right there is actually, it says master45000 slash kads.gcr.io. So what we did was it was pointing to gcr.io, which is Google's container registry. We just prepended our local little temporary registry to it and pushed it in there. We're doing the same process over and over. This is how we're doing Jwix right now. So when we deploy to Jwix on C2S, we're doing the same process. We declare our state, we test our state, we then shift it up to the high side and produce it up there. Before I go on, have I completely lost all of you? 80% lost? Okay. Awesome. One person is tracking. We have 1% success. Great. Cool. So I'm sorry, it's a super nerdy stuff. I'm trying to balance the business but also like, it is, it's a very complex problem. So we went pretty far, we thought, but I really felt like we needed some more help. So Space Camp, we decided to do some contracting work. So we actually hired Rancher Labs, WeaveWorks and a local company in Colorado called Soon Innovation, which has TSSCI cleared folks to help us. So they just started this past two weeks. So they're not coming behind us to kind of clean up what we've done and make sure it makes sense because the idea here is, if you imagine like the air gap deployment process, think about an F16, right? I can't just stream a change to it right now. Even if it's networked, I probably wouldn't want to. There's a control process, there's a change management process there. I need a way to segregate that change and restrict that and to control that over time. So this same process is what we're doing with F16 right now. We're attempting to take a new building today, which is brand new, new capability and test out on these components on the jet over time. So incrementing it in piece by piece. The process doesn't matter if it's an F16 random, you know, LRU or if it's C2S, which is essentially a good plot on JWICS. The process of deploying the solution space is the same. So that's where we're at today is attempting to build that out. So I'll try to give you a couple more things when I switch computers now because this is boring to do it on here. So if you'll bear with me for one second, I'm going to move HDMI cables. Hopefully it doesn't blow up. Bonus, we'd like you both to see this screen. Make this readable real quick. See if you guys can see this. I'm a projector, I believe in you. Hopefully this is reasonably big enough for you guys. So I'm going to do the same thing. The way we do our programming in order to prove this concept out, we have to build this over and over and over. So we end up building these clusters dozens, sometimes several dozen times a day. And our pipelines, our continuous delivery pipelines are also constantly redeploying and building these out. So we're constantly iterating on this concept and adding new capability to it. So essentially what it boils down to, we have a few basic commands we use the one we care about right now is camp start because we're unoriginal. And so as it's building out, I'll talk through some of the stuff that's happening here. So we're using a technology called vagrant, which is just a way to automate virtualization. It's just a virtual box under the hood. We use it in a way to be predictable about it. So it's doing its boring vagrant stuff. After it does that, once it pulls it down, it'll start using Minion to actually create the cluster and build the capability and deploy the stuff to it. Now this is starting from a bare bones box. The only thing we did was we enabled virtual box additions because it takes like 10 minutes to install it by itself and we got really impatient. So it's starting with bare bones Linux. It's now creating all the dependencies it needs installing them. So you see get there for example, we need get so we can do the get bundle process. So it's going to do that and add those capabilities. Once it has its base OS built, it'll then start doing what's called Minion prepare. Now this is going to take a little bit and it's kind of weird what it does, but essentially this is that retagging thing. So if you think about the way Docker works, Docker essentially has labels it points to. Those labels match to a server slash some folder path essentially. It's basically what it roughs out to. What we did was we took those the server slash folder path and we just made that server part also a folder path what's called retagging and made master 45,000 the hard-coded version of that tag. So that's what's happening here. We're taking this in from Google. Don't die. It's in from Google and then we're retagging it on to master 45,000 and making sure everything's set. So it's going to take a minute. So I'm going to pause for questions right now. You have to ask questions or leave me hanging. It's awkward. Yes sir. What is the scope and scale for SMC type of missions? Where do you see it going in the next couple of years as you continue to build up that scope and scale? Do you have a customer if you have a mission? What missions are you looking for? That's a good question. So space camp's really weird. We're somewhat self-formed. One of my bosses here in the front row of the central command, he was nice enough to let me go hang out in Colorado for a few months initially. That was the initial ask and it was a partnership between central command and the Air Force Research Laboratory and a couple SMC folks, not very many really. We were kind of doing our own thing just trying to deliver it to the NSDC. What happened was Coby Washington Barreiro is a name of the organization that SMC has under Colonel Kay. She essentially said, I'm going to divert some resources to support this more and we became partners. So what happened over time was we started serving the broader SMC community, which is the C-Spot and NSDC. We were focused on one customer up front. At the same time there in Colorado Springs, there was a space accelerator so it was constantly folks running through for different programs. So we started partnering with Northcom. So NORAD now has something to deploy to the capital region to do drone protection. We built that and they're deploying that under Northcom. We have some southern command stuff. We have some cyber command stuff. We had a lot of different partnerships. And then a few months later Level Up came to partner with us as well under Nick Shalon, the CSO. So today where we're at is we're just trying to refine this process of delivering for our customer. But we're also trying to codify it in such a way that we can do what Nick Shalon says is push button factories and servers. So that's probably the one to two year vision. Though he might tell you the three month vision, he's more aggressive than I am. But essentially it's you come in, you're a wing commander and you want smart people to build software and you want it to be legal and accredited and done right. There's not a process for doing that right now. The process is you know someone who's done that before and you call them and they start making trades and negotiations and they've been MOA and you can start to partner. It's a very ad hoc process. It's all the same folks. We're all doing the same thing. If you look at all the different factories out there, we've all either worked at them or touched them or are leading them or have been a part of them at some point in the past. It's the same people over and over. So we're trying to codify a process that allows us to not just have this what amounts to a bro network but have an actual way to do this that's legitimate because right now it literally is just that it's all just who you know networking right. We want to make it to where anyone can do it and we're talking really like the group commander level, the wing commander level has resources they're going to put towards a problem. There is a separate effort that's me talking about. Which out of Bespin, which is the any American code concept. We're not looking to solve that problem right now. Lauren Asselberger has people doing that separately. This is all about level up is about a group of people want to build software as a factory but they don't have the expertise to build the accreditation and the pipeline and do all the rules and do it right. We're trying to enable that piece. Other questions? Yes sir. Yeah so that's actually a really good question. So I'm not sure if you guys are familiar with Istio but essentially Istio is a tool that we use as a part of the DevSecOps model that the CSO has established which essentially with Kubernetes when you pair Istio and Kubernetes together you really get kind of a platform. So what it does is it hijacks the traffic of every pod you deploy. So you have an app, Hello World app and it literally just says Hello World right, pretty simple. What Istio does is it puts a pod or a container right next to your container and it takes all your traffic and intercepts all your traffic at the kernel level and then it applies policies to that and it secures it. So in fact what it's doing right now well it's still a bit of the cluster right now but what it's going to do in a minute is it's actually waiting for Istio. What it's doing is once Istio is created it creates what's called a service mesh. So every single lane of traffic that flows across the service mesh is monitored and encrypted. So there's not just encryption to be encrypted actually it's mutual TLS authentication which essentially means that if you've ever used your CAC to authenticate your CAC is a strong hard token. It's a PKI backed token. What Istio does is it issues tokens or certificates to every pod and says here's your issue here's your issue here's your ID and then based on policy it allows them to talk or not to talk. What this means is that I'm an app developer, I don't have to worry about encryption. In fact we tell our developers not to encrypt anything. We just don't do encryption at all. We enforce it at the cluster level so I mean they can do it if they want to but it's just going to be overhead, you know, compute overhead as it has to do that encryption so this allows us to not worry about encryption, authentication so authentication we used to do on J-Wix we were doing MTLS auth on J-Wix which meant that when you went to one of the apps you'd have to put in your J-Wix certificate, your token and that would give you access. Now Istio does that for us. It can enforce that and your app never sees any traffic until it's already passed the gateway and been certified and met the policies. I think Istio is one of the most game-changing invasive things on an orchestration layer and I don't use that lightly. It really is from an app developer perspective. It is truly game-changing. Now from an assist admin perspective from ups, day 2 ups it can be kind of a pain because you're dealing with a lot more policy, you have to think a lot more about traffic, you're much more, you're much deeper into the network than you usually are and typically you have other systems that will do this for you. Istio does this itself which has its pros and cons but to answer your question for security let's see if it's actually started yet it's already gone through. You'll see some stuff about Istio being created there. Istio is what we use to secure the entire mesh. Every piece of traffic going in and out is either filtered or restricted based on policy and the last thing I'll say to that is what Docker did for the file system, in other words I can guarantee an absolute path and it's portable everywhere because it's a container, it's like magic Istio does for the network layer. So I tell my app developers don't put an environment variable for the database, don't put an environment variable for your Kafka stream, just say Kafka and the cluster will handle the rest and what this allows us to do is have app developers not focus on annoying configuration things, they can build value and build products and then we'll handle out the cluster layer, the policy layer, how that talks to what because today that might be a Kafka internal of that community cluster, tomorrow it might be a DAS somewhere, we just don't know. This allows us to be able to change on the fly without telling the app developer to go recode something. Other questions? It's so quiet it makes me nervous. Okay, we'll keep going. Let me know the chance for questions. Okay, so what's happened here as we were talking? I'll scroll up a little bit. Essentially there's a bunch of stuff that happened here including my broken minion screen. Alright, so we saw install a bunch of RPMs. We're working on this part. We want this to not be a dependency management system, so one of the things we're working on which is kind of interesting is we're looking at statically compiling code or open source libraries, so I'm sure you all have tracked the news, open SSLs had issues in the past SSHs had issues in the past CentOS and Rail do a good job of back porting their security concerns, but we think we can do it faster. And if we're compiling from source like nightly snapshots and the minute the CVE comes out, we can track that and rebuild that on the fly. We're playing with that right now, but we also have a partnership with Red Hat to do upstream binaries and update them on the fly as well. So we're kind of comparing options here and seeing what works, but I would personally love to get out of the game of managing dependencies here and just do static binaries and just inject them. If you don't know what that means or why that's valuable it's called dependency management. It's a running joke in Linux. It's called dependency management hill. Essentially you have this dependency which needs these 10 things, which need these 10 things, and then you have a whole tree of things. And Git's actually one of the worst defenders we have. Most of that stuff there is for Git. It's just for Git. Git has like 30 dependencies for CentOS. So we can statically compile it and then it's one binary. Which is much more portable as it turns out. So we're playing with that right now. If that makes sense for us, we're also looking for RKE, OpenShift Tasman stuff they do in that space as well with the libOS tree. So there's a lot of options we're playing with, but our hope is to get to the point where your clustering technology isn't told to you by level up. You pick what you've purchased because a lot of people purchase Red Hat, a lot of people purchase Pivotal. They've already paid for these things. They're going to use them. You use what you need to do. We'll just solve the very unique government problem of how do we incrementally deploy to air-gapped environments. That's really what we're trying to solve. The rest of it, how you deploy your cluster, how you create it isn't as big of a concern to us. We think there's a lot of good tools in you that already. So it did the dependencies, installed them, a bunch of stuff for Git, all those pearl things for Git. And then it creates the Docker registries. And again, this is kind of strange, but it literally creates two Docker registries pointing to the same source, which is inversion-controlled. And this allows us to do the incremental snapshotting. We're playing with other options besides Git. Git's not the best at speed because it has to do a binary check, which means it's actually hashing those values, which is great to guarantee it really is what you said it is. Not great because it's kind of slow. So we're working on that right now experiment with other options, but for now it's Git because it works well. So we're doing all these images, re-tagging them, bunch of stuff. Let's see, go back up here. Okay, after I did all this tagging stuff, then it's going to go through and process these manifests. Let's see if I can show you something interesting here besides green text. Okay, so what should be done now? I'm going to just jump in here real quick. And now I'm going to go from this machine into the virtual machine, which is running this cluster on the configuration. So we want to check it out now. And I'm actually awful at typing. It's just absurd. So I'm going to aliases. Okay. So nothing can see all this craziness here, but the few minutes ago before we started this talk, this was an empty CentOS box, nothing run on it, no configuration. It was just Linux, and that's it. We could have gone back further, but it would have taken longer. Well, we have deadlines. So during that time when we were talking, we used those Istio system things and deployed Kubernetes, deployed Metal LB, which is a load balancer we use here because we're on offline environment. On C2S we use ELB, which is Amazon's load balancer because it's available. And then CCS and Space Camp Kafka, Space Camp Gravitaw, Space Camp Sentry, those are apps running, but the important part to the security question earlier is how do we do security? And this right here is how we do it. That two of two is very important and it's very easily. That means that for that one declared pod, there's actually two different images, two different Docker images running side-by-side. They're sharing a file system, they're sharing network IO, and the sidecar, which is Istio is hijacking all the traffic from that other pod, intercepting it and enforcing policies on it and encrypting it for us. So if I look here, all right, so you see here, 161010 is our load balancer, so if I curl this, I'll start to see traffic on there. Unfortunately, all these apps require classified data to be interesting, so I can't really show you much because they're just dumb interfaces. I was really hoping, hoping to get our VR one done and pushed up in time, but I didn't get my keyboard in time, unfortunately. We have one called Space Cockpit. It's done by a company called Saber Astro, which is based in Boulder and Australia, they're split, which is interesting. They're building a, it's a VR and a 3D WebGL interface for space situational awareness. It's very interesting, you can actually see all the satellites orbiting live in a 3D space, either on a VR or just on the WebGL desktop, kind of think computer game. It's written actually with Unity 3D, which is a gaming engine, and it's written by game developers. So when I talk to them, I don't know gamers, I don't really understand what they're talking about, and they'll talk to me about the multiplayer setup and they'll talk in terms of how a game we're talking about players and winning things, and I don't get it at all, guys. But it seems cool. It looks really impressive. Unfortunately, I couldn't deploy it in time because mistakes are made. All right, so all that's interesting here is this is just community stuff. It's nothing super fancy. It's just communities with this deal. What's interesting to me though is that when we took from nothing, we deployed everything that we're calling to deploy on Jwix. This is the same declarative version as Jwix, and now it's just running. If I wanted to change something, if I thought that there was something I wanted to tweak or I could go and apply those, but let's say I completely changed the application, I could also just re-declare it, and so by doing camp start again, what's going to happen? It's going to destroy everything. The entire cluster is gone, the machine is gone. It's going to literally destroy it. It's going to recreate it from scratch. It's going to rebuild it again and do the whole thing all over again. If I'm doing this a bunch during a day, I might turn off the prepare step because I know I just repaired it myself so I could speed it up, but the point being it'll just start over from scratch and we'll do this dozens of times a day. Most of it's automated. We don't usually do it manually, but the point being we're able to iterate over and over and find the right vein of solution so we're able to codify that, to code that in and to predictably reproduce that over and over and over because it's not just something some guy clicked on a screen and put a checklist for. It's not some spreadsheet that says here's how you solve that. It's declared, it's committed, it's tracked in code so there's no variance or deviation there. Additional questions? Space camp, recommend either Istio or anything? Yes, sir. You mentioned you were using Istio to manage security across the entire deployed environment. Does that mean that there's no security configuration on any of the pods that are in there? Like I noticed the Kafka when it was in there so you're not doing TLS or any actables or anything in the Kafka environment? That's right, yeah. And we've gone back and forth on what the right answer is there. So I think that where we're at today we're leaning towards no configuration at all for things like Kafka. There actually is using a password. It's space camp. You all have the password now. But we don't use it, right? It doesn't matter because what's actually authenticating is the TLS tokens on both sides. Istio issues them through Citadel and then that's how they actually negotiate. That extra password is just like a speed bump you have to do because Kafka doesn't know how to handle unauthenticated properly so we have to have some authentication but our goal right now is to try that with Istio but that's not certain. There's still some thorns there with security we have to work through explaining to them that by the way an X99 server is far more secure than a 20 character password that's it's just not quite got to all security folks yet. Our environments we have a platform and an infrastructure ATO right now official ATO and we probably I think we're at continuous ATO now we had a pen test a couple weeks ago and the results because they were hitting Istio they found nothing. Both teams found nothing. In fact they told us darkwarp solutions told us that we were better than they've seen us in the commercial space and security. Not because we're actually good but because Istio and Kubernetes is up for us and we locked everything down. We also do hardened containers though so working with the desoc which is the Air Force's vision to do hardened containers we're building out really really restricted containers we whitelist binaries which means that you don't get all the free tools inside a pod to do bad things you take them away from you. You can't even write to the file system so we gave them access to the cluster even and they still couldn't break in and get across because with MTLS enforcing even if I'm inside your pod even if I'm a bad guy and I have physical root access to your pod I still can't do anything because it's a zero trust network. So regardless of how hard you try and I did because I'm also our insider threat at space camp we tried we couldn't break out of the pods I could break out of a regular alpine which is the secure docker image pod on J-Wix in about 10 seconds. I couldn't touch our pods. Now there could our pen testers so we think it's a pretty good solution but as far as like final answer on that I don't know yet any questions anything else? You got a super board I don't know I don't do these conferences very often I don't know how this... I'm sorry I said super nerdy I know this might be not everyone's thing alright so just to wrap it up space camp is delivering right now for space I think Kessler runs space. Level up's job is to be the factory of the service incubator if you will for all the factories down the road and try to codify all these practices using Kubernetes and Istio there are options to not use Istio as well because Istio is hard but the requirement is essentially do what Istio does which as it turns out is actually harder. When it comes to where we're going to head in the next two to three years we still don't know we don't have all the answers here and we always tell everyone at space camp we don't have all the answers but we try to find smart partners to help us build that future version of what this should be so we tell every person we hire and bring in please tell us where we're stupid tell us what we're doing wrong we want to know things you see today the red flag so we can make it right because the last thing you want at space camp or level up with these organizations is to assume that we have all figured out because we just don't this is a moving target other questions thank you for your time