 All right, so hi everybody. People still rolling in. Come on. Talking about air gaps, it'll be fun. All right, we'll get started. So hello everybody. My name is Mike Ferris from Stark and Wayne. And I'm Vincent White from Agile Defense. I am with Agile Defense. I just like Stark and Wayne so much. I was supporting him today. He likes the shirts. He's supporting the shirts. Just want to make sure you know Agile Defense is in the house. All right, and so today we're going to be talking about air gapping. And so Vince and I have actually worked on a project together recently where our two companies came together to work in an air-gapped environment. And so we're going to be talking about some of the pitfalls we faced with installing PCF or Cloud Foundry in general, or really any PAS in general, in that kind of environment and still being able to provide the experience that users expect from Cloud Foundry in that air-gapped environment. And if you don't know what air-gapped environment is, we'll talk about that in a second. So next slide. Okay, so what is an air gap? It is the 21st century moat. If you're in an air-gapped network, you are in that castle and you can't, nobody can get into the castle. Nobody can get out. I don't know if there's a drawbridge here, but no bridges in an air-gapped environment can't get in, can't get out to the internet. And so a way more boring diagram with zero castles is right here where you see the internet at the top and when you air-gap the environment. Next slide. Boom, no internet. Everybody clear? Internet? No. Pop quiz. Okay, so what does organization use air-gaps? Okay, so basically it is to protect us from internal and external hackers or, sorry about that. Okay, so basically yeah, it's basically protecting the environment from internal or external hackers. Right, where obviously external hackers are people trying to get in, trying to compromise your system, trying to take advantage of ports that are open to the internet that you might not know are open to the internet or you might not know have a vulnerability where they can get in, get root access to start doing funky stuff and obviously the best way to make sure nobody gets in is make sure nobody has physical access in the first place. And then internal, by internal hackers we're just talking about people in your organization, rogue employees that might want to install software that has not been deemed okay by the organization and just rogue employees in general that might compromise your network by thinking they're smarter than everyone and installing whatever they want. And so obviously there's a lot of benefits, the benefits that we just talked about, but with any benefits, obviously there come cons. And the big cons with installing a Paz especially, like Cloud Foundry or Kubernetes in one of these environments, is that one, all those binaries that you use to install Kubernetes and all the documentation, everything is tailored towards an environment. Typically all the detail in the documentation is about an environment where you have access to the internet, all the installation patterns are going to go and grab things from the internet, pull them down. And so when you don't have access to the internet, you are going to hit some complications that we're going to talk about that will ultimately slow you down. That's the big problem that we want to show you in this presentation, how to solve, how to not get slowed down too much by an air gaps environment. And so for developers, obviously there's implications on every engineer using every software engineer within an organization. So for developers, the implications are going to be that their apps obviously cannot use the internet. So don't write your apps to use the internet. Pretty simple. And then also for building your apps, a lot of, especially like in Cloud Foundry for example, you use build packs and a lot of those build packs, they actually go out and they'll download, if it's the Java build pack, it'll download jars that you need, a lot of things that you need for the build process will come from the internet. So in that case, you want to use things like offline build packs. Basically, like I said with the running of the apps, don't make your app use the internet with the building of the app, don't make the building of the app use the internet. But in this presentation, we're going to be focusing on the operators. So let's go to that. And we might be biased as operators, but here's how complicated air gapping makes the developers' lives. And then here's how complicated it makes the operators' lives. So with those complications, right, so it comes installation and complication for us as whatever role that you may be playing in on the platform team, from the DevOps team to engineers, because one, you're talking about now, all right, what, like, like we stated earlier, what installations can we download that needs all of the supporting detail or supporting softwares or installations to map that out and then from an engineer level and then take that over to the air gap environment. So, like, that within itself is a complication because as we know, there's a lot of moving pieces to it. Upgrades. So after you have a platform installed and upgrade, I mean, installed, you at some point in time, of course, you want to look at upgrades. So now, whatever role you play, of course, it goes back to understanding what pieces are needed to upgrade, making sure that it falls into guidelines and the protocols that is necessary in whatever environment or situation that you're in. And like Mike stated earlier, no use of the external services so that those external services are GitHub for repositories, Harvard, Docker, any of those type external services. Now you have to engineer, okay, we got to build this on the air gap side to make sure that we can emulate the same process or scenario on the non-air gap side. Right, and we, let me just go back one slide. We just called that out specifically because that obviously complicates your life in when you're doing installation or upgrades, you're pulling down from external services. For example, with PCF, you pull down from PivNet, you'll pull down the Docker containers you need for your concourse pipelines from Docker Hub, and you can't access those. Another reason that might complicate your life is depending on how many hats you wear as an operator or on the infrastructure side of things, your developers who now also can't use those external services might be coming to you to replace, to give them an alternative to use those solutions. So it can kind of have a double whammy effect of increasing your responsibilities as an operator. And so let's run through an example of, so with installation of PCF, obviously Pivotal ships concourse pipelines, and so those will install your PCF on an IaaS and it's going to, if you're not air gapped, those pipelines are going to be grabbing artifacts you need from PivNet, it's going to be grabbing the Docker containers you need from Docker Hub, it's going to be grabbing the pipeline tasks from GitHub. Your life's going to be easy, it's going to be great, except, oh wait, we're air gapped, we can't talk to any of those. So what you instead need to do for installation is you need to have, like we mentioned, internal services to match each one of those, an internal service corresponding to each of the external services that you can no longer talk to. So in this case you'll have an internal blob storage where you'll put the things that would typically be in PivNet, and so that can be anything like Minio, any typically S3 compatible blob storage is typically your go-to. Minio, Ceph if you're on OpenStack, anything will do. And then, so to match Docker Hub, you'll have an internal Docker registry, VMware Harbor is an example of that. And then to match GitHub or whatever external Git repository you were using, you'll instead have your own internal one. Like Vince mentioned, now you have to build out all these services and that is increasing your responsibilities as an operator. And just to add to what is, like I stated earlier, this piece, right, is a part where, okay, now from whatever, ideally the engineer will have to come in and put it just together. Simply because if, to be effective, you want it done the right way as much as possible the first time, because there's a lot of going back and forth. So the engineering piece will definitely be a tremendous help to that. And so obviously you're wondering, you're thinking, okay, this is great. We've got these services that magically contain all the blobs we need. We've got a Docker registry that magically contains the Docker containers we need. This diagram kind of implies that everything's all set to go and we just got to build these, but we actually have to populate them with the things we need. So how do you get the things that are typically in PivNet into your internal blob storage if you're in an air-gapped environment? So the best practice, and you'll see there is an asterisk next to that best, because you might go to your security team and start the sentence to say, hey, we might want to have a D, and they'll say no, you can't do that. So best obviously depends on the environment you're in, on what kind of regulations your security team is imposing on you. And so this is best as in it's going to make the operators' lives easier, but it might just not be feasible within your organization. So it really depends on the organization. But anyway, so the best practice for making your life easiest as an operator is to have a DMZ, a demilitarized zone, which basically ends up being one VM or a couple of VMs that are whitelisted on the network and can talk to the internet but can also talk to the air gap. And so what you would do is run a concourse pipeline or whatever your CEI is and specifically have a YAML file defining your pipeline that calls out each resource that you need to grab from where it needs to be grabbed from, which version you need to grab and then where you need to put it into your internal services. And this makes your life really easier than the alternative, which we'll get to, because now you have, in addition to that pipeline being the thing that actually does all of the grabbing for you, it also ends up being documentation of which resources, which versions of which resources are in your system. And so you've just got a nice list that you can then modify and run later if you need to get a new version. And so that's the best practice. Worst practice. So basically it is we're walking from one building to go to another building that has internet access to burn our binaries, artifacts, build packs, whatever those are to a DVD. So as we're walking, we're making sure that we have a case of CDs or DVDs in our air. And we're going over to the other area to burn what we need. So in adding to that you find out, so you go there, you download whatever binary build pack you have and you put it on a disk. You get back and you're ready to go. You know you got those binaries on that disk. You saw it before you left. They're all there. And you get over and you start pushing those build packs or uploading those binaries and you find out that is corrupt. Completely hypothetical. This didn't happen to us hundreds of times. So you find out they're corrupt. So you're trying to figure out now, and we literally sat and did this big, like deep diving, we know the files are there. Like maybe something is wrong in the file. So we're changing, you know, different things in that Emma file or in that build pack. And it's just like, oh no, okay. So we found out that it's corrupt. So now we are marching back across the street to put that same file, that same binary build pack on the disk in the DVD to make sure that it's there. And then we're coming back and trying to basically it's a rinse and repeat type thing, right, till we get the right one. So ideally what you would want to have or before. Yeah. Okay. Yeah, if you get one thing out of this entire presentation, this would be it. Yeah. Just in situate if you find yourself in a situation like this where you have to be going manual somewhat manually or completely manually going and downloading all the resources that you're going to need for your system. Don't wing it. Plan ahead and think about which binaries you're going to need because it's going to make your life a lot easier and it's going to make you way more effective as an operations team. And if you are winging it, like I said, your developers are going to notice eventually and because you're going to be moving very slowly and your team is kind of going to be a dumpster fire. So plan ahead about which binaries you need. Make a list of all them beforehand. It's an easy, it's a harder, harder said than done, but it will definitely make your life easier. That's something that me and Vince learned on the job. Yeah. I think the other piece to add too is just making sure like you have those necessary parties in there to plan ahead like developers. If you're not, you know, developers on the developer side is making sure, hey, you know, can you sit with us and we do our whiteboard on pieces that you guys need or have used in the past. So we can factor all of that in because now what you're doing is making sure that you have the supporting tiles and those sort of things to make sure that those developers are able to be impactful in what they're doing as well. So as every operations person knows, once you've installed the platform, your job's done. You're good. Good to go. Mission accomplished. And so obviously that's not true. You still need to maintain the platform. You still need to upgrade the platform and you want to be doing this in a regular cadence and providing functionality that developers are requesting and being able to do that in a timely manner. And so in order to do that properly, it's going to go back to exactly what I just said. If in the best case you've got your pipeline defined in a YAML file where you're just changing which resources you need in the YAML file and you run the pipeline and grabs them down and you're good to go and you've still got the YAML file defining nicely every single resource that you're pulling into the system. And then in the worst case, where you've got to walk across the parking lot with the DVD and come back, stand in line maybe, planning is going to be everything in having a good regular upgrade cadence and velocity and being able to provide requested features to your developers. Yeah. And so let's talk about, Vince and I each have our own respective experiences outside of our experience together on that one contract. So I've worked in, I've had a lot of experience working in situations where the best practice that we talked about was allowed. And so we would have, like I mentioned, that pipeline that was just defining everything that we needed to pull down. And once we had that pipeline defined and everybody on the team was aware of how that pipeline worked and what you needed to do if you wanted to upgrade and get a new version of a binary or add a new version. And who you needed to talk to in order to make sure that those whitelisted VMs in that pipeline were within the constraints of your security team and you weren't going to get yelled at. Once all that was taken care of up front, the operations team weren't slowed down very much, if at all, by the air gap. And so that's why I say it's the best case if your organization will allow that kind of setup. And so yeah, I've been on a couple of engagements where that kind of setup has been allowed and it's, like I said, it hasn't slowed us down too heavily and it's allowed us to do installations in a timely manner without us looking like a dumpster fire and do upgrades in a timely manner and be able to stay up to date with versions of the platform as it comes out and being able to provide functionality as it's requested by users. And so Vince has been in a different situation a couple of times and so he can talk about that. Yeah, I mean it's somewhat similar, just some more, there's added pieces like for us in the air gap environment you have to have from top down ATOs, right? You have to have, there's a number of things that you have to have. I won't go through that list because it's long. But there's a lot of compliance that you have to ensure that you're staying inside of. So planning is, like you said, it's definitely like, that's like number one. Like if you, the planning piece can make that our installation in an air gap environment much smoother. And when you have the right parties and people that's a part of that. For us in our environment we had different people that we reached out to that help in different perspectives to help make our planning a little bit easier. And I'm talking about from the installation piece, from the security piece, from the compliance piece, from vulnerabilities, from all of those aspects. Like we had different people we could reach out to to make sure that we were inside of that arena to get past the hurdles when we started on our own. So what Mike was talking about earlier is, you know, going back and forth. And I just paint like a little small picture of, you know, points that are not high is, you know, you start out with this offline build pack and you find out there's other pieces that you need of that offline build pack to make that build pack work. So if you're talking with somebody like Mike that knows the outside pieces of what makes that build pack work. It's like, okay, hey, let me jot this down. We'll put a list together so that when we walk over we can have as much as we can get on those DVDs or whatever and bring it back over and be successful in just pulling that offline or pushing that offline build pack so that we can be overheld. Yeah, those trips start to add up if you're winging it and not planning ahead and coming back every time and saying, okay, now we need something else. In that case you might want to hire an intern to do it for you. And just a side note about each of our respective experiences. My experience is where we are allowed to use that best practice that I talk about with having a pipeline do this for you and basically self-documenting all the resources that you've pulled in via YAML file that also doubles as your pipeline that does it for you. Those have tended to be in financial institutions and whereas Vince's experience with completely not being able to do that and not even being able to finish the sentence asking for permission to do that. Those have tended to be in the defense world of side of things. So if you are a consultant who might be doing defense type stuff in the future or you are in the defense sector and you're thinking about doing platforms, I would definitely recommend heavily investing yourself in your team in planning ahead with which binaries you're going to need and knowing that ahead of time. Obviously it depends on the platform that you're installing, the flavor of the platform, everything to know what those are. But yeah, that would be my one big takeaway from this presentation if there's any planning ahead and not trying to wing it. Like we've said, Vince and I did that at first and it didn't go well. We had to pivot and think of a better way to do it. The entire installation really, we were pivoting with our team. We were pivoting through the entire situation. Making a better process really made all the difference towards our velocity as an operations team and being able to actually provide functionality to developers. And to add what Mike was saying about if you're a consultant, it's just definitely sitting down with the devout team or whatever team just to understand the restraints up front that they are dealing with. So that can be a factor factored into your solution or installation plan as well. Yeah, so you know which things to download. Planning ahead. Because the software is not going to do it for you in this case. So yeah, that is our presentation on air gaps and installing platforms. Anybody have any questions? We'd be happy to answer them. Yes, sir. Yeah, so that would be, obviously it depends on the solution that you're using that's actually doing the looking. But for example in concourse, it's just a matter of changing the YAML file to changing the URL in the YAML file. Maybe some of the parameters if it's a completely different kind of service, how the credentials are being passed and what not. But I've never used Jenkins or anything like that or any most other CI things. But I can't, I imagine it's really similar to changing the URL of where that's pointing to. Most of them seem to be pretty plug and play at this point where you can just interact with different services by just changing the URL. Maybe changing some of the parameters. Yeah, exactly. That's exactly what it is. Ideally. Hopefully. Yeah. Yeah. So it depends on the environment obviously. Some environments, the proxies typically come into play in that DMZ type of environment. Where you do have access to the internet, but it needs to go through something that might be regulating your traffic, checking HTTP headers, things like that. And typically the biggest problem is that I've run into with proxies in an environment like that is forgetting that the proxy is there and forgetting that you're talking to the proxy. And thinking that you're talking to something else and trying to decode the error messages if you're talking to something else but forgetting that the proxy is there. So does that answer your question? Do you have a more specific like... Yeah, I got you. Okay, yeah. Typically in that DMZ pattern that I talked about, there would be a proxy there. Yeah, correct. Yeah, I think just to add like in the defense world, most of the time if you're sitting in that DMZ, you're pretty much good unless you come across pieces that are hard-coded. If they're not hard-coded, you should be pretty much good as far as like any traffic going in and out or through that DMZ. That's in the defense side, like I said, most of the time. Yeah, good question. So obviously it really depends on the organization. But for... So you're talking for a security team to vet those and say, okay, yeah, you can pull these in. I think the minimum I've seen it take is a week and the most I've seen it take is longer than the engagement I was on, so I don't know. We're one another. We're still waiting. I'll check my email after this and see what's going on. We didn't email you. Yeah, but it can vary. But that's definitely a very, very good point. And that is another thing that can greatly influence the speed at which you can deliver to your developers. And so getting on the same page with your security team as far as we are trying to be an agile team that is delivering functionality iteratively to our developers and we're going to need... This is not going to be a one and done thing where we come to you and say, hey, is this good? We need to come back and say, hey, is this good too? The next version of that or the next version of your Docker image and being able to do that iteratively? Yeah, that's a very good point. Thank you for bringing that up. Yeah, getting on the same page with your security team about the iterative fashion at which you plan to work is very important and can either facilitate or definitely slow down your process as a operations team. To add to that too is getting with the CM because they were configuration manager. They would definitely be overheld because you can ensure that versions of and versions before or after has been vetted and good to go. Yeah, good point. Thank you for bringing that up. Anybody else? Yeah. Yeah, that's a good point or a good question. So you could have... I've seen a lot of organizations have like a lab where you can turn on and off in air gap functionality. So you can pull everything in that you need via the typical way, turn off the environment and make sure that using those binaries that you've just pulled in, everything runs properly. And then you can say, yeah, from there we're good to go. We can figure that we're going to be good to go once we move that into an actual air gap environment. That's where we're going. That's going to be a big help. Agile Defense, they're our lab. And so it's a duplication of what we have in the air gap, what is needed in the air gap, and then Agile Lab is the same thing. It reflects the same. It mirrors the same. So that testing and that break and fix piece will be done in that lab and then we can push it over. Yeah, being able to... having that lab such that you as an operations team can toggle that air gapness, that would be imperative towards the testing of it. Otherwise, you're going to be going back to the same security that's air gaping your actual environment and it's going to end up probably being just as slow. So yeah, good question. Thank you. Any other questions? Comments? Concerns? What's he got? You don't have a scenario where you use the proper license in the air gap. You just throw it into the office or you burn it right away. So you're still automating the whole thing down. In the same manner, you're just being able to get it in one place together. Yeah, that would definitely be an ideal scenario. So in the specific scenario that Vince and I worked in, that would not have been possible because the computers in the place that we were working on, they were Windows machines and if we said, hey, can we install concourse here, they'd be like, what? Or any CI engine, it would have been just a complete no-go right from the start. But yeah, that would definitely be another ideal scenario obviously depending on your organization. Good point. Is that a question? Alright, I think we're good. Thank you for coming everybody.