 Okay, so oh this guy right here is not here. Mike is going to be my My James Coppock today. You fall back a little shorter little balder. Hey So you are oh, yeah, and I'm sure I'll mess most of everybody in here knows me but I am the operations for Boeing and Mike is our Altura's Helper I came in as part of a pivotal project to help Provision PCF and operationalize it So standing in for James go ahead Okay, so if you come to Boeing and you want to be a part of our team You get to pair in these awesome pairing stations best pairing station in the world And this is We had to work really closely with our security team and EJ was always telling us Ah, you can't use about to you and He was a security focal. Yeah, yeah, yeah, security and So he was asking about all the controls and multi-factor and how are you gonna get all this done and we ended up pairing Like this a lot DJ to get her Obviously with Boeing being a government supplier and being and commercial You know aerospace and and satellites and so on security and compliance is a huge concern Obviously, so we spent quite a bit of time working hand-in-hand with security Which is huge when you're provisioning the product and trying to get the platform And into the enterprise right to work closely with enterprise or security and compliance It's key. So we obviously he had some concerns a boon-two was not a standard within Boeing Right CentOS and other and unixes, right? Right security controls obviously they have some if can we go back a slide? Boeing has some standardized security controls on Unix systems They they minimize and harden them and then they have their own security controls that I like to put on top of that Right multi-factor authentication, of course is key with them with them boy They're trying to go passwordless or they are they are passwordless in many places Right and direct access of course goes along with the whole security Concern even though the hosts are exposed for a number of reasons on the internet The you still cannot access them directly So we'll talk about how we kind of addressed some of these concerns Thank you Mike, right and this is how we did it we did add some add-ons to our runtime config and For each security finding we added a section to the runtime right and so what we see These Boeing's concerns are not uncommon. We see this throughout the industry, right? A lot of customers will naturally to go down the head. How do I patch the OS? How do I? Harden it minimize it just like I would with all my other deployed systems So you're the natural tendency is to try to create your own stem cell or your own OS image, which is Highly advised against it's it's a bad practice The way to do that is to create what are called Bosch add-ons Which are Bosch releases which can go on all VMs or selectively go on subsets of VMs For example, we can say that Windows specific code only goes on to Windows Virtual machines for example, so we we addressed a lot of these security concerns by creating a number of of Bosch add-ons To address Boeing's unique security requirements Such as Obviously most places have legal texts that they want to put on hosts when you if you were to access a host via Like an SSH session present a legal banner, you know all activities monitored subject to whatever You know D6 DCI X is a Boeing specific set of security controls that we ported From existing Unix systems over to to Linux systems Likewise, there's a Windows based Version of that as well We did a custom add-on there for Boeing Radically different than the Linux version Antivirus It's common thing on Windows not so common on Linux the support solution Typically out of Pivotal and others is to use clam AV Which is real popular in the ecosystem however clam AV Suffers from the lack of a centralized console since so centralized management and reporting is what it lacks So we we created a custom Bosch release for McAfee antivirus or V cell By our scan enterprise Linux, I believe Which was it's all challenges for sure and then this one is actually this one's my favorite IP tables release here This is It's not I don't believe we open sourced it. We should consider it though. No this this allows us to Address the unique concerns of of access at the network level So the host being exposed or maybe getting ahead of myself the host being exposed on the network Presented a problem. They're not in on an unrouted network here like is best practice But instead we we put host-based firewalls on all the systems to say don't allow any connections From anywhere outside of yourself So the the PCF network blocks all the hosts within there can can talk to each other But other than that nothing can talk to it except for the load balancers So obviously you want traffic to come in from the load balancers to your go routers to the routing tier We allowed that obviously and then we allowed traffic from the management hosts the jump boxes effectively creating You know an isolated network without the isolated network and we had to do that because we didn't have NSX or any Software-based software to find network there at the time We're still using it today go for it and then as far as windows we worked very closely with our security team we Paired with them to you. We had to get the McAfee client Working in our new environment, right and that's where also Mike helped out Yeah, unfortunately on a v-sphere environment, right you have to create your own stem cells Which kind of goes against what I was saying about creating your own And you know windows image or your own OS image But it just is what it is on v-sphere if you were to go out on AWS or Azure anywhere else you get those The stem cells for free. You don't have to go through that the task of creating them yourself But fortunately what what did work out nicely is that those are the bowing stem cell that we used Already had McAfee baked in right, but it did not play nicely with cloud foundry In particular scanning the staging area or the the droplet area the droplet cache on the windows stem cells prove problematic, so there was a bit quite a bit of Back and forth there just debugging and working actually closely with McAfee to figure out the problems Yeah So in the end we ended up just having to exclude a bunch of folders var v cap This and that and whatever to actually get it to work Which you could argue probably reduces the the effectiveness obviously The antivirus and make has some issues. That's another story Yeah, so that was the windows side on the Linux side. However We we created a boss release to pull down the the McAfee antivirus bits and do an install Remember we weren't allowed to use a bunch you and we had to get that pulled into bowing. Oh, yeah Yes, we had to work closely with security and compliance to get a boon to push through and as a matter of fact the a lot of the Concerns that were there around the hardening of the OS minimizing it were not only addressed by the stem cell creation process from the community actually went above and beyond and and this is what we see it a Lot of customers actually is that they say they look at the stem cell hardening repo in the documentation They say wow this is even more than what we thought like we're gonna start incorporating some of this Community goodness and to our own custom builds, which is pretty great. Yeah and then in terms of the Cheryl don't let me take all your what don't be steal all your thunder here. Oh, but I'm reading so I would say though on the on the Linux side the the McAfee antivirus is challenging to say the least We found that You can do the install and you can pull down the initial Definitions from an enterprise policy server an EPO server, but you cannot reboot the VM so If there are any issues, and you know if you do like a Bosch SSH to get in there And you think I'll just reboot the VM no harm no foul, right? It's clustered or you know ha It we found that it just doesn't come back McAfee for whatever reason the version of McAfee that's used there McAfee antivirus It's completely hangs the VM. So I was a hard lesson learned actually. I thought I was still in the port I'm sorry. I thought I was still in the port Before sql. Oh, there's that as well. Yeah. Well, so to finish out this thing though We end up having to say recreate Bosch recreate the VM Just if you're having problems, and there's something crazy wrong with it just You know, it's a reproducible environment So and actually we do repaves every every week there Which security loves anyway everybody loves it and the cool thing about repaves that often is you effectively test Tested your ability to recreate the environment for whatever reason a disaster or whatever you say So he says what what's your level of confidence that you can recreate the environment? Well, we did it at least 52 times last year We can do it again, right? Works nicely and then to Cheryl's point another issue that we ran into with McAfee unfortunately and this is actually I've heard of this with another product as well is stealing ports so when McAfee would come up it spins up its own web server and would steal port 80 I wouldn't say 80 80 or something like that, which was also hard-coded in one of the Bosch releases for another product I think it was a MySQL something or another so we ended up having to and it turns out the McAfee port was not used within Boeing So what we did was we in the installer we put in a time out We put in a delay delay a timer in there for like 10 minutes or something like that To give the other processes on the box time to come up and claim that port So then when McAfee tried to go claim it it was a port in use and it failed It was a is a hack, but it worked We learned from that one on the next is upgrade to you. Yeah So several challenges around McAfee aren't in a virus, but it is what it is if it's required You know in the enterprise then so be it, right? Sorry, we're reading through because My other half was really the one that was doing all these. Yeah, he couldn't be here very great But so what do we added security controls? Yeah, so this is I'm sorry go ahead, please same bracket speak up to Yeah, so the custom security controls on Windows Because PCF is a cloud foundry. Sorry or Bosch created VMs. It's a it's a dynamic very dynamic environment It kind of changes things in terms of a lot a lot of companies want to spin up a VM have it register with a system Like a system of record. Here's my IP address the running on it and so on But in a Bosch controlled environment We don't really have that luxury right so very dynamic environment And that kind of the challenges with that flowed through to this this all the security component as well It's custom security controls on Windows So rather than having a fairly static VM With the security controls on it that's known it's here It's run these that runs these apps and it can check in frequently for run its scans and report what we had to do is instead Very simply create a an API client to say instead of having the security server reach out to me and connect to me I'm going to reach out to it and that was actually a very simple thing to do in PowerShell Simple enough where I'm not a PowerShell developer and I was able to whip it out and nothing flat Very straightforward actually and we just did that as a as a Bosch add-on Simple PowerShell to make an API call with its Arrest call with its IP address and a few other things and then exit zero at the end of the PowerShell And it works like a champ and as part of that we also in PowerShell command super cool command let to create scheduled tasks So then we we have it do its one thing one time and then schedule a security run every day To initiate that API call that rest call So it was a surprisingly Nice and easy experience for a Unix guy to do this on Windows Yeah Really, I think the key lesson for me out of that or thing I learned was that That doing all this stuff through PowerShell is actually super easy And the .NET frameworks are fantastic and really Creating a Windows Bosch add-on versus a Linux Bosch add-on. There's really not a whole lot of difference. It's just it's just the script language honestly Yeah, yeah, and so this we touched on earlier. This is again that Boeing specific the security controls We just ported it from from Unix to Linux which is super easy You know that the thing that was interesting though we we took some extra We took extra care with things in VAR vCAP we remount some file systems and Was differing permissions and so on to address some unique Boeing concerns And of course anytime you're mucking around in VAR vCAP You end up breaking things if you're changing it at the OS level so we went through several iterations of I'm trying to figure out why things broke You know in particular there is An SUID executable under VAR vCAP that has to run its console It's provides the service discovery and it turns out if you remount VAR vCAP without SUID say don't allow any SUID or root Executables under that path you break service resolution name resolution and you break the whole platform so That one was surprisingly fairly easy to find though some fun troubleshooting. Yeah Let's see This is We already talked about this. This is that super easy PowerShell script It makes a rest call and also schedules the task Again, it was just a matter of learning PowerShell versus bash or something like that piece of cake Yeah, this is all stuff. Yeah, did I go backwards? You did I think yeah You're welcome. Yeah, we need to reiterate that Host-based firewalls. Yeah, and so this one this is again We talked about the host-based firewalls which is actually even if you have even if your network is unrouted or using a Software-defined network. I think this is still a good idea. I'd love to see this in the stem cells You know just for defense in depth strategy Okay, so you have a firewall at the edge of the network Maybe we got some VM based stuff at the IaaS maybe even in the OS itself. Maybe it should even Try to protect itself defense in depth. I think it's not a bad idea in general And then it also allowed us to do some custom like this custom access control list entry for the for the jump box One of the problems that we had or challenges we had is that Boeing wants multi-factor authentication However multi-factor authentication into ops manager Pivotals GUI on top of Bosch Doesn't exist or did not at the time. I'll think it does even still So the way we addressed that it was you said, okay Well, then we just don't let anybody into ops manager directly, right? So we had tied it into to their enterprise authentication system But then that meant anybody From anywhere it could log into that thing to ops manager. So how we addressed that was we went back to An enterprise supportive image windows with all of its security controls and domain join goodness and all that And we said, okay That's going to be our entry point from an administrative perspective into the platform, right? so we can get to there assuming you're an admin and then and then only from that host Can we then go to ops manager via SSH or via web? And so that kind of really locked it down to address that concern But then a key piece of that too was to be sure to allow it not only from that data center's Windows jump box but from the others because what happens if that Windows jump box goes down for patching or whatever reason? Kind of a fallback plan. You want to be able to get in from somewhere else? Yeah And this is the same thing the interesting thing here, right? You are pointing this out earlier is the get repository So we we for those IP tables rules for the host base firewall rules We store those in get so as an administrator rather than having to modify a Bosch release Which can be challenging. It's not for the faint of heart we stored those rules in get and When the system comes up it pulls them it has a set of base rules kind of basic stuff But any customizations you can pull in from get dynamically And then you have a nice good You know history of who changed what and that's always nice. Yeah Captain Andrew changing stuff on us All right You have anything to add there. No, no, all right, not unless you do anybody All right Yeah, and here's our pipelines We use runtime config on these and actually Andrew is doing Charlotte right now. He's our automator and When we get to the questions and answers might want to ask him a few questions So so this address is how do we actually? Get these bits on the servers, right? And so it's via Bosch release and via what's called a runtime config Obviously, I don't know if anybody's familiar with that but a runtime config simply defines the Bosch releases That you wish to go on all of your VMs instead of having to say okay for Diego cells Put these Bosch releases for go routers put these Bosch releases If you have a common Bosch release that you want to go everywhere you put it in this runtime config It just centralizes that that configuration And so but then of course you have to push that runtime config out you have to deploy it You you push that out to your Bosch Director and so we maintain those or Boeing maintains those those runtime configs and get and pushes them out with a concourse pipeline I think it's that straightforward Concourses our friend. Oh And that's it Any questions so this started out with all four of those people in there Can you see it? Maybe need to make a figure Yeah That's correct. Yep. There was some unique security requirements within Boeing if I can't say much more than that, but Yes, basically yes, we just wanted to limit all ingress traffic so that only the VMs within Cloud Foundry could talk to those talk to themselves typically, you know You'd rely on you would create an isolated network a completely unrouted network And it may be through in a NSX or through true firewalling or just disconnected completely You know at the VMware level We did didn't have that luxury so we created our own solution Which I think actually because of defense in depth. I think it actually could be used elsewhere Yeah Can you repeat that question? So with the we have an SCR and ACP that we have to be compliant with internal document internal documentation. Yeah Yeah, and standards Yeah, which is is a lot to you know, well, I would say hundreds of pages of internal compliant stuff driven by government and other needs right So there was a lot of upfront diagramming and pulling all the parts trying to understand the traffic flows The security team in particular wanted to know every traffic flow between every component And it quickly became apparent that we needed to really lock this thing down Yeah, yeah And a lot hours of work But to make a lot some of the teams happy we had to deal with okay the dynamic nature of this thing Whereas you used to spinning up the server putting all this time into it and the Karen feeding of it And now it's completely dynamic It's gonna come up on a different IP address with some random host and it seemingly random host name You know and so we had to do things like okay negotiate Well, we'll make instead of you contacting us the server will reach out to you We'll issue an API call to you tell you here. I am you know and I belong to the PCF project or whatever Yeah, there's quite a bit of that kind of stuff and then also We also had to do some integration with the team that that inventories the VMs from the VM level So we started we queried vSphere in this case the vSphere API to get a list of the VMs periodically and send that off to the to the system, which doesn't always get Updated correctly because that's you know, and that's one of those things that you're ongoing fixing This is telling us that we have servers VMs that were there last week Today, they're already gone. Yeah, and since since you repaid every week, right? And at one time the purging wasn't happening properly and it said like you had thousands and thousands December was Entered into an ecosystem where they would record what the VM's purpose was and all of its compliance issues and all these things And it's it's a traditional system We've had for around around for a while and we were always worried is it going to be able to keep up with us? And then our VM where guys check some box I forgot what one it is but they check some box and it started to balance the VMs from One host to another while cloud foundry was trying to balance hosts from one to another and so we ended up We got an email that said that we had like 10,000 repays because I was getting an email say hey cool We got 200 repays because there's 200 VMs. It says you have 10,000 of them It's like what and and and that was repaid that means also 10,000 recreates You know, I mean delete and recreates and that means that was 20,000 entries and exits out of that ecosystem And it worked and they didn't even know it. They're like Yeah, yeah Bosch was fighting the other ecosystem the other the VM system it worked, but we were running out of space, right? European hogs. It was pretty cool anyway Multiple how many levels are there in a tiramisu? We're the spongy level No, no it oh John any more questions This is about Cisco ACI the application-centered infrastructure, you know, right from Regular NSX switch mode to ACI mode Where you can actually you know when you build the network you can build the network You can I segregate the network based on in our applications So how can we integrate that kind of network architecture into this You know, I'm not quite sure I haven't used that maybe Sean Do you have any any thoughts there or we could sync up we can maybe get you connected with somebody who might know You have any thoughts Sean And we already have that at the application level with micro segmentation within cloud foundry, right? So but the next level would be to do it at the service level as well Great question There are some Do we have any more questions I will finish passing out some contraband stickers. Thank you Sean