 Well, good afternoon everybody. My name is John Michelson. I am the director of partner integrations for a startup called Aqua security I've been doing network stuff for 20 years this or network security stuff for 20 years this year I've been to a lot of security conferences and one of the things that I just hate about security conferences I'm not naming any names or three letters or anything But I really don't like vendor driven things and I really also really don't like things that make you sit there for 40 minutes And go with I wish I was doing something else. So what I try to do is provide a tiny bit of levity That into account what is a really heavy topic? So let's just kind of move forward from there One of the things that we really really need need and there was a session earlier that wasn't very well attended I really wish more people would have heard this message, but about the community and community helping with the security landscape this See excuse me. I was Forester did a did some research in the last couple of years They pointed us at the fact that there's something like 20 X million developers in the world and in the US alone We've got roughly 4 million security professionals That's a big Delta. Don't you think and when you add in Europe, obviously, I'm from the Midwest, you know, and you guys are You know other areas but When you add in Europe that number is like 28 million 32 million 32 million to something like you know three and a half million Security pros in Europe and Asia. So what we really need to do is Push this message out to the development community and say you really need to help with this the hard part about that is instead of putting metrics on developers from a Operational perspective we need to educate them and and show them how they're doing things wrong and give them tools that make things easier. So Last fall when one aqua we got started we were dealing with Docker and Kubernetes and Docker e and all these things primarily When we were looking for other markets to to attach to Cloud Foundry was a really obvious one because there is it is very community driven. It is very developer centric so One of those things that we really need to try to do is provide those tool sets And I'm going to show you some of ours, but we have some open source ones that you can use too So this is where the fun part comes in You can call this guy Sven or Joe or Bob or Lucy, whoever you want, but say we've got a CF application developer They've got a task in front of them and we all know developers have a myopic view They look on their JIRA board or whatever they're using and they've got one thing to get done in a couple of hours You know every couple of days they've got their very specific tasks The PMs are the ones that are delegating the tasks and know the holistic view right well this guy Or gal he's got work to do. He doesn't care about anything else. He has to meet his goals So, you know, that's the whole idea is go as fast as you can and well develop security people and Developers and I'll draw a little parallel here and in at hard There's a little bit of laziness to everybody that likes programming The idea is that you want to go as fast as you can With as little effort as you can and have that big ROI of like look at this thing. I've built. It's awesome Right and but the idea is most folks start with Google honestly I mean come on think about it. How often do you go and read a manual? I mean I Peruse them and then I look for examples, right? So He's gonna go out here. He's gonna look at Google and he's gonna look for well He's gonna start going around. He's googling around. He's looking for a sweet go library He was told as to use goes he's gonna use go maybe likes Python better and maybe likes no better But the idea is that he's just goofing off for half an hour trying to find something that will help with this job So in the end here, he's talked looking at twerp and then he starts looking at node And there's some other goofy articles, but he arrives at a Specific package or a library or something that looks like it's going to help right so the at the end of the day we end up at This iris thing and they go he's like oh this looks great cool. All right. Well, let me grab that I'm gonna go download it. It's a github, you know, there's branches and forks and we all we're all aware of that But he finds this library and he goes this is great He's pulling it down. He's building stuff Right and as he's building this I'm gonna ask the question. What's wrong with this picture? There's plenty of people in here. I know your eyes are just look at the pictures. What is wrong with this picture really? So what is wrong with this picture is that Google isn't pointing at the original of this product It's not pointing at the original of the package. It's not looking at the original of the library Google is driven this Probably really good developer off the edge and said hey go use this thing well So What can we do? What can we do to prevent this? Well, it's it's all about education really but the problem with this picture is that The bot herders of the world and and these are you know terms you've heard all over the place and we've been saying these things for 10 years There's tens of thousands of hundreds of thousands of computers that have agents on them already and are owned We call it pond, right? But the idea here is this guy is unwittingly becoming part of that ecosystem when he's trying to build a container He's trying to build an app in cloud foundry trying to build an app somewhere So my talks are more generic they're not not specific But the idea here is that he grabbed this library. It's already got something nefarious in it and if we move forward Here's the real important thing this looks totally normal. It's got a nice read me It's probably 30 pages long and we're gonna read half of it, but fast forward This specific project has over a thousand forks and What github doesn't do is? Evaluate any of this stuff unless somebody reports it so what the bad guys are out there doing right now? Is there forking every project they can? Adding malicious stuff to it and just letting us pick it up There isn't this isn't even click-driving. This is just plain old laziness. That's all it is They're relying on us as to as the development community to not pay attention. That's all it is So we really really need to pay attention to this stuff read the notes go back to the road if you think it's good You know, that's all it is. It's pretty simple. There are some tools out there things like helm You know, we keep hearing about with from the the the container registry from VMware It has some a tool called called Claire built into it Maybe some of you are already using Claire Claire as an image scanner. It's an open source thing It's maintained by a few people, but all of the big vendors are using it as part of their stack now And what we actually compete directly with them because we do it better But I'll I'll get into that a little bit later again. I don't want to be too product-pitchy But this is the major source of crypto mining pickups today This was an article on ZD net, you know six months ago You can go out there and read it if you want and look at the validity of all of this But the idea here is that we're doing it to ourselves We're going as fast as we can and we're not paying attention now It started the talk by saying that we need to provide the community tools so that they can do their job without slowing down Right, so that's part of what we do at aquas. We've got image scanners and Assessment tools and all that kind of stuff that lives in your CI and all the way down to the workstation level So as these people are picking things up and using them and having metrics put that put on them They can actually see hey is this okay, but we really got to back up a stack and Look at the look at the the mentality of the community overall So My title has to do with what about app runtime? Compared to container runtime and this after this next slide. This is where we'll diverge But what does CF app runtime do to protect itself? Molly and teams and they're doing a great job. This product is awesome. This is one of the reasons I really really pushed our our Organization to get to join the the CNC fms and to join some of these things and start working on them App runtime does a lot of really great things There's one thing at the bottom. That's confusing. It's in the documentation, but we'll talk about that in a second One of the things it does is you know, it's got read-only base file systems and container speeds to specific layers We can do CPU affinity and bandwidth Hog detection and all this kind of stuff to prevent DDoS on their own platform There's all of these great things you can do, but it's all about containing the platform. It's not about protecting the application It's running So this is not a bash in any way on the platform at all and like I said a couple of times I think the platform is fantastic Except for the network side The network side, I don't even think Istio is gonna fix it because that's more of a service mesh kind of thing And it's more about discovery The the the outbound connectivity to the internet that's something to pay attention to The number one thing I have customers ask me for as we're working on the agent for this that we are developing I need to be able to I've got 400 cloud foundry apps in production Okay, great. I've got it about 70 different Oracle instances that support all of those applications. Yeah, okay, great I can't separate what app gets to go to what database because everything's a layer three inside cloud foundry That's a big problem. I mean the the community understands it and there there's new hooks and different things We just saw something about service fabric the session behind in front of mine really cool Gonna help with some of that stuff, but again It's it's not at that the threat prevention at the network layer. So some of the things that we need to think about are Layer seven type application filtering through in a lot of cases third-party tools because the It does not built into the stack today. Maybe in a couple years. It'll be different But right now that's the biggest single flaw to pay attention to It's easy to control inbound access because there's the go router on the front and everything's name-based, right? And we separate it on the egress not so much. You've got ASG's and they're all just layer three But we did see this morning the roadmap is is you know, it's looking better. So One of the things to pay attention to everything else. It does great So overall yay, you know two thumbs up cloud founder does really good But you got to pay attention to those apps because it's not going to stop you from using a library That's gonna reach out and do something bad Moving forward container runtime. It's a much different story This is not the fault of foundry This is because kubernetes is immature You know, yes, Google will tell you we've been running kubernetes in prod and on GKE for three years great That's cool. They also don't have hardly any security tools baked into it There's all kinds of really wonderful nerd knobs. You got to turn and dial in to get all the stuff working for you So the difference here that same scenario Bob Joe Sven Larry whoever goes out there and Starts googling. What do they find? They don't find libraries They find entire tiers of applications And I actually have a little demo I want to get to so I'm gonna go a little bit faster here But look at this. This is a great demo Encourage anybody that wants to build some tiering out here to do this There's nine different containers with their own operating systems. This is very different from cloud foundry Application runtime you have stem cells you have stem cells, right? That's it When you're talking about the container runtime everything you start up is not an individual process It's got a whole separate OS associated with it We should be using thin OSes like alpine and things like that, but we're not we're pulling down things like centos Centos is good, right? It's great It's free red hat guess what 7.4 the latest one still has a bunch of red vulnerabilities in it So it's just something to pay attention to and I already said this the root only file system the CPU affinity all of these things These have to do with the the platform protecting itself Not the applications and Kubernetes only makes it worse because you've got all kinds of other layers inside there that today Cloud Foundry can't manipulate. I really believe in the future That those things will be tied together The prem project and a bunch of other things that are out there. They look great For the roadmap, but if you're using it right now, I really got to pay attention One thing that Pivotal is doing not to plug Pivotal specifically, but they're putting the most effort into security of the internals of Kubernetes We have a beta product where we have agents that support pks and I'm talking to the other folks of the other distributions and whatnot But they were simply just first with the native Kubernetes stuff. So My view is that if you use NSX you're going to be doing a lot better But no not a lot of folks want to use, you know a full stack from a vendor So that's kind of the condor to conversation you have to have with yourself one of the things that up security with containers is supposed to do for us is have this whole Mentality of we're not going to patch stuff anymore, right? You've got a blue-green scenario in app in the app runtime And In the container runtime you can kind of do the same thing with Kubernetes the difference is is There's little more variables. So one of the things that we're supposed to be doing is Not ever coming back and doing the diligence after everything's done We want to have the diligence done all the way on the left side. That's the idea of shift left security It's probably heard that for the last couple years shift left means employer Developers to do things better up front use tools in your CIs things like Aqua that will do a scan And of the of the code as it's going live and then as it goes live It's you already know it's there you've got an enforcement point on those nodes They'll actually lock these things down. So the idea is that By the time I got to the right, I know what's in the code. It's been scanned. It's been registered We know exactly what's what's in there all the way through So by the time it goes live it actually we understand that an engine X container has the engine X binary Whitelisted and nothing else I can use those containers from CentOS or whoever that have you know bad VI libraries or something Because all the way at the right side. We're using an agent that actually locks down the processes So they're not allowed to do anything else Sorry got ahead of myself. That's the point Right there a microservice should only do one tiny thing If it's engine X if it's Apache if it's sequel of some sort if it's whatever that one binary binary And it's supporting binary should be the only things that are allowed to run Istio and all these other things they're they're really great for providing encryption between containers and between meshes and On-prem and off-prem and all that kind of stuff But they're not going to stop traffic from being generated within those containers to authorize locations Does that make sense? If I've got a payload not even a sidecar if I've got a bad payload inside that container It's just going to pass it along happily it's going to encrypt it like you told it to and it's going to send it Right over where it's supposed to go So the idea is that you got to start all the way over there and make sure it's not doing what it's supposed to Image hygiene is really really really important the public registries and the private registries are being populated with stuff That's just full of nonsense be honest There's all kinds of different Publications out there within the Docker hub That it's just complete garbage It's and there's there's no other way to say it how many of you have heard of Plex Few okay, so Plex is a is a media server. You can run it home essentially it's it's you know It's it's a really good tool. I actually paid for it right because I liked it so much I actually bought the version you pay for Plex does not have an official repository on docker hub They tell you which one to use it's called Plex Plex Inc slash Plex But if you go search docker hub for Plex There's only one other one I would recommend using from Linux server Linux server is a group of it's an open source consortium of sorts That's generating a bunch of the stuff, but they also have about 45 other ones that other people have built and pushed out there They might be nefarious. They might not That's the whole point. You can't just grab everything and use it In a corporate environment, especially I do the stuff at home, you know, make sure everything's clean for my family and all that too, so Really got to get into the idea of approving things up front I'm gonna skip through this one. We had this this is more of a promo for aqua We are part of CNCF and we do have all kinds of great features that plug into Native stuff. So let's dive in it and do a quick demo The first one is something called that we call image drift protection and The idea here is that containers are supposed to be immutable. That's the whole point something's wrong You know, you've probably heard the the cat pets versus cattle mentality, right? If something's wrong, you shoot the cattle You get another one. I have a herd. I don't care. I don't care and feed and patch containers, right? So let's do that the crypto minor one. I actually forgot to start my lab and it takes a little bit So I'm gonna skip to number three and maybe come back to two if I don't run out of time But these are real things that are better actually pretty powerful and and we'll come back to that briefly the Couple things we have on github the open source tools. The first one is called kube bench. It's been out there for I Don't know like a year and a half or something There's a lot of publications that are starting to push these kinds of tools out there and tell people about them what kube bench does is is Does a CIS? analysis of your container ecosystem So if you're running Kubernetes somewhere you can use this to go and hit it up And it'll tell you with the Center for Internet Securities benchmark All the configuration of your specific environment what it looks like how bad it is how good it is Coop hunter is something we launched about six weeks ago. This is a pen test tool for Kubernetes So you grab this one you can run it offline or online if you do it online We'll we have a little sass we built that'll actually beat on your front door for you Otherwise you can grab the open source and run it on prem and check out your actually environment one of the things I talk about Sometimes depending on the on the audience is Is you know the Tesla hack Anybody know what happened with Tesla last year the spring actually Okay, so they had a bunch of crypto mining took over their Kubernetes environments That actually wasn't a hack they left their Kubernetes management console on the internet with the default password We had a whole branch I can go off and talk about on that So things like kube hunter will let you make sure you're not doing silly things So apart from you know, why was that on the internet in the first place? so So what we really feel strongly is that containers present an opportunity to do security better Cloud Foundry in and of itself has a really good wrapper around this stuff As a security professional, I believe Docker in general set our industry back 20 years two years ago. I Really feel like that. I'm I'm talking to folks that have no idea what we're talking about again We spent 20 years getting the industry to understand that you really got to do stuff up front and you know We don't want to wait till your app is done to do a pen test. We want to be part of your process Docker knocked us all back a huge notch and it's it's great. It's we're moving faster, but I Guess so the idea is that If you're scanning things if you're looking for things based on CVE's and scores you're looking for privilege Privilege escalation you're looking for control capabilities within the configs all of this stuff up front, right? So there's tools that'll do that. We're one of them. So let's demo time So the first one is something having to do with that immutability and Let's see And that guy know this one Sorry this one so what I'm going to do is I'm going to exact into a container running environment Okay, so that's just I'm basically going to use kube control to connect up To an existing container This is a place where we have our agent running that I should be really clear about that our Enforcement point we have a scanning solution that ties into our enforcement point the enforcement point is an agent You there you go. Thanks. So this happens to be just a standard tomcat container. I built it using something called the wordpress orders Demo app or something something off of github. It's basically basically has the struts vulnerability baked into it You can grab it off github and go build it and build the demo and show people at your org Hey, look, we should be protecting against this kind of stuff, right? Spot so if I do it LS being Okay, it's ping right no big deal Sorry It's paying we all know what ping is Well, what if I copy ping? Angle so now I can pong all day, right ping and pong That's working. Okay, fine. Well if I have a system like Aqua enabled What we should be doing really is going I'm just going to go to my runtime policy One checkbox is going to do this all I did with scan images so that I allowed on the run in the first place I didn't even look at their policy or anything. I'm just letting them run And then the second thing is this prevent running executable not in original image. This is our drift prevention Bit so if I hit save I and notice I can go to image profiles And there's all this other stuff I can do the container firewall and all of these things that I can do right All I'm doing is one really simple protection So if I go over there and I try to run Pong again Simple great wonderful demo, right? The idea here is that I exact in as root And I'm still being stopped from this the deal is Ping as a binary We have the hash of all this information and we know exactly what it is Pong didn't exist when I started that container therefore. We're not going to allow it to run That's it. It's as simple as that So if we get into one of the more complicated scenarios looks like I got about six minutes Let's turn that back I'm going to put my runtime and audit mode only but then if I look at my images You'll see there's a red and that's because this is one of the struts vulnerable containers that I built So if I were to Exit that and I want to attack it That'll do it So what this is actually going to do? I scripted it so I didn't have to bang it all out publicly But essentially what this is going to do is take that struts vulnerable container. It's firing up another pod That's actually looking for struts vulnerable containers on the Kubernetes cluster and then attacking them So this specific injection attack It's not super complicated what it was within WordPress was it doesn't handle a base 64 Command being thrown at its login prompt Imagine that why should it right it should drop that but it doesn't it crashes and allows you to log in so hey We got to wait for the load balancer to come up because it's that's what we do But that's essentially all it's doing. It's a very very simple attack. It's a very very simple demo But once it's running Here we go Once it's running, and I'll show you this If I go to my orders application This was Sorry lost that one So this is a very very simple application But imagine this is a bank imagine. This is social security numbers Imagine it's some type of remote manipulation, right? So I'm gonna go and create a order and let's say a PCF They're gonna give me a thousand dollars for well, you know, whatever because they are imagine it's some kind of application Right, so now if I come back to this shell, you notice my IP didn't change It's just spawning a whole another segment onto that same namespace so if I Cat my read me because I don't want to type all this in and Just imagine that this is all done programmatically, right, so What I basically did is I just created a script I dumped it onto the root drive and now I'm gonna have to go run it Right, so because of this shell I have I just created this big orders that sh. Okay, so I have full access To the whole that whole container. That's what this reverse shell is doing I Could do anything I want through this window, but the window is a demo, right? So the idea is that you're gonna attack programmatically nobody's ever gonna use this window. I'm just doing this so you can see it So if I come back and I go well, I got a chmod that yep got to do that All right now let's run it. They do have about a minute and a half I think so I'm just gonna run that right it's gonna take a little bit because it's looping there's a bunch of fours in there It's basically just bash, but if I come back to the orders app now What do I see happening? All kinds of them now imagine I was exfiltrating some type of HIPAA or GDPR or whatever what have you right? So this is this is one of the examples and if I go back to our system where this agent lives I Can see the scan Happened I can see let's see go to Sorry container name I'm filtering by orders And I've got all of this stuff that's occurring here. I've got a curl with a post To the client name with all those randomizations that I mentioned right Username 23 amount 17 etc etc. So this is that attack happening. It's just using curl under the covers It's all it's doing repeatedly sending a bunch of curl calls Because I was able to do that. So if I try to run that attack against the protected container Instead We go to three. Oh attack protected this one Assuming I didn't change any settings accidentally like I do sometimes This one will never come online. It's never gonna happen because the agent is actually blocking it from occurring, right? And I was not in protect mode just like I said perfect. Oh wait, that was too. Yeah, that was too So let's hit that again hit refresh again, you can see out the events and do there we go Nope, those are my pongs So after my last demo, I didn't change my settings back So everything was wide open and I just attacked my one with the profile on it accidentally so What's supposed to happen here is that you will see a root performed etc. Etc. And that's that's the whole idea These profiles are very easy to generate We provide a profiling tool and Everything you can do you can do all of this programmatically in this example this specific orders container is allowed to run you name their name tty Java and dash and You can get more complicated and use the shea hashes of every single every single little bit if you want You can take a second profile and apply it You can do all different kinds of things with limits and volumes, you know mount volumes Specifically and make them read only and share them. There's all this kind of all this other kind of stuff you can do But essentially that is the demo and Am I doing a time? I'm done. I'm over. Okay, great No worries Thanks for coming I'll be around all day if you need to talk to somebody about the Aqua stuff But if you take anything away from this talk We're not the only vendor out there. I think we're the best one. We're not the only vendor out there So make sure you are looking for tools and using them because the platforms themselves. They're not about protecting your applications They're a product protecting and making sure your apps are running. They're not going to check and see if the apps are doing stupid stuff So thanks. Have a great day