 And I'll start the recording started now, my name is Mark Morales, and I'll start the presentation here, introduce myself and Eric, and then we'll get on with showing you some slides and then a demonstration and discussion about application security. So I'm a partner solutions architect at sneak, and I'm presently living in the Philadelphia, originally from Chicago, I spent many years writing software including embedded code for for automobiles. Last 1015 years I've been doing the DevOps, really excited to be here today, because I feel that my experience doing a lot of say DevOps pipelines and working in security is going to show in some of the enthusiasm I have for the presentation. I'd like next to introduce Eric, and I'll have him introduce himself and describe what he has. Hi everybody, my name is Eric, I am a software developer for the last gosh 30 years or so. Based in North Texas, I have been a Docker user since gosh back in dot six version days or early early days before we had any orchestrators or anything. I've been around quite a bit and the CI world used to be Jenkins ambassador. I'm currently a Docker captain and just got my CKS certification so I got all three. So yeah, I'm happy to be here. I am involved in the dev relations department at sneak in the cloud native and containers area, which is what we're going to talk a lot about today. Okay, so we'll have an agenda we'll go through some slides and then do some demonstration and then wrap it up with whatever kind of Q&A we have. At this point, Eric, I'll let you start describing a little bit of how we get started here. So first, I want to talk a little bit about DevOps and security and how it plays together DevOps to a lot of us. You know the first thing you hear you might think it's a buzzword right. But really it's not it is to use the can thing it's a culture right it is development and operations and everything in between working together to get our software in front of our customers. And that's overly simplified but you know when we think about DevOps we often think about pipelines and getting that you know the software delivery life cycle getting our code from you know ideation really all the way to production in front of our customers. And when this is a simplified view of a pipeline obviously the ones you see at your musicians are probably more flex and more stages to them and whatnot but they're generally fall into this kind of a thing. So I've added our little patch logo our dog to a couple of areas in the pipeline that we've historically seen security applied table stakes is production, we've always had security and production right you've got. Since the dawn of time we've had firewalls we've had access control tripwire alerting intrusion detection you name all those things have always been baked into our production systems and our sysops and security teams have done great job you know, making sure that we're not heartened and we do our best to not to anticipate, you know, at the perimeter what people can do as CD started to become a term, as you know, we, we saw security start to get applied in those pipelines as we started to call them pipelines honestly, but at the late stage, far to the right, if you will, and that was often because the tools were either not either it wasn't automated it was just a manual gating stage where your security teams would do audits of the application deliverables and make sure that everything looked good, or they might have been automated tools but the historically those tools are really slow need to be fair I mean, they can take overnight sometimes to run if you have a big enough application are complicated enough one. They also might have very complex licensing schemes to where only a couple of your CI agents can run them or they only they could only run in a certain environment in the enterprise. And I think that caused these stages to be far to the right usually after functional testing was done after performance and load testing and things like that happened. And if heaven forbid, something came up. They pull the cord stopped the line and you'd all have to huddle around and figure out, what did we do, how did we add an additional vulnerability or whatever the audit came up with, and kind of just like any late stage bug find you've got to now bisect the data to figure out what it is you did that and what do we need to do to remediate this to make sure we go out still on time, or do we need to just put it on the backlog and fix it later, which is never a good solution because now you're going with known vulnerabilities, for instance. So as things matured, we moved security farther left down the pipeline and we added things to the repositories and registries, especially around containers which is what I'm the kind of my my area of expertise and the registries were the first place you really saw this is where you when you pushed your image into a registry, it would do a stand for you and come up back basically the spreadsheet a table of CVE say you've got these problems have fun. That's good I mean it's better than not having it and it is at least hurdle you're in the stage so you know as soon as you produced an image that can be deployed that it has XYZ vulnerability. And also we saw this in the repository, this you know the continuous delivery, or the pipeline part of the repository, as the image was being built you could actually do it right there if you have a tool that could be called from your stage. So you can say hey build the image and right before I push it I want you to look at this system, and hopefully as fast enough to come back with a decent amount of time, give you the same kind of information. Even better, we move towards our branches and pull requests. So, every time if you're on a get type of workflow often you have a pull request and you will have automated reviews of some time whether they be smoke tests or code style tech checking you know all sorts of things like that well this kind of a security product if it's if it's performing enough can be applied at that point to see you aren't allowed to merge to main or whatever your development branching strategy is until you've resolved these problems and you can do some sort of ratcheting techniques you do any other tests to say if vulnerabilities go up or if severity is increased or whatever you can restrict that. And that's, that's pretty mature if you've gotten to that point, but the Holy Grail of this is to get this kind of analysis and scanning all the way into the the day to day workflow of the developer on their workstation. So all the way in that left box here the coding as a developer I'm coding things up I'm often building my application and usually if you're in a containerized world you're putting it into a container and running it locally so that you're running it in some semblance of what you'll be doing in production. Well, why not just scan the image right there. If you've got tools that are fast enough to do that, and tools that are gives you proactive information that can help you solve any issues that they come up with by giving you advice on, hey, this is broken in this version it's fixed in this one. So you just added this, you are now in control it's it's right at the top of your mind. I know what I added and I know I just added this extra vulnerability. I can either take care of it right now or I can pull my lead over to my desk or onto the zoom or whatever we're doing at the time. And if I need security is involved and I can pull them in now proactively and say hey I'm seeing this and we need some help understanding the priority of this and and the impact of this and just to handle it right then and there, and this is all part of CI asked feedback. So this is where you hear the term dev sec ops and yeah there we go. So this isn't really a thing dev sec ops, it's a term people use to make that basically remind you that security needs to be part of your dev ops patterns and world. Just like QA just like deployment just like everything else that we bundle under that dev ops umbrella security needs to be part of the game we need to invite them to our party right. Go ahead and next one. So, specifically what are these challenges we face as developers writing containers nowadays. The first one is the increased scope of responsibility that we're facing so as as a developer as containers started to come into my life. I understood things like operating system packages existed. I knew that, you know, file system permissions and user ID restrictions and things I knew about all those things from a point of view of a consumer of the advice that the dev sec ops security network teams everyone else gave me and I fit into their rules, but now all those things are in my purview because I'm writing my Docker file or my Kubernetes YAML or whatever else that those aspects now are controlled by their this infrastructure as code or infrastructure as configuration as some folks say, and it's up to me now. And while I have those teams available to me to help me, the code is right there in with my application code often. And if I make a change that breaks it I need to know. And historically we have a lack of expertise around that. I'm not a app sec expert necessarily as a developer on a big enterprise application. I'm not a network expert I don't know how firewalls work and configuring computer policy might be about you know something I'm not familiar with. So we're having to learn these things. And on top of all of this, the whole point is velocity right where the agile dev ops. CICD is all about getting things from ideation production as fast and as high quality as possible. Anything we introduced to you as a developer that is going to slow that velocity down is business is not going to like in fact they probably want you to increase that so we want to make sure that anything you're you're looking at you know helps you know keeps the velocity up so let's go to the next slide. So, what is it we developed. So we're pretty familiar with you know the application we have source code, for instance. So we know we need to write secure source code. We have libraries we depend on external dependencies packages modules depending on what line would you write things in. It would be obvious to us we understand that we need to do that, but when it comes to containers we now have things like the Docker file, which as I said before. This is operating system level kind of things in this little screenshot example we've got Python. Okay, I know what Python is but what distribution of Linux is the file system in that container based on what else is coming in with that base image. What are the packages I'm adding what are the, what are the, you know, versions in those all of those kind of questions start piling up, and they're things that we didn't necessarily throw over the fence before, because we, we knew hey I know I need stress energy for whatever my application, but the management and maintenance of that was often not our problem, if you will. So source code is another aspect so you might have terraform or cloud formation or other things right alongside your code that you know handle the wiring together of all the infrastructure under your app that adds some complexity, and the grand idea of mall of course is this massive API that changes three times a year now and it's just, it's, it's complicated, to be honest, and it's enough to get our apps running in this that we might hyperventilate, but keeping it hard on top of all that can drive us crazy sometimes. So, go ahead and go to the next slide Marco. So I'm going to hand over to Marco right now to talk a little bit about some of the ways we can mitigate both container as well as other, you know, other aspects of our security in our apps. Yeah, thank you Eric. So I'll start transitioning here to the demonstration I want to give you a little bit of context right so I have a some bullet points item, we are going to be looking at a repository the vulnerabilities, some fixes and things along the way. Eric and I have enjoyed developing this story because we're using a newer repository to it's a it's a Java application. It's named after our popular line of vulnerable applications called goofs it's Java goof. We will provide links to the repository and our documentation and other things for the folks during this session as a cut and paste, and it's also going to be shown on screen, but I'll go and take you through a journey now, where we're going to use a couple of different tools. And as Eric and I were developing this we said well what do we want to show. Let's show a little bit of everything right, a little bit of everything here is yes we're going to have a Java application it's going to be in maven. It's going to be hosted on Atlassian bit bucket cloud, we're going to have a little bit of Docker some pipelines, we're going to be deployed to AWS ECR, and in the end we'll have an EKS cluster. There's probably some other stuff that I may have missed, but the idea here was to kind of do this in your language, the audience, and way to say like I seen those things I've done those things. So if we happen to use something else like a chaos. I hope you'll see that some of the patterns we have here aren't really specific to a technology. It's just that the language of the parameters are along those things and I'll try to explain those along the way. So from this point I'll stop this part of the presentation, Eric will pick it up later, and I'll move into the demo mode to show you some of the things that have been describing. As always, you know because we've enjoyed developing this demonstration if you have questions pop them into the slide, Eric and I will try to read to the best of our ability, and if we need to answer right away we'll do our best and do that. So what I actually had in the first tab is this is going to be the original source of truth for the repository. It is on GitHub we will give you a link in his Java Goop and we have some nice documentation. What I did is to work with an Atlassian BitBucket to kind of give it like a different feel. So you and your team will have a set of repositories, and I imported this directly from GitHub, and have it in here. Many of you may have access to your own BitBucket and I encourage you because we will be putting out a workshop that has all the steps already there we're almost there we may be ready as early as this Friday. But here we are as developer. I may look at the webpage on the on the one side Eric did mention about the ID. I use Visual Studio Code. It's my preference. I do have the sneak integration that's that's something I do, and I can see some of those details from the ID, but we're not going to cover today. I also portion around with the CLI. We know that some people like typing in commands and seeing the results in JSON format. That's something that we do in support, but we're not going to dive into that too much because we feel the visualness of the demo and explanation is. But I do want you to know that's available to you if you want to just kind of do your own jq and and whatever else you want to do to process the results. So I have my repository I can scan and I can look at it, and I can feel like I am contributing on a team. One of the things I did is I onboarded the sneak integration in a separate time. I've actually put out a couple of videos that show how you do this in a couple of minutes to just plum and sneak and then you get this nice little word it's a sneak on the side, which used to be the word security. Clicking into it is going to help is going to be how we started journey. So I'll click into here, and I'm going to see a summary page of the vulnerabilities for my application. So here's a couple of things. One is I mentioned we have a Java application. It is a nested Maven project. And for those Maven people out there, you'll kind of recognize what's happening. I can see I have a parent palm. And I have some dependent pond in here, and we kind of speak those. Those are what we're going to call as projects, because we look at your code from that perspective, and we generate results. We also have some things like the Docker file, which we're going to speak to next. But what we're trying to convey here is we'll look at your whole source tree, and kind of try to do our best to just put details around it and you can see I can see summary lines at the top I have a thousand vulnerabilities, and I have them in different colors for critical high mediums and lows, which is stuff that people like. If you pay a lot of attention you'll notice that the top level palm at the very list, bottom of the list has zeros across work because it's pretty clean it's the stuff underneath. We can go into each one but we're not. I do encourage you at some point if you want to try it, check it out and see the results and we will give you instructions. I'm going to focus right now on the Docker file to give you like an example of some of the things you'll see and you should expect as you do your DevOps and you do your security with the solution. So clicking into this I'm going to bring this in line presentation page that shows all the vulnerabilities for the Docker file. Eric has mentioned things that you know we have code and infrastructure and other things. This is along the Docker file side, and the readout I want to give you here is, you know what we're working hard with, you know, as part of the efforts we do to make things better for the community, is we're trying to give the development teams the right information, so they can make decisions more quickly. I've used a bunch of different tools as you have, and you know sometimes the docs are good and sometimes are not and so on what we're trying to do is say well let's put ourselves in the mindset of a developer, and what are they going to know. They're going to want a nice summary so we're trying to do to the top right, you see the, you see the duplication of that summary line with 66 critical and the other numbers. One of the things we do right out to start is we order all these vulnerabilities by this score. And as you click into the different information icons for the different ones, you'll see things that contribute to why this score 714 sometimes they get really high in the 900 and sometimes they're very low. For this one here, what we're telling folks is this has a high score because it's critical, and there's a fix that's out there. We have a great talk track that we do externally and even internally where sometimes there is a vulnerability, but there is no fix yet available. Other things that you may see as we dig into other ones is that some vulnerabilities have known exploits and will list it out there, or the exploit is only available as a proof of concept or there is no known exploit, and that changes the scores. But then what we're trying to do is give you enough information so that as a person, you can look top down and say you know what I'm going to focus on this top level one first ahead of the one that might be at the very very bottom because the scores aren't as interesting. So if I look at this other one, as a person who's new to security and may not be an expert. What I want to know is, what information am I getting, you know we give you of course the title and things like in a ticket kind of way. We also give you links and I'll click into them I may not navigate into them but these are links to external databases that back up the way this, this vulnerability is described. And we have our own database but they're all public available and we have other ones that we use. And what we've learned is that folks that are highly motivated to kind of really understand and say I just want to know. What does it look like to get this detail. And how do I understand what I'm being given so I believe it. You know what is that information. And so we have those things available for for the such as the first time users to gain confidence that what they see actually mean something and you'll see it as a common pattern that you know different vendors and tools us at last in one and so on. They try to refer to third parties or sources that say, you know, don't just take our word for it, take the industry's word to say that I'm making the right decision. And after a while we hope regardless of how you use security that you build up enough confidence say I can start trusting the different vendors including sneak that they're telling us the right thing. And when you do that, then you can start working on the real part because someone's going to say look I get it I believe you. What do I need to do, and this line that I'm going to highlight here is kind of like a signal to the developers who are now just dying to make a changing code when they see this guidance you know you have this vulnerable version is one 185. And we know what speaks in this newer version, people may automatically say like I know what to do I know to change the dark file. I know where to modify that line let me do that right now, and that's totally okay. So that is your prerogative. I'm going to show you a little bit later how we make that a little easier, but we're trying our best to convey the right information so that your teams can do the right thing. There are other ones that we can dive into but I want to switch from here and click on the visit sneak link to show you just a little bit more detail because this is something that people will do. I want to dive even deeper the first one of course is a nice presentation I get this synopsis and I can start talking and socializing my team. This next part is something that people will start to really dig into and try to figure out well what do I need to do next. So we're back on the, you know, there's other screens I can show you where I can have reports, but I want to stick with this Docker file story for now. What we have here is we have a read out of this of this file, and you can see a lot of the same details. As you would expect in any contemporary DevOps tool, there are ways of filtering and searching and ordering and sorting and that's for your user convenience we find it's really helpful for those folks who may have to do some kind of triage work as part of group right this is this is like table stakes or anything, but I do want to point out a couple things. This are once again provided here, and you can see a lot of the same details with the links and so on. But there's a couple of things that I do want to draw attention. What we've learned, and we're working all the time to make it even better is that during the triage process, sometimes folks will say something well you know what, I'm in an Alaskan shop, and I value this vulnerability really a lot. This is a created your issue issue. I, we do offer this ignore for those cases where someone says it's not a vulnerability, but rather than say that let's, you know, see what happens when I click on the created your issue. As for those folks who have done this before in others fooling and environments, you know the outcome, right, you know that the idea is that we're just going to fill in someone the last minute details in this case I'm going to sign it to me. This is my test project, but in your world, the access will give you the right project and you can do this, and you'll see all the details that as you expect are going to be auto populated. And everything you do this and any tool in the DevOps universe will have this behavior is we're trying to make it easy so that your teams aren't doing the minutiae the little things that that are kind of annoying. Let's just get to work my faster by click on this link here. We'll bring up the giro ticket you see it's a low number because I have a brand new project for for this, but the idea is, is that as we navigate in here, get all the links, and your team. When they're driven by Jira issues, and maybe grooming sessions or however you call it. That's the behavior that we would support we're not here to tell you, but we will support you in that kind of behavior and you can see the link is on the page at the bottom. Now we know this is real. The last part on this partners there's always more and Eric if you want to let me know if there's other stuff. The last thing I want to show you is one of the things we learned is is remember when I mentioned how the developer may look at the guidance and say I know what to do. I know how to fix this because I know where in the file I need to change things. I wanted to show you something that we've been working on over time to make it even better for the folks and here is this idea of the base image. The results from this video today may change at any time because there's always a newer base of image. And so I may have guidance that you're going to see and you may already looking at the numbers on screen. Tomorrow it may be 8572 for all we know. But the point is, we know, because we're working with Docker and other folks and that's a tool that I did not mention up front for working with Docker to find out what are the different images out there and what are vulnerabilities. And can we give a nice message to the people on the other side. So they can say, I have information I can use to make the best decision. So you see we have this current image of 8521 and we're recommending 8571. We also have other ones, and you'll see the vulnerability count drop from 634 to something smaller. One of the things we've learned is that sometimes teams need to stick within a certain version family, I have to stay in 85 I can't move because my stack requires where other teams are saying like we don't have that requirement. I can now look at Tomcat 10 and so on to do the right thing. We feel that given this kind of information folks on the other side can can now, instead of digging into a lot of details can almost make a much more simple, not binary, but near binary decision is do I do this or not. Right. If they do something like click the open a fixed PR. What's going to happen is I think what you should expect, you know, as a developer. You may be checking out files and doing things in a branch and so on. We've learned that's like a road to operation. So I clicked on the one button I'm going to confirm that this is what I want to do. I'm going to fix PR. What this does, it's going to bring me back to last in bit bucket to show me the results of that we populated a lot of details. And what we'd like to think is that when you have your universe your your tooling and stuff that supports this kind of behavior, you're not so much worried about using an editor to type in numbers which you miss me this type and I was talking to Eric. I don't know how many times during the creation of these demos. We forget to add the right parameter like oh my goodness I didn't write, you know, 851 or something. It's sometimes it's probably almost always better to have the solution figured out to you. So the punchline here is if I navigate to the bottom and draw attention I'll even magnify it because it's a neat thing. When I draw this you may think like it's just a simple one line thing. I would like to ask folks just to kind of ask yourself how many times do I have a simple fix like this, and I still make that typo. I know I do it quite often and we would feel like you know this is this is simple. It's a one file example, but one of the things I don't show you in this, in this demo part is, you can actually multi select a number of vulnerabilities and apply all those things so I've just selected 10. I know they're one line things. There is a little bit of strength in that, and it helps out my team to do this. I'll keep moving on because I'm now in Bitbucket. Is there any question Eric I think you've been answering questions that correct. I just got one. They were asking if we have integration with GitHub as well and yes we do. Yes, thank you very much. Alright, so now you see right so we're in we're cool we're working and get lab I see the because I'm watching the screen at yes to get that right. So one of the things I want to show this like so we've done some vulnerability we kind of some fix we do some pull request I'm not going to approve this right now because I don't want to teach but I want to hop into the pipeline. So so one of the things we thought of like let's show a pipe right you know we we do have GitHub actions and other things circle CI and a lot of great tools out there. But for this thing we're going to keep it in the family and show you a a Atlassian Bitbucket pipeline, which I've been running and I think one of us that I do the fix. I'm sorry if I if I automatically applied it that was not intentional. So the, I'll click on this one from yesterday where I deployed a fix, but the, the evidence I want to show you here is, you know, we do have this plumbing and we can work with him. And for those of you who are doing that lasting bitbucket pipelines, the reference implementation that we provide you has a build, I'm sorry, a pipeline YAML file available to you so I encourage you to check it out. I did run it did deploy to Kubernetes. I'd like to show you some of the details, because that will speak to those folks who are like a quaint or acquainted with all those things. So it's mostly a standard bit buckets pipeline YAML. There's a couple of columns out there to use common behaviors so that you and your team can more quickly identify it. So one of the first things I have just to do a quick shot is the first couple of things. The first couple of stages, they're just pure bit bucket operations we do a check out we do a maven build, we do an Atlassian scan that's that's something that's we feel is a good best practice. And we would encourage folks to always consider having some kind of operation like this because you know it's a sanity check. It is a diagnostic operation, everything seems to be working. So when I'm trying the next level stuff as Eric showed us moving to the right. I can feel that this is really really working and it's good hygiene. The next item which we don't have to get into a lot of detail but I do want to call it is that we do have for those bit bucket folks pipes. We have sneak integrations as pipe first one is that we do run we allow you to inline the scan of your image just think of it this way. Not me and my development team, we're building these containers I feel really great. But I want to include the automated pipeline response of a scan for its results and we have that right. And if I were just to draw a little bit of attention not for detail but just for kind of like the visual. If I highlight these lines and show the pipe. It's not a lot of detail right. It's just some configuration data versions and so on. The net result to you is that you'll get all the beauty of getting a sneak container scan within your pipeline. The other one here for those that are acquainted is we are you know I'm just using an AWS ECR push right. I learned a lot, you know in this process and you know it's exciting to see the registry and it worked. Because one of the things I also did is I pushed this to a Kubernetes cluster. This is the part where I now start transitioning over to Eric, because he's going to show you this thing, but I did want to leave this thing here right you saw me build you saw me scan, you saw me push the container to a registry, and you see evidence here that I am pushing to a to a Kubernetes cluster. And the one of the bills actually work in recent times. At this point, I think Eric, if there are questions. You know we can, I can start the movement over because I wanted to just finish off where we're carried through right we're in pipeline mode less not seeing some stuff working in a real running container because the application is running. Any comments, Eric, you ready to take over. I'll grab the screen just a moment. There's one open question about if we support things like Yachto. I'm not personally familiar with Yachto I just looked at the page real quick. And hold that let's hold that question till after I do my demonstration and you can see if it fits in or not. And if not, you know I'll look into that and I'll tweet about it over the next day or two. Okay, thank you. I'll stop sharing here. You can take over from here and I'll start answering your questions in your bath. Let me get my screen share going. I actually have open somebody else had asked if we support a few different things and I just wanted to address those real quick on Prem support is possible it's not in the free tier. It is in the enterprise here for on Prem. If you scroll down you see all the differences. Everything I'm going to be showing you and demonstrating today is in the free tier. I'm not in sales. I'm not I don't have a quota. So I'm going to show you what you can do for free today. You can sign up for free account and do anything I show you without having to pay for a thing. If you'd like to pass, we'd love it. That's obviously but the other thing people were asking about integration. So, if you create a free account. And this is probably available on the main website but you go into your apps and go to integrations. And you'll see the tab integrations, all these different integrations get lab was the one other one that was asked about yes that is in there. We have integrations with various container registries including the big ones like Docker hub ECR GCR so forth. So the container integration the Kubernetes piece that is part of what are paid tiers as far as monitoring a live Kubernetes cluster I'll talk about that a little bit, but I won't go into it and see I and IDE plug all sorts of stuff so that's enough about that. So let's get to, let me jump back to my other screen. Let's talk about this so Marco was talking about the Tomcat version the base image version. And let me set this thing I'm going to do a little, little mock play here right. We're at a company, and we are a Java company we have a Java app that has been making us money forever it's the world's best to do list. It was something that back a couple years back 2017 maybe a few years back we decided to put it in containers so that we could deploy it to an orchestrator like Kubernetes and get to take advantage of that that fabric. So this time we built this Docker file out and if you're familiar with Docker files this it which is the these are the instructions to build an image. This is a multi stage Docker file, which means it has multiple from lines, each from line starts a new stage the last one of which is the image that ends up being produced by your Docker built. So in this case our first stage basically is just using the official Maven image at version whatever to build up our water file and a couple other artifacts we need. Then the final phase is our stage to be is the official Tomcat image at 8521 and yes that's hello old, but this is the version we were at when we lifted and shifted. And if it ain't broke don't fix it we're just going to we're going to live on this and then this is the version we know how to support and we're going to stay on it. And you see we copy copy a couple configs in and we bring over the few artifacts from the build we just did, as evidenced by the copy from build that's if you're not familiar with multi stage that's what that means is take this path from that stage named build and bring it down in here. So at the end of this we have an image that's based on official Tomcat and our stuff. Pretty, pretty simple. Let me jump over to my browser now I have alias my corp.com to the EKS cluster and load balancer deployment that's a Marco's pipeline has deployed. And here we see if I just hit it at the top we're seeing the sample server so we can verify you know we're on 8521. If I go to to do list the context, there's our awesome to do list that copyright to do list NBC. I'm not concerned about the app at this point. Yes, if you go look at our repository after all this. There's a ton of things you can play within here the struts is in here so you can do the Equifax act there's all sorts of stuff. But what I'm concerned about is containers, and specifically concerned about the Tomcat container. So if I take off my, I work for this company hat and put on my black hat. I'm looking for things to exploit. And what I may be doing is I, you know, looking at this is Apache's official vulnerability fixed list for Apache Tomcat eight. And if you scroll through here you see all sorts of goodies denial service vulnerabilities. What we got WebSocket denial service blah blah blah there's a bunch of them in here that the one that's interesting to me right now is the remote code execution vulnerabilities which means the ability to run remote arbitrary code on somebody else's server. And if I scroll far enough down here you'll see here's one that was fixed in 8523 details about it this means that anybody running on a version of Tomcat in the 85 family. Older than this could be vulnerable. Now if I click on the CVE I've already got it open here this is our sneak vulnerability database again this is just like the MITRE database which just we've added some extra niceties to it and an interesting information. I could dig into this and I could look at what exactly is the problem and I start researching this as a hacker to try to get it but I noticed there's some exploit dv links. And one of them has a Python script. Now I'm a middling Python program I'm okay, but I don't even have to know what this thing does. And I can just grab this and apply it now what it does is it's a proof of concept. It says give me a web endpoint, a URL, and I'll tell you if it's vulnerable and I'll put a poc exploit out there. Let's give it a shot. So I've got in this shell, a couple aliases I've got one called check that is going to hit my desktop.com now imagine that I've got this running in some kind of like the old school war dial and looking at different endpoints on the internet that I think might be running Java app servers. And I have this one in my list I hit this one and sure enough it says it's vulnerable. This is just the Python script unedited. I just fed it that URL, and it says it put in a poc jsp. I'm going to jump back over to this page. And let's look at poc jsp. It injected that into this website. Now all this is a bunch of a's that's not very nefarious and by any means but the fact that I could inject jsp into a running Tomcat server that's not mine that I have no shell access to or anything. It's not good. You can imagine what else you can do. In fact, if I run the other half the other area that I have a phone. I'm going to go back to my right keys here. I'm going to change this from poc to PWM. I get this nice little form. Let's get to the ENV. These are the environment variables available to me in the container. I don't even know what I'm in the server. I'm attacking right now. And I can look at this and I can say, Hmm, there's the host name of the machine I'm on Tomcat version I'm on. Kubernetes port. Ah, I'm likely in a container on a Kubernetes cluster. Service host Kubernetes ports. That's my API server in Kubernetes. Now I don't have time in the hour we have to actually hack the cluster. There are ways to get at the Kubernetes cluster if a bunch of perfect storm things happen, and things are available to me, but let's see what else I can do. Let's go check who am I. I'm root in the container. I love being root in the container because even though I'm contained that means I can do some more things. Let's see if I can, I should be able to LS dash LTR. Sure enough, I can see everything that's the odds of anybody can do that. But let's see if I can do this. Let's see who. Okay, this this standard out. There's no return so I didn't return anything but if I do that LSLTR again on Etsy. You can see a foo file was just created. So I have root access to write to Etsy. I have a read write file system. Most likely, because I was able to write to Etsy fact if I pull. Let's see. I can see what file systems I've got. Oh, look at this. What's here. Let's see what that is. That's a directory with a token in it. Now I'm not going to show you my token. That is my Kubernetes service account token. Now I know just because this is a demo and it's nothing super secure. That's the default namespace service account token with that and a little bit of cunning. I can start talking to the API server and start finding information out. Let's do something else. What else could we do on here. Okay, and map doesn't appear to be installed standard error doesn't come out of this, this JSP hack. But see, can I can I get out of here curl is yes, Google.com. I got a redirect which is good sign that I can probably get out of here and curl down things I can bring down my own executable, I can bring down my own JVM, I can bring down hacked Java files to stick into your web app. In fact, I could probably if I do a PWD. I'm LS. LS dash L web apps to do list. There's your app. So I can start messing around in that. Honestly, at this point, read write file system routes. I'm probably just installing a bit of some kind of crypto miner and taking as much CPU as I can get off your box. And I'm done, you know, or I'm a bot that's doing that, you know, whatever. But you can see how, even though I'm in a container, I can do a lot of things instead of a beachhead. If I had end map I can fact I can, I'm going to do it right now I can install I can run app get and install things if I've got access to some app repo. I can install end map and I can start perusing your network and see what else I can find. And I've setting up a beachhead in your network. So, not good. So put the brakes on that for a second. What can I do as a developer so hopefully, you know your intrusion detection is going off in production this is like what bad things, you know, but what can I do as a developer to have caught that before the reactive intrusion is have to, you know, come in and set the alarm off. So let's switch hats back to being the developer. And let's talk about a couple things. Let me flip over here. Now, I'm going to flip to my workstation so I can quickly iterate and do some things here but if you look at my images, I've built a few things ahead of time to speed things up. I've got a version of this application. I'm going to say I'm a developer working on this application and I'm doing this locally. And I've got a Java goof one is exactly what we're running out there in EKS in the prior example. And if I go back over to IntelliJ. Here's the Kubernetes deployment that I'm going to use to deploy locally. I'm just using Docker desktop you could use kind you could use mini cube you know whatever, but I can deploy this locally and run the same app in fact if I do K get all. You'll see that I am already running I've deployed it. And one of the nice things about Docker desktop is that it will give you an automatic load balancer. So if you set a service of load balancer type of, you can then just do something like this. Find my screen and change this to local hosts. And so I've got the same thing running here. But I want to know what's, you know, let's say I've been working on this app and I've done some change I whatever and I want to see well what's vulnerable in this app that I might have added or has been vulnerable for a while. If I do a sneak container tests on Java goof at tag one and I'm going to feed it some metadata I'm going to tell it to the Docker file. One directory up more and Docker file is here. So you saw what Marco showed you on the web UI. That is an active monitor of his bucket repository that actually every night will re scan the manifest that it saw and let you know if any new vulnerabilities show up day after day, which is really cool because even if you don't change anything you're going to get alerted with an email notification to say hey new vulnerabilities have been found in code or Docker file or whatever other things we scan. You should now probably take a look at because the thing you thought was secure yesterday isn't anymore. This is a command line version of oops. This is me typing incorrectly. They do wrong. Container tests. Okay. I'll look at that. But anyway, so it's scanning and what is doing now is pulling up the image layers and it's looking through all of them for all the packages you've installed what is the base image we have. It's comparing the layers to the Docker file I told it I used so it knows more specifics about it so it has all the information it needs. I'm having a network problem right now of course. So the SAS system on the back end. This is the magic of scroll backwards and I'll show you what it would have shown. Sorry guys and girls. This is live demo doing it live. So this is what I expected it to show you my home networking hopefully I'm you can still see and hear me okay. With something like this which is exactly the same information you saw on the web UI, just in a command line form and I can see all the information here but the interesting part is usually for container scans is at the end. And I'm seeing the same recommendations of that we saw on the web. I'm on 21, which is hello old and has lots of critical vulnerabilities. In case, for example, 71 only has three critical vulnerabilities. At this point, I would probably pull over my lead, my pair, my security team say hey, here's what I'm seeing. Should we do this and I might scratch the chain and look at it say well, there's no benefit really in going to nine because our app doesn't unwell on anything better than eight five. We're not ready to drink the Kool-Aid and go to the Amazon Coretto version of Java to get these zero vulnerabilities so let's go ahead and just go to the latest eight five for now. And we'll do you know all of our regression testing on it make sure it still works and then we'll we'll slate for later to look at newer versions of Tomcat might get as past as vulnerabilities. Well, it's very rare to see zero vulnerabilities, by the way, to get to zero, you're basically saying I'm going to handle every possible edge case that there is no fix for in the open source community for whatever libraries, you know, the transit dependencies everything else that are in here. So what they've done in Coretto is they've probably just carved out everything that isn't just the very very basis, which may work great for you but this is a completely different JVM. So that's, that's a little drastic for for my case. I'm not going to go through building and running that you just have to believe me that works because I want to we're getting low on time. Let me jump back to my slides though we're going to talk a little bit more about what you can do to protect yourself in depth against things that may not be known vulnerabilities so there may be zero days out there there may be vulnerabilities that don't have a fix that you can't get around right now and what can you do to help mitigate that. First of all, images, when you're building your images understand what the meat what the reason why you always hear people talk about minimizing the footprint, and that's not just about storage and transmission time and all that stuff. It from our point of view it's about security. You don't want to add a tool that your application doesn't need because a bad actor that gets in there can use that tool. For instance, pen testers will often you'll hear them talk about living off the land. That means using executables and content that are on the machines they're trying to exploit. So as to try to avoid triggering alerts, like they've installed a new application got installed in a container or on a machine. If the application already there, like curl that you just saw, I can use it. Hopefully, without triggering anything. So it's a better case but so don't give people a tool to beat you over the head. Understand how layers work. So if you're new to containers, the images are made up of rewrite read only layers and each one of those lines and a doctor probably roughly equivalates is the equivalent of a layer. You just said RM curl, you run RM curl to remove curl from your image. It's gone from the containers file system at start. But if there's another exploit that allows me to somehow get access to the host file system. Let's say somebody buy mouths in bar, which be whole horrible, but they might. I can get in there and look around under bar live Docker container D and see the sea layers that are, you know, historical in the image so I can get at those tools that you may have removed. Understand different build strategies. The multi stage is a great tool. Use it. You might also look at other tools. So if you're a job shop, you might look at jib jib is a great tool for building containers that's 100% JVM based. You don't even need a container engine to do the build, or any special tooling you just put right in your maven or gradle script and it will be built out of it you can drop it to a tar you can send it to your registry to whatever you want. And there are other tools like that. And if you attended coupon, or if you've been at it all involved in any of the CNCF discussions over the last 12 months, you'll know that secure supply chain is the buzzword of the year and it's not just a buzzword. It's things that those of us who you know worked at Docker and other places in the past have been harping on for years, where you need to pay attention to where your artifacts come from and images are artifacts. So don't deploy anything to a server, especially to production, that you don't have an audit trail for a chain of custody. And the new developments around stick store and some of the other open source tools are looking really great to help us codify and document with a software build materials where these things come from. And only images that are built by your automation that is auditable should be going to any deployments. Eric Smallings image that I built and pushed to Docker hub, that should never get deployed anywhere important, because you don't know where it came from. When it comes to runtime, this is where we can really start to mitigate things so I was rooting a container and you saw that even though I'm contained that means something I can I can run app get I can run all sorts of stuff that shouldn't be able to do because I'm you ID zero. There are other mitigation efforts that you can do for your host, but understand that if you volume mounts, if you host host volume mountain like a bar example I gave, I'm you ID zero, I can read and write to that volume that's mounted if it's not mounted read only. I could edit things on your host that way. So that's a bad day, just don't run as root, change your root user to something else. You might have to refactor a little bit of your deployment to make that work but it's always a good idea. So that's all most the open source official images start as root, but many of them included a user that you can switch to so look into that privilege containers by default or that's turned off thank goodness. This gives you root access to the box I mean you got host device access and it's horrible never use privilege containers. It for business applications I've never seen a good reason to use them if you're doing something like, you know you're writing a monitor for a cluster or you're Oh gosh. Networks that you stack manipulation. Maybe you're writing a CI, but for most business applications most enterprise stuff you're not going to need privilege container. Same with Linux capabilities. Linux capabilities or kernel system calls that your application might need to make a call to for instance. Ping needs this it uses capabilities to be able to get at the network stack to do ICMP calls. That's a capability it needs your application again business applications often don't need any of these system calls. And you can just usually safely drop them all which is an easy command line change on Docker or a manifest change in your pod spec. There's a ton of blogs out there you can look into on determining what capabilities you need and S trace and Falco and things like that can help you figure that out as well none of time in this call to talk about it. Read only file systems also another key one if you can make your file system read only as far as you know the base. What that means is when the container engine brings up all those read only layers of the image get codified into one thing. It normally applies a read write layer on the top and all changes are handled through a copy on write operation and everything happens at that top layer. You can tell don't include that and everything now it's just like you mounted your drive read only. If your app can run that way do it if it can't figure out how to make it do that because immutability is king. There's another 12 factors so you should be doing it anyway but I'm Mount host empty doors for for workers or or do things that you know as as you can to make sure that you can get read only if you if at all possible pay attention to your volume mounts as well make sure you're not over extending those privileges and deploying from those sources, same as the secure supply chain piece. Kubernetes specifics. I'm just going to leave this on the screen. Use these things. Make sure that if you're using secrets for instance that your your cluster ops have encrypted them at rest it's not there by default if they're building their own clusters. You know manage clusters have that automatic use our back just do it security context learn about it there'll be a link here in a second I'll give you some info on on all the security context you can set a lot of things I just talked about are configurable through security context in your pods network policy is is not people get scared of it because it seems complex it's really not it's how do pods talk to each other can they are can't they. I'm a fan of setting it up a zero trust no pods can talk to each other at all, except for what I add to the allow lists. So make sure if you do that you add DNS because you can do that you can't look up any services, and all of these things I've been discussing you should enforce them with some kind of tool pod security policy is one way to do it but that's deprecated and going away in 125, don't put me on that. It's being replaced by pod security admission which is still in beta, and as a 123. OPA with gatekeeper for Kubernetes is a great tool that a lot of people use to enforce these things cavernos another great tool. I'm going to skip this slide because we're out of time. These are links, we will share these slides I will tweak them out myself so if you follow Eric Smalling on Twitter, you'll see I'll share these slides out and Marco probably will share as well as well by a slide share or something. And quickly, the repository we used to be in there are links to our documents. I mentioned a security context document this is a cheat sheet that will go over a lot of them in some of the you know the bullet points on how to use them right. The other one I want to call out here is if you are using pod security policy today, you want to get off of it and be or plan to plan to get off of it when they deprecate read this blog about it. And if you want to know more about security context, watching my videos. So sneak.io sign up for a free counselor playing with this stuff today Marco what do you got in the last great so I did my best to answer questions. Thank you Eric. Looking forward to, you know, if folks want to wrap this up or ask additional questions. Thank you. Wonderful. Thank you both Eric and Marco. It's been a pleasure having you here today and thank you to everyone who joined us. As a reminder this recording will be on the Lenox Foundation YouTube page later today and we hope to see you at future webinars. Have a wonderful day. Thanks everybody.