 So, hi everybody, thanks for showing up, for the talk that holds you up for lunch. I'll try to not go over in that exercise. So, my name is Guy Pujarni and I'm here to talk to you today about security, specifically opens for security, securing the libraries and the packages that you have inside your application. These stranger written libraries, hence the title, Stranger Danger. I've got a couple of words for context about who I am or kind of my background. So, I have a fairly extensive application security background kind of through the Israeli army and then work on the first web app shield, web app firewall app shield and scan or app scan. And then subsequently went on to find a web performance company called Blaze, that got acquired by Akamai with CTO there for a few years. And now I am the CEO and founder of Sneak that it deals with opens for security and helps you sort of address and secure those libraries. So, let's get right into it. So, open source is awesome. I think probably everybody in this conference is kind of sold on this notion. Open source is great. It lets us share the work that we do on the flip side. It sort of lets us consume and build on the shoulders of giants and all the community created content. And fundamentally, it helps us focus, right? It helps us just build the things that we want to build and we want to make good. And as a result, open source usage has exploded. Cloud Foundry is one example of that. We see a lot of this in the library world as well. And there's a gazillion stats of it. Unfortunately, the fact that something is open source does not mean that it's secure. This is a little bit of an older notion that may be true. There was this concept that says that because something is open, it has eyes on it. And therefore, you know, it's scrutinized and we will find the problems. That may or may not have been true in the past, but today with the volume of activity in the open source land, it's not true. The fact that something is open source doesn't just make it inherently more secure. It doesn't make it insecure at all. There are amazingly secure projects out there, but it's not something we should take for granted. And we've had a lot of sort of sobering examples of this in the last little while, right? You know, ranging back a couple of years for Heartbleed and Shellshock and Logjam. This is not a real logo. This is just the Cloudflare kind of ripping off the Cloudflare blog. But all these, like, designer vulnerabilities or brand name vulnerabilities that just make headlines and kind of create a lot of noise around some vulnerability. And the reason that they're so important, the reason they're creating so many waves is because open source is everywhere. These are severe vulnerabilities, no more severe than in some commercial entity. But because everybody's using these open source components, the impact of them on the web, on the world, is greater. So, you know, this problem is very prevalent. We see there was all these different stats on, I always kind of debate which one to quote, you know, one stat from BunionOps that was a little bit contested. I talked about how 30% of images on Docker Hub carry known vulnerabilities. It's a little outdated, but the stats are not that different today. And, you know, there are other stats around just the prevalence of these types of known vulnerabilities in components. On the other side of the fence, you know, when you talk about the problem and sort of defending against it, you've got the attackers. And for attackers, open source is very, very compelling. And as a result, they're targeting open source. They have, you know, the vulnerabilities are already found for them. These are just evolving projects and there's a flaw that's found. And then one vulnerability has many victims. So you don't need to silently try and detect a vulnerability in somebody's website that is very hard. It's hard to do without getting caught. And then it's subsequently useful for that specific organization that you targeted. While open source components, if they have vulnerabilities, they're reusable components and therefore these are reusable exploits across all those using them. And you can see this in the real world stats. You can see, again, a million stats to quote from here and all sorts of reports, but Verizon has one stat that states most attacks exploit non-vulnerabilities that have never been patched despite patches being available for months or even years. So these are patches that have been in the wild. These are non-vulnerabilities that have a fix that you did not take and therefore they're rolling out. Symante goes on to say that by 2020, 99% of vulnerabilities exploit, this is a bold statement, you know, like only 1% of other exploits would be different, would be because of vulnerabilities that have had a patch for over a year or at least one year. So it's a real problem and it's something that's sort of on us to try and take on. Cloud Fundry tackles this straight on, right? When you use Cloud Fundry, a lot of the great mechanisms that it has in it and sort of the stem cells and the updates to them that Bosch does help secure the operating system dependencies, right? It helps update them above and beyond the securing the code itself in Cloud Fundry, the Cloud Fundry code, which also is done very well. Just the vulnerabilities that might be in the operating system and the containers that you're running, definitely if you use it as a pass, Cloud Fundry handles a lot of that for you. But all of that is the operating system dependencies. Like a lot of those are in these open SSL libraries or image magic you might be using. What about app dependencies? So these are Maven packages, NPM packages. They're just as hacker-friendly. You know, the vulnerabilities are already found, they're prevalent, they are hard to update, it's hard to wrangle all of these dependencies for the app developers. So they're just as compelling and just as exploitable. But they're not quite as well-handled. We see this, on one hand, we see this in the handling of the vulnerabilities. So we look at Maven, again, one stat to quote, 59% of reported vulnerabilities in Maven packages remain unfixed. These are reported vulnerabilities that are known vulnerabilities and just do not get fixed because open-source maintenance is tricky. You know, you have a free open-source maintainer that's providing value to the world for free and they may not be working on this all the time, right? They may not be, they may have, like, had a child born or maybe, you know, at some point just ran out of energy. Of those 41% that do get remained, there's also a larger lag. The mean time to repair the vulnerabilities is 390 days, take over a year to fix these things. Even the CVSS 10 vulnerabilities take a mean time to repair of 260 days. These are like the really bad ones, the ones that really create a massive risk. We've seen this a little bit recently. We've seen this big vulnerability in the struts framework in Maven. Who here has kind of experienced that joy? A few hands. So this is just one timeline of showing this big vulnerability that I'll actually demonstrate in a sec where in struts there was a massive remote command execution vulnerability that was detected by seeing exploits in the wild. That's sort of how the vulnerability kind of came to light. And this is sort of a graph of its attack timeline. And you can see that as soon as the vulnerability was discovered, there was this surge. The attackers were very quick to exploit it and just sort of went on and started to sort of apply those attacks in real time. And then they sort of tapered off as they refined their techniques and they figured out where did they attack. This is the Java world and there's a lot of examples on that. In JavaScript, the situation is the same or arguably even worse. In JavaScript, you've got NPM and NPM is also awesome. And as a result, NPM's usage has grown dramatically. We talked about open source usage. NPM is just kind of the epitome of this. I need to update this slide every time. It's over 400,000 packages in it. Over 6 billion down as much. Mind-blowing numbers at an even more mind-blowing pace of growth. And it leads to the situation where this is your app and this is your code. And this slide sort of has... People think that it's a negative slide but it's not a negative slide. It's great. You could get all that sort of orange value just by writing that purple code. You just needed to write this little bit and you could build on the power of the community of crowdsourced components. It's also scary. It's scary as hell. Here's an example of this just to sort of give a more visceral example. It doesn't really matter what the code does of a serverless function. It's a real serverless example. It fetches a file and stores it in S3. It's 19 lines of code. To do this in 19 lines of code, it uses two dependencies which in turn use other dependencies as well. So it uses 19 packages of role. Any guesses as to how many lines of code in the packages? No? No takers? 10,000? 50,000? 190,000 lines of code. So the primary volume of function in this when this function is deployed is not in your code. It's in those components. And where the majority of your code is is also the majority of your risk, the majority of the vulnerabilities. So each one of these dependencies is a security risk and you have to ask yourself, do you know first of all which dependencies you're using? It's really hard to lose track of them. If you have two components, they use 19. Sometimes those graphs are bigger. And for each one of those components, how well are you handling this problem? Do you know if they have known vulnerabilities? In the world of NPM, I quoted some stats from Maven. I should do this for NPM as well. About 14% of NPM packages carry a known vulnerability. Many of these packages are very lightly used. It's actually the popular ones that tend to also have bigger trees. And as a result, we see that about 80% of apps that are being tested actually find vulnerabilities in them and those would have vulnerabilities. Some of these vulnerabilities, just like in Maven, have big remote command execution or sort of severe process scripting or denial of service vulnerabilities. This is one example. There was a remote command execution vulnerability that was found in the DustJS templating library that was used by LinkedIn and PayPal. And in this case, it was found in a bug boundary like it was found by a researcher, which is a good thing. So fundamentally, most of the code in your apps, as I mentioned, comes from these dependencies and vulnerabilities comes here. Let's talk a little bit about Cloud Foundry. Cloud Foundry does an amazing job at the operating system level, but it sort of stops there. Once you've built a droplet, it doesn't manage the contents of it. The droplet floats around and whatever packages that were built may be in Cloud Foundry and pulled into it, they now stay there. And it's on you to understand what is in those components, what is in those droplets. And over time, the libraries in those droplets grow stale and they grow vulnerable. They don't grow vulnerable. The vulnerabilities are already there, but vulnerabilities are discovered and therefore now they are easy for attackers to exploit. And that's an inevitability. All software has bugs, and some of those bugs are security bugs and vulnerabilities. If you don't treat it, if you don't track them, if you don't update them, they will grow vulnerable. So with that, let me switch to the more interesting part of the talk and do some live exploiting of those. Let me switch to the browser. And I will show you a couple of friendly apps that we have here deployed in this case just for convenience. I use the Pivotal Web Services for this. So one is called Goof2Do, and this is an NPM app. It allows you to do things like CallMom and allows you to put some emphasis using markdowns to say buy, be your... things that require some further embellishment. And it's very proud of itself. It has a nice about page that talks about how it's the best to do app ever. I will challenge you if you say otherwise. And it also happens to have a bunch of dependencies that are being used. So let's look a little bit at this application. So I'm going to use Sneak, which is free for open source, and you can try it out over here on your own app. Sneak can connect to Cloud Foundry. So I will go in. I've already configured it to point to this account on Pivotal Web Services on my Cloud Foundry instance. And I'm actually going to test two applications because I will show some examples on the Java app in a second as well. So I'll choose Java Goof and CF Goof, which is just our example of the NPM and the Java Goofy applications that are a little bit more vulnerable than desired. And what this would do is it would go off, download the droplets. I kind of cheated here. I already did this before. Download the droplets, expand them out, see which dependencies are in them, and then run the reports against our database of known vulnerabilities. So let's look at the NPM one first. You're going to get this long report, and it says it has 19 known vulnerabilities, 43 vulnerable paths. So this is important. When you talk about a vulnerable library, question number one is, am I using one? Like, do I have a library that has vulnerability? And question two is, how can an attacker access it? So you can look at the dependency tree and see, for instance, that in this case, for instance, I've got this component called negotiator. It has a high severity vulnerability. It is one vulnerability, but it has two vulnerable paths that lead into it, each of which is a path you need to fix the vulnerability. Let's look at a couple of examples here and just do some exploiting. I'll start with an easy one that's actually a more medium severity one in the ST package. So the ST package has a directory traversal vulnerability. ST is a package that we use here to serve static resources. So we use it for CSS, we use it for JavaScript, but we also use it to serve our awesome about page. And it has a directory traversal vulnerability. So let's sort of switch the browser and see a little bit here. I'll switch here and I'll bump the font size a little bit. It's good. So it has a directory. First of all, let's cheat a little bit here. I've got some shortcuts. So we'll just sort of curl. We can't really hack in the browser. The browser does all sorts of convenience normalization for us and things like that that are just not amenable for hacking. So here, we're asking for this about page using curl. You know there's a directory traversal vulnerability here. What would you try to do as an attacker? Trying to get some audience engagement here. No? Directory traversal implies you're trying to break off, traverse away from the folders you're supposed to be in. Yeah. Dot dot slash. Thank you. So you just do public and this is the folder and you're going to try and do dot dot slash or dot dot slash dot dot slash. Let's see what happens. Now, fortunately or unfortunately as an attacker, nothing happens because if I scroll up, I see that I just got the homepage. I got the title over here, the goof to do. I don't know where my cursor is. Over here, because SD is actually a very security conscious library. It did try to capture these components. However, it has a vulnerability. If you were quick on the eyes, you might have noticed it in the advisory that it doesn't look for a dot dot slash, but it looks for the URL encoded version of it. It does look for the dot dot slash, but it doesn't look for the URL encoded version of it. Now, I can just do . . . . . . . . . . . . . . . . . . . . . . Processor again. Oh, here we go. Found it. So the beauty of ..slash is that you can chain them to the extent you want. Go all the way, all the way, and then just add etc.passwd, and voila. Which is fun, but actually not that dangerous because the etc.passwd, now that there's a shadow file, it's not that dangerous anymore, but it's just a good demonstration of it to download source file. So this is a very simple one, straightforward, easy to understand. Let's get a little bit fancier. Looking back at the results, I can see another one in Mongoose. Anybody here heard of Mongoose? Of MongoDB, I assume everybody heard. So Mongoose is the primary way that MongoDB is accessed in the world of Node, and it has a remote memory exposure. Remote memory exposure is basically the same vulnerability that happened heartily. It means that you can extract memory out of the server and expose that to an attacker. And this is done because of a beautiful object in the world of JavaScript, not to get too detailed here, called buffer. So buffer is a JavaScript object that allows you to construct it. You can initialize it with a new buffer, which would allocate three characters, 1, 0, 0, or you can allocate it with 100, in which case it would allocate 100 bytes of memory. And if I do this again, you will see that I have different memory because buffer doesn't zero that memory for a variety of performance reasons. I don't know if it's sort of good reasons or bad reasons, and this behavior is kind of in the process of being changed, but it just sort of allocates those bytes of memory. So if I'm an attacker and I can trick the application into swapping the component that I passed in, the constructor, to be instead of being a string, be an integer, which is a process that's called type manipulation. If I can manipulate that type to be an integer versus a string, and I can get that information to be exposed back to me, et voila, I've leaked some pieces of memory. And this memory is whatever it is that was stored in memory through the, like it's a random piece of memory. So over time, if I do this enough, then I would actually extract out secrets from the code, including source code, including anything that you're sort of environment variables. So this behavior of buffer tripped many packages. There are quite a few vulnerabilities that are related to buffer, and not the least of which is monboost. We use buffer for our, in our database, so in db.js here, we use buffer as that content so that we can support images and other sort of fun stuff in our to-do items. And as a result, this component might be vulnerable. So I'm just gonna, I'm not gonna belabor too much this exploit, but the way that I would exploit this one is to switch again to the command line. And what I will do is, most applications, many modern applications today, don't just support an input of a form, like I submitted before, but they also support JSON input. So I can just the same as I kind of fit in call mom. I can also go here invoke a curl command and say, with a JSON input, put a feedback. So because this is JSON, I can do something like this as well and say, I don't like 800. And go back here and I will get a string of 800. What's my next step? Remove the quotes. So if I remove the quotes, JSON is an awesome format, so easy to use. And when it is deserialized, the types are deduced. And this is the case in most dynamically typed languages. So if I do this, this for convenience echoes back the note to us that you'd already note that it's a longer note. And if I do this, you'll see a bunch of binary information here. If I do this again, and if I'll get lucky here or not, sometimes you start seeing some actual source code bits here. I didn't get that here. You see some none over here, but you would increasingly get more and more pieces of memory. And again, if you do this silently over time, you will extract all the memory. So these are a couple of examples. I show one quick Java example and then I'll switch to talking about the interesting part which is how do you fix it? Or the important part, maybe not quite as fun as hacking. So a bunch of examples, Goof is an open application and you're welcome to, it's on sneak slash Goof, you can clone it. Hacking is illegal, don't go off and try to exploit websites out there, but you can download this app and exploit it locally for education. Let's just sort of finish off with an example of a spring and struts based Java application that I have here, which is also a to-do app. It allows you, just like before, although you can tell that it's a Java app, not a note app. Say once again, call mom. I have to say when and create that to-do item. So if I look at my report over here, I also tested the Java app and I can go back and I can see, amongst other things, that it has this struts vulnerability that I mentioned before and just to sort of demonstrate how bad this vulnerability was, let's exploit this. I'll switch to terminals here. So all you need to do to exploit this vulnerability, no code changes required, no specific configuration required, is to send this payload as a header. These are just a set of a content type header that is specially structured. So all I need to do, and again, I have some shortcuts here for convenience, is cut this file, this exploit headers file. I will replace just for convenience. I put this command placeholder that I can replace with the command of my choice, which in this case is catetcpasswd. And then just channel that, I'm using htpi versus curl just for a syntax highlighting. And voila, it's gonna run and extract that out. I can change this command, I can do say n, for instance, to get the environment variables, I get the environment variables. Super trivially exploited, enabled by default, super severe vulnerability, which justifies what I hope was shuffling by everybody in this room that deals with Java to find out where are you using this and how to fix it. So these vulnerabilities are, they can happen in your code as well, but they didn't. The developers of this applications use popular packages as well. In these examples, use popular packages that many, many, many organizations use. And nothing, you didn't do anything wrong by using those packages. But over time these packages might have vulnerabilities and some of those are very, very severe. So let's talk a little bit about what do you need to do them. So the first step we've already done, we found, we tested these droplets, we saw that we have those vulnerabilities and you need to do that recurrently because new vulnerabilities are disclosed all the time. In this case, we will do that, you can run these tests daily and just sort of get alerted on new vulnerabilities. But the other side of it is that finding the vulnerabilities is clearly first step, but what you want to do is you want to fix them. So let's talk about how do you fix them. So I will do this again with Sneak, you can follow the same logic on your own tooling. I'll connect to GitHub. I'll find these same applications here. And I'll just use a CFGOOF and JavaGOOF and run the test. Now a few words of note, these are kind of anecdotal, but they're useful to keep in mind. One is when you're talking about Java applications, oftentimes you will find there are multiple POMX metal files in the root. You want to make sure that you test them all. So I can just go on here, expand, see all of my POM files and this part of the process, I can go in, see the report, looks very similar to what I've done with the Cloud Foundry apps. I just, instead of looking at what was deployed, I looked at the source code and I extrapolated out the tree and I can find in my CFGOOF, I can find my Mongoose vulnerability for instance, I can find it here in the reports just like I did before. Similarly, over here in the web struts POMX ML, I can find my struts core arbitrary code execution that I had before. So this first part is just about finding the vulnerabilities, but in this case, I found those vulnerabilities in the source code, I found them earlier. And because of that, I can actually act on them in a kind of more preventative fashion and I can also look to fix them. So I'm going to do this here again as well. I'll just hit open fix PR. And in the case of Sneak, we'll do that heavy lifting for you, but I'm just doing this to demonstrate how you would fix them. So the first step that you have is about finding these vulnerabilities. When you find them in the source code, you want to say, how do I make them go away? Fixes are done predominantly using upgrades. Like the gist of it is, we show those stats around Maven packages that are not fixed, but in many, many, many cases, especially for popular packages and especially for widely used packages, there is a fix. You just need to know that the problem occurred and you need to find the upgrade to do that. This is sort of a simple JSON example, sort of a package. A JSON example of how you would do it with node. You just need to figure out which dependency upgrades you need to do. Sometimes it's a little bit more like work. That's just sort of the way it is. You might be using A that uses B that uses C and you need to figure out how do you upgrade C and how does that manifest down the dependency chain? You can use tools of your choice to try and help you elevate that or sort of alleviate that problem. But upgrade is really your primary, primary path to solving these problems. The second best path you have in case you cannot upgrade is to try and patch. Now patch is a concept that is very well known in the operating system world. Like you do up get upgrades and you get all these patches that have been either curated or created by Canonical or by Red Hat and they just make the problem go away. In the world of application dependencies, they're not quite as prevalent but different ecosystems give you different ways to do this. So in the world of Ruby, Rails tends to publish patches for vulnerabilities when you cannot upgrade them. So you can pull them down. Sometimes, sometimes specific Java packages give you patch versions of them. And specifically for us, we have open source patches that you can choose to consume which for JavaScript libraries allows you to patch a vulnerability after installation. So you would basically build time, install the dependency, apply the patch and make the vulnerability go away. But patching is a new methodology in the world of app dependencies which I expect to grow because it's all about making fix simple. So I guess my kind of guidance in the context of fixes, first of all, upgrade when you can. Second is tee yourself up to be set up to fix quickly. So to be kind of a, if you don't make fixing the vulnerability easy, then the reality is that people will just not do it, right? You'd get into that mindset of looking at the vulnerability and trying to convince yourself that it's not a problem to the extent you can. Let me go back to my slides. So I gotta say that something is the new black. So I would say that I think that as the world of infrastructure evolves and that platforms like Cloud Foundry and trends like serverless abstract away the infrastructure from us and that the pros are handling patching of the servers and dealing with operating system dependencies, more and more of the attacker functions and attacker focus and tooling would move up into the application layers, right? Just because you made it harder for the attackers to attack, they're not just gonna give up and go home, right? It's not gonna happen. They will move up to the next open door and application dependencies are those. So we have to think about these components of infrastructure that we have embedded in our applications and think about how do we handle them. I talk about many of these steps just to kind of structure them. Again, regardless of which tool you use, I think you have four phases or I would suggest four phases to deal with this problem. First, you have to take the blindfolds off. So find these components. Know whether you're using vulnerable dependencies. You can do it in source code or in the code that you've deployed. Second, you want to fix them. You want to invest in making fix easy. Sometimes fixing is just restaging droplets. Sometimes all you need to do is reinstall. Sometimes you need to actually change the source code in the process. And sometimes it means you need updating code. The third is that once you've fixed these vulnerabilities and once you've found them, you fix them, you're in good shape. You want to prevent the addition of new vulnerable packages. You can add tests with, again, your tool of choice as a part of your build pack, as a part of your concourse pipeline, as a part of whatever pipeline process that you have to gate the addition of new vulnerabilities. Don't get worse, right? Sometimes you can do this even before you fixed it. Just don't allow the addition of new vulnerabilities into it. And then last but not least, be set up to respond to newly disclosed vulnerabilities quick. Once a vulnerability that is in your system, in a package you're using gets disclosed, the attackers are going to jump on it. We see these stats again, again, again. So it's a race. Who will act first? Will the attacker exploit the vulnerability first? Or will you fix it first? So again, investing in an infrastructure in a framework that allows you to get these alerts proactively and to be able to respond to them quickly. That's it for me. Open Source is awesome, but please enjoy it responsibly. Thank you. Can answer any questions? Cool, yeah. So Sneak and I think pretty much any tool in this space would work both on direct dependencies and indirect dependencies. If you encounter anything that only works on direct dependencies, that would be severely limited. You shouldn't limit yourself to that. So in general, again, for Sneak, but this is also true for the tooling in the ecosystem, the tool does expand out and try to sort of explode the full dependency graph that you have, including dependencies that are sort of bundled in or otherwise included with another dependency. When you talk about deployed code versus source code, that's oftentimes, that's the subtlety. When you test deployed code, you start from the facts. You sort of start from, okay, what's on the machine? I'll identify those components. I'll tell you if any of them is vulnerable. And then I will try to backtrack and say, how did it get there so that you can be advised of exploit paths? If you start from source code, it's the other way around. You get the manifest and you try to sort of approximate what will the tree be. In the majority of cases, both of those lists leads to the same result with like a 5% delta. The best practice would be to attach, like to test in both cases where deployed code is closer to the risk. You know where your vulnerability lies. While source code is closer to the fix, you know how do you fix it and make it go away. Cool. Well, I'll let you go to lunch and I'll be here if anybody wants to ask further questions. Thank you.