 for software developers who don't know security at all. So this is a very interesting topic that has been wildly popular the last couple of months. And I'm personally very interested in this topic too. So I'm curious to see what he's going to tell us today and how all of you can make your projects more secure. So please welcome Sai. Thanks. So OK, I can't really see you guys, but it looks like it filled up pretty good for like 2 PM talk based on Java stuff. I will just give you a quick introduction to me or myself. And we can show you why and what and how we can do all this fancy automated security testing stuff. It's not necessarily this order. If you have questions, maybe you can raise your hands. And maybe I will see them, but probably not. Or you can just scream at me. And let's give it a go. I'm Chris. I'm a system developer. I do Java backend stuff mostly. I do Kubernetes and cloud stuff. And I'm trying to get into security. I'm also a co-organizer of the Karlsruhe DevOps Meetup. I work for Cinex. Just a quick shout out. We do mostly Java development for our customers with there for the special needs and stuff. Our slogan is code with attitude. So we try not to work for arms stealers or people who do nuclear energy stuff. Yeah, we also do consulting. And we have glitter shirts. And they paid for our camp tickets. So just a quick shout out. If you're ever in southern Germany and you're looking for a job, thank you for listening. Any questions? No. I got questions. Is any of you directly involved with a software product, developing it, running it, maintaining it? Surprise. Who of you uses Java? OK, the talk will, or first part, will run mainly with Maven context. So I hope it's not too far fetched for people who haven't seen this. I think it's quite self-explanatory. Who of you uses Docker or Kubernetes or something container in production? OK, who of you scans your software regularly for security problems or known vulnerabilities? Who of you runs software in production where they know they have known security vulnerabilities? OK, I think we got back to this later on. OK, so what's the problem with software security issues? What could go wrong with software that has not only technical issues but also security stuff where people could break in and do bad stuff? Well, you could lose your business data or your customers' business data. You could also use your actual end customer data, which is a very hard topic. I think it has been quite interesting in Germany in forever. But since 2018, the GDPR kicked in in Europe and apparently a lot of companies have never thought about securing their users or customers' personal data. I'll have an example about this later on. Security issues could cause service interruptions, like somebody is trying to attack your web server. If your web server goes down, your business is fucked. You can serve your customers. You can make money. Yeah, well, anyone can think about what happens. You could use or you could have industry mill function. We have the Kritis Infrastructure in Germany, where it's infrastructure that's used for your daily life and that's absolutely necessary, like sewage cleaning stuff and electricity and stuff like that. And maybe you've heard about deaths that actually happened because automated cars drove over someone because they thought it was like some kind of paper bag flying over the street. That's a technical error, but this could also possibly be used by a hacker if you have a security issue in your automated car that people have shown that they could remotely stop cars, start the ignition, brakes, and whatever. OK, does anyone know about Equifax? It's like the German Schufa. They do credit monitoring for their consumers. So if you apply for credit, if you have trouble paying your bills, it could happen that if you live in the US, you show up in their databases. In 2017, some hackers breached their web server through a vulnerability in Apache struts. It's an open source project where you can build web views. Turns out that struts had been patched. There was an official patch for struts for like four weeks and apparently Equifax didn't know about this or nobody cared. And people were able to infiltrate their servers and take out a lot of personal data, like I think 133 or 43 million social security numbers. And if you know about the US Social Security System, it's a number that you get assigned by birth and you can never change that. And there's a lot of services where you can change your passwords. You call them on the phone and you have to give them the last four digits of your social security number and that's their authentication for you. So this is pretty bad. There was also a lot of credit card numbers involved. And well, you can think about what happens next. Vulnerabilities can also happen in the platform that you use to run your servers. Germans might know this under the name Panama Papers or maybe everyone. I don't think we have time for this, but they had a bad Drupal version that not upgrade this for their web servers. And well, that turned out not so good for them. So why would you not patch your software? Well, there's negligence. So you just, like, you don't care. You don't have the time. It's priority problem. Your product manager is like, no, we need the feature. Maybe you might be preoccupied by doing performance tests or something that you prioritize higher. It could happen. You might not have the training. Like, when I started, interesting myself in this whole security thing, I had no idea what to do. I'm not even right. I've been doing this for like two years now. And I'm far from being an expert in software security. I have more like the average, basic knowledge. So you might like the inside. Or you could just say, security is not my department. I've seen people at customers that claim that they are software developers. And the company has a security team, a pen testing team. They have a test engineering team and people that change your diapers in the morning. I don't know. OK. And there's the, to tackle all these problems, there's the Open Web Application Security Project. It's a, I think it's a nonprofit organization. And they give out a top 10 list of application security risks. And sometimes they renew it. So I think the last one was from 2017. And what we are trying to tackle today is the top nine. It's about using components with known vulnerabilities. So this is like somebody has found a vulnerability in struts. And they reported to someone at Apache or at the struts project. And they would, the report is publicly available. So everyone can see it. And everyone can also write a scanner for this. You might ask the question, why do we start with a nine and not with the top one problems? Well, you can have all the top eight problems and many, many more inside third party libraries that you use. And so we try to get the easiest way to tackle this problem. And it's also, you can learn a lot by this. OK. So what can we do to address the problem? We can search for known vulnerabilities. There's public databases where we can, everyone, also the bad guys, but also the good guys, can look for vulnerabilities and fix them and see what they have to do, what versions they have to patch to. That's pretty easy, technically, because there's tooling for this. We have to implement a process to fix it. You find a vulnerability, but you need the time to fix it. There's priorities, there's features to do, there's other stuff to address. And you have to find a way who is going to fix or to patch or to delete vulnerable software. And I think that's the hardest part for a company. And I personally would also treat security issues like technical debt. The longer it takes you to patch this, the harder it is to upgrade or to patch, because there will be new vulnerabilities, there will be new features. Every one of you knows this. If you wait for weeks to patch software, technical issues, vulnerabilities, whatever, and the software progresses, it's much harder to fix it and to update it. So and also, well, I think the best thing is also always to automate everything. So I don't want to know, it's Monday morning, 10 AM, I have to run a vulnerability test. No, it should be automatically. It's the same as with unit tests, integration tests, performance tests. There's software, it changes, so it should be automatically built, it should be automatically tested, and it should be automatically scanned for security issues. OK, so let's check this out. How we can find dependency in third-party libraries. I've chosen to make a quick Maven project. I hope if you haven't used Maven or Java before, you still get the message. We will scan for components in Docker images, because stuff like Drupal could be packaged into a Docker image, where you put your actual web server or web software or website in. And if you've got time, we can also scan our API later on, because we are doing a web API or web app. OK, and well, I said we want to automate it. So I tried to show you something that could be in a continuous delivery pipeline right now. So this is from the Jenkins Blue Ocean plugin. It's an open source tool. Every Java developer probably has used it before. And these are steps that I think are in some respect necessary for modern software development. Like you want to unitest your stuff. You need to package your new software, so you can use it. You need to upload this artifact, so it's safe for later. You run a Docker build if you use Docker, and push the Docker image to the Docker registry so it can be deployed to some servers. And this is just a demo project. Obviously, usually, you would add more tests. And maybe it would be deployed to production or to test systems automatically in this pipeline. So we could add additional steps to this pipeline to increase the steps and improve our software security scanning part. I have added three demo steps. One is running dependency check. One is running a container security scan. And one would be running an API security test. It could also look like this. Yellow usually means in Jenkins that the build has been unstable, so it has been built. There's no major breaking point. But something's fishy, so you have to look into the build. I'll show you this later on. I think the build should always build. There's people who say if you have a vulnerability or if you have bad tests, well, tests should break the build, in my opinion. But vulnerabilities are a little tricky because you might have to fix two vulnerabilities at once. But you can't do both at a time, sometimes, like maybe one of them is missing a patch, but you still want to fix the other one, so your build has to run. So in my opinion, you should have some visual or some alert that the build has not fully, or the pipeline has not fully run, like by setting it to unstable. And the teams I usually work at, we have big dashboards for our development processing. And we usually show traffic lights, so dashboards that will show by color if the builds on the most needed branches, like the master branch or any stage systems are fully built. And so you would directly get visual feedback if something has happened by the automatic build process. So you don't have to necessarily check every build. But rather, you get a visual alert if something went wrong. And then you have to implement this process, how to actually fix this problem or address the problem. OK. So what's a vulnerability? Vulnerability. Wikipedia says the quality or state of being exposed to the possibility of being attacked or harmed, either physically or emotionally. So something might happen to your software, and it's in third-party libraries. The problem is that it might be publicly known because you haven't patched the software. And well, how would you know this? There's a public reference database, or multiple ones, that lists common vulnerabilities and exposures. Well, a CVE, Common Vulnerabilities and Exposures, is actually a description for a vulnerability. So I've taken the Equifax example with Apache Struts. So you see there is the vulnerability description for the Apache Struts problem in this particular version. You see there is, on the lower left side, there is a base score that says something critical, and you don't have to be a security expert to know that you should probably address this problem. And it has some information, and you get info about what's the actual problem in this Struts version. Like there's something, content crafted package you could send to this thing to run code on somebody else's web server. And there's also some scoring, and some exploitability scores, and there's much information that you don't actually need at the first place when you get started. It's more like an in-depth information thing later on. OK. I'll jump into the Maven and dependency part. And this is just an example how Maven dependencies in a modern Java application, what you look like, I have written, or actually, I have just generated a Maven project, and I have added some dependencies and third-party libraries that I think are the minimum that I would use in every new Java Spring Boot application that also runs something web service. So we have the Spring Boot Starter. Spring Boot is a framework on top of Java that actually does a lot for you, so you don't have to program everything and just can focus on your business case and don't have to do the Java basics. And it's the most common Java framework right now. So you can add the starters to this. That's third-party libraries that bring certain functionality. Like, obviously, we make an API, so we have to secure it. So we need a starter for security that does authorization and authentication stuff out of the box, so you can just add little snippets into your code, and Spring Boot will actually have this all programmed out. We want to do an API, so we need a web starter to run a web server. Actuator can give you runtime metrics and information about the health of your application. I think this one is absolutely necessary if you want to monitor your application later on. And then we have a test scope, so this will not actually run in the production compilation of the software, where we can test the actual Spring Boot framework and also the Spring Security that comes with Spring Boot Security Starter. And you don't have to necessarily understand all the parts of this and become a Java developer to actually use this. But I think every language that has a dependency management framework can do stuff like this. And obviously, we added the vulnerable Apache Strats version, so we can find something that we actually want to fix or whitelist, maybe. OK. And the problem is right now that these six or seven dependencies bloat the software. You see them on the left side, and they bloat the software. And they download like 80 more dependencies themselves. And they call transient dependencies. So every dependency brings more libraries. And they bring more libraries and so on. So it's a big bloated library. And well, I wanted to say fuck up, but it's depending on what you're actually using. Spring Boot is very, very well maintained, and they have a quite quick reaction time if they find something. OK. I've just set this six dependencies. You can find this on GitHub. And I will tell you something more about it later on. And almost 70 transitive dependencies. So it's like 80 overall. And they can all have vulnerabilities. And you want to at least patch the ones that are publicly known for having security issues. So it's like low hanging fruit. And everyone can find this. And you should too. So what can we do to find vulnerability dependencies? Or third party libraries? There's tooling for this. And this is a really big hype right now in the software industry. There's quite a lot of vendors who add products for this. GitHub, I'm not big on advertising for paid services. But GitHub does this for open source public software. So if you have created GitHub repository, as the owner, you will see when GitHub finds a vulnerability in your dependency system. If you use one of the big programming frameworks, and they will show you an alert for this. And if you set the correct notification settings, you will also get an email for this. This is pretty nice. And a couple of months ago, they added a system that can also patch your vulnerabilities or patch your third party libraries. And well, you might think if the owner sees one of the problems, that's OK, because it's still hidden. Nobody else sees it. But if someone forks your repository, they are the owner of the fork. So they will get a notification if GitHub finds the same vulnerability in their repo. For the demo and for this whole talk, I'm using dependency check because it's an open source tool in the open web application security project. So if you contribute or if you donate money to the OWAS project, some of it will allow the maintainers of dependency check to make the software better. And it's really well maintained. They react quickly if you ask them something or if you set up a fuel request or something like that. OK, how does it work? There's this database run by an American government organization publicly, where you can get the CVE data. So that's the database where the actual vulnerabilities and their descriptions and scoring are held. And dependency check downloads this into a local database. And then you can run a dependency check in Maven. You can also use Docker if you're not using Maven. You can use a command line client to check your software. I think for Maven or Gradle, that's the same for Java software. It's really easy to use it this way, but it's up to you. In the end, you should check this in any way. It's not necessarily the best one. OK, let's just see how this works out. OK, so for the guys who know Java and Maven, I have programmed a Maven goal for this. And let's just run this check and see what it tells us. And it takes a couple seconds. In the demo repo, I have disabled the automatic update. So if you run this, if you check this out and if you haven't never used dependency check before, you will be told to update your database beforehand. So some vulnerabilities were found. So a build broke. So this will end in an exit code one. So anything different from zero is bad. And here you can see the vulnerabilities that were actually found. So we have Jackson data bind. This is a can you read this in the back? Thumbs up? OK. It's a JSON parser that you can use for FOIA application. It's pretty well known. And this is one of the transient dependencies. I have not actually chosen to use this, but it comes with one of the spring boot dependencies. Then there's spring security. And actually, this is an actual vulnerability. There is a patch for this in 2991, hotfix. But the spring boot version that actually brings this dependency has not yet been upgraded for this. So this is a, if you have a process for fixing this, you could manually upgrade this to 2991, for example. And then this should work out. If we have time later on, or if you join my workshop tomorrow, we can play around with this. There's spring security core. This is actually a false positive. So spring security core has had a problem in 50 something. And they have fixed it. But somehow, this is still in the vulnerability database. So the vulnerability database will still have the information that this is actually problematic. And this is for authentication stuff and authorization. So if this was still an issue and you're using authorization annotations in Java, you should probably upgrade this. And then there's our good struts vulnerability, which has like 10 vulnerabilities. And it's really old. It's from 2017. And the comments file upload is just another one that comes with struts, so we won't care about this. What's the cool thing about dependency check? I forgot. The app itself does nothing. It doesn't even Hello World. It's just this is the basic what you get if you use these dependencies. You get a web server. You get a login screen because we use security. And you could never log in because you have nowhere to find any user data. OK, dependency check not also gives you the command line references, but also generates an HTML report that you can use for other stuff. And if you like, search for the struts problem. Whoops, there you go. You find a description what's the actual problem here. This one is passed from the CVE about struts. And there's some more information about what happens here. And then they have a list of all the vulnerabilities found. And there's a very good description what's actually happening here and some scoring. And this is the point where you can actually learn just by using this dependency check and checking out the HTML report. You will find these vulnerable third party libraries. And the report will also tell you what could potentially happen here. And sometimes there is also a link to exploit DB. So exploit, now, this is the big one, the critical 10.0 PASA thing that I mentioned earlier. And you can also find working exploits that you can use to test, please, only your own servers, nobody else's. And usually, this is a very good way to learn about vulnerabilities if you find them. If you have no idea how big the problem is, you look in the report for exploit DB links or check some of the other descriptions. And some of them are very, very helpful in understanding this stuff. One more thing. The report also has a prepared XML statement that you can use to whitelist false positives. Oops. Spring security. There you go. So we want to whitelist Spring Security Core because we have read reports that this is actually false positive. So we go to the suppression button. And if the CSS is working, we get some XML data that we can use for a configuration file. And we can just copy this and add this to the, I think I've opened it before, to a suppressions file that we have configured in our Java file. And now this CVE that I have suppressed here. So this one will no longer show up because the dependency check will now whitelist it because I've configured this XML file to before whitelisting. Or somehow I miss something that, OK. I did an upgrade of the dependency check plugin earlier to prepare for the talk. And apparently something in the XML description changed. So you just have to trust me that this would actually have whitelisted the false positive. OK, let's go on with some more fun stuff. OK, you can also use the report to run some other reporting or to use it in other reporting tools. Like this is Sonar, a code quality tool that shows you trends and stuff. And we don't have time for this. So Docker, who in here has actually scanned their Docker containers for problems? Three people. Very good. Docker is, well, you use Docker so you know how it works. You pull some container that someone on the internet has built for you. And this is like the YOLO style of running software. And there's also a scanner. It's called Clare. Has been written by CoreOS, now owned by Red Hat, now owned by IBM, I think. They do container stuff and Kubernetes stuff. And so apparently they needed some container scanning utilities. And it works similarly. They also query the NIST database. But they host the vulnerability data in another database on your local servers. And they run a Clare server. And you can run a client that actually feeds your Docker container to the server. And the server will scan all the Docker layers and tell you what's happening. OK. So I have started a Clare server and a database on my machine. Usually you would have to run these servers in your infrastructure somewhere. And I have packaged this app that we wrote earlier into a container. And I have used a basic Java image for Java 8. It's one of the official images from OpenJDK. And let's see what happens to this. So this takes a couple of seconds. So we're pretty late. So we'll just go to the, you see a lot of red. And earlier I ran this test. And it found that the image contained 96 vulnerabilities. And they are all already in the base image. So this has none of the vulnerabilities that we found in the Java application. This is everything you get when you download a new OpenJDK package. And this is the slim one. So it's not as bloated. And there's fun stuff like G-Lib C, obviously. And UT Linux. And you can actually also whitelist false positives. Or you can also whitelist stuff that you know that are not applying to your software. Like if you have a problem with the SSH daemon, someone could probably break into your server. But in Docker, you usually don't open the port. So it should not be so hard to just not scan for it anymore. And then you can also set thresholds. I will not show the examples now, because I'm a little behind. So you could choose only to see high or critical or medium vulnerabilities and upwards. And everything else is auto-approved. And to run this in continuous integration, you might have to add some more fun stuff. Typically, if you run stuff in modern build tools like GitLab CI, GitHub automated tools, you won't necessarily have the power or the time to set up infrastructure. And you don't need it. So you maybe don't want to run a cloud server that you have to care for all the time. You don't want to manage another Postgres database. So what you can do is, inside continuous integration, just before the scan, the actual scan happens, you can start a database container and a clear scanner container. And this guy, Armin C something, he has a French-sounding name, and I can pronounce it. You can find his repos on GitHub. He actually came up with this solution. So just before you run the scan, you start those Docker containers. And after the scan, you just stop them again. And I have obviously just started them on my machine. So after the scan, I would just stop them. The demo repo contains a Jenkins file. So if you use Jenkins, you can also run all these tests. You might have to play around a little in Jenkins with the credentials of the Jenkins user and the permissions that they have. There's also public repositories that do the scan for you, like Quay is an alternative to Docker Hub. So you can choose to host your repositories with Quay, and they will do a scan for you. GitHub or GitLab offers a method to scan your container for you and give you a hint about broken stuff. OK, and I think we have some more time to do a quick API scan. As I mentioned earlier, I've started the application on my local machine. And I could now run a Zap mode of time. I'm so sorry. OK, so let's start over. Sorry. We can use another OWASP tool. It's called ZAttackProxy to actually attack our API and dynamically scan it to find actual vulnerabilities that it's serving, like stuff like that that offers SQL injection or cross-site scripting. And what we do here is just use Docker again, because we have great Docker and container fans. So we can download a Docker container that they provide that actually has the scanner built in. And then you can just run it against your website. And I will just show you the report that is actually coming out of this. And I've taken the liberty a couple of weeks ago to scan the Synics website. So please don't hack us. And the scanner shows, and this is very, very cool about this tool, also a great description about the actual vulnerabilities, because I have no idea what the xFrame method option header is. Does any of you? Not the two people. Very good. But it actually is, I think in the solution, there is always a description what the actual problem here is and what you can do to fix it. And there's also references where this is where there's a description for this and where you can find more options what you can do. I will run one more attack, I think. Yeah, this also takes some time. So I've already done this before for you. And if you run this against the Synics site, nowadays we run WordPress. So you will find a lot of warnings that might or might not be critical. We have a company that does the page for us. So maybe we should address them with this. And then again, this is just a spider. So this one, this attack will go to, I think, I configured Zoonix DE somewhere there. And this will start at a landing page that you configure at a target. And it will just spider all the links inside this page and just traverse into deeper into the page you give it. You can also run this and feed it an open API spec. So if you develop your own application, you usually know which API endpoints you are serving. You might generate this using tools like Swagger, if you know this. But let's say you have three endpoints like Login or, I don't know, Shop and Web Page, something like that. You might tell the tool which API endpoints or which URLs to look for. So you save the time that it would use to spider all these links. And the longer it runs, obviously, the more it's able to find. OK, I'm a little sorry that I had to rush through the end. If you want to try this more in depth, Daniel and I are hosting a two-hour workshop about this, where we will show you more the usage of the tools. And you can ask questions and might just get your hands dirty and try this yourself. Tomorrow, I think it's 3.15 PM. It has been rescheduled for a couple of times, so you should look into the 300 monkeys' schedule, I guess. If you can make it, you can also stop by our village, Falcon Business World, and ask your questions. And, well, usually I would discuss this in the end. This is about the process. What can you do? Maybe one sentence about this. In my current team, if a vulnerability is found, the creator of the current pull request is also responsible to fix this as soon as possible. We have a zero vulnerability policy, so anything that is found has to be fixed immediately if we found something on our production code. We will also have a job that scans this nightly. And the first one to be in the office, or the one on call, also is responsible to at least look into the issue. So they could find out it's a false positive. They might just suppress it. They could find out that it's a horrible, horrible, bad possibility to bring your whole enterprise down, so they might want to patch it immediately. But this is mainly up to you, and I think, as I mentioned earlier, this is the most interesting part for a company and a development team. Thank you. All right, we have a couple of minutes left for questions. So if you want to ask a question, please come to one of the microphones in the front and in the back, and ask your question. All right, I don't see any questions at the moment. So if you, oh, well, there is one. Please go ahead. Hi. Thank you for the talk. My question is about handling false positives. OK. You already showed an example, and especially dealing with something like ZEP, you get a lot of false positives. But sometimes, doors are pretty hard, like stuff in libraries which you actually don't use. So if you use APIs that are not vulnerable, and I mean, talking to development teams, I know that they spend most of their time handling exceptions. Do you have some general recommendation, or could you share some experience regarding that? Yeah. OK. Especially in dynamic scanning with URLs or APIs or web service, this is like really time consuming. I can maybe share an example that I've seen in a team of a friend of mine. They run API tests every night. And when they started this, they just sat down for a couple of days, actually, because they have a very big application that has grown over the years with a big, big API. And they scanned this with the Open API specification that they had generated earlier. They scanned their app, and then they actually sat down and checked every vulnerability that ZEPs bat out. And I think that's the only way to do it. You have to take the time and get all of those. And sorry, if you're at a point where you have zero new vulnerabilities or zero vulnerabilities found that you think are worth fixing, then you can do, for example, a weekly task or a nightly build that checks your API. And if something new is found, you have to fix it according to your policy. But I don't think there's a good way to do this differently. Well, you can always, if you have libraries that are vulnerable and that you don't really need, I'm the biggest fan of deleting your code if you don't need it or deleting or black-holding APIs that you don't need. So I think if you're new to this, you should always maybe talk to your security team or if you're not sure to the networking team or if you're the networking team yourself, you should probably think about your firewall rules and stuff like that. But I think you have to address every vulnerability that is found. That's painful. All right, then maybe let's do one last question from the back. OK, thank you for your talk. Do you have any experience with Graal VM or Quarkus? Because if you have all this bloat in your containers, it's easy to get confused. And to run the bare necessities, it could be interesting to have experiences with more or less bare metal or bare code containers without any operating system. Well, I think it's a general rule to use as little software to run your code on as you do. It's also one of the Docker best practices, I think. In Docker, you can use the official, like in Java, you can use the open JDK official alpine images. And if you download them right now, they won't come with any vulnerabilities. A little problem with the scanners that I have used is that stuff like Graal VM doesn't have a container, or it's stuff that Google provides if you use JIP if you know this one. It will dynamically create containers. And they are not really containers in the Docker sense, so the scanner can actually scan them. I think these are projects that you have to rely on the maintainers to find vulnerabilities and fix it. But it's also more easily to handle it, because you can just upgrade your running systems, like Nightly or something like that, or every week, every hour. All right. We're out of time. So thanks so much, Sai, for the talk.