 I'd like to introduce you our next speaker. She's a teacher of, she teaches developers about application security. She has learned over 20 programming languages. I told her like, she's like a developer polyglot, right? She is a founder of GoldHead Security and now teaches Upsack full time. And she's going to talk about vulnerabilities that hide from your tools. Give a round of applause to Jillian Redleaf. Hi, can you guys hear me? Okay, is this one on? Okay, good. I don't see a volume button, but you can hear me okay in the back. Alright, well thanks for coming. I'm really excited that you're here. And I'm also feeling a little vulnerable along the lines of the talk. So this is my first time speaking in DEF CON and I'm a little nervous, but I'm here to help you and help hopefully make your jobs easier. So thanks for coming. I'm going to start out with, oh, I introduced myself first. I'm going to start out with me. This is a friend of mine that I made in Bali and me, a baby elephant. I started my own company this year because I wanted to teach full time. So I teach application security to developers. And I have over a decade of AppSec experience now. Just by a little bit. And I love to travel and tell jokes. They are not always good jokes, but I will tell them anyway. So nobody brought tomatoes, I'm sure. So I'll give it a try. So first, sorry. Alright, just a quick history lesson. So we know kind of the history of where the tools that we used today came from. So back before I was born, there was the very first static analysis tool called Lint. Now Lint was basically a syntax checker, right? So it didn't check for any security vulnerabilities like SQL injection or cross-site scripting because they didn't exist yet. So they came into existence in the late 90s. And then we started to have some dynamic testing tools. So Nessus came into existence. And I don't know if any of you remember back in the day when you had to use a modem to get connected to the internet. Back then, websites were just pretty much static HTML. And then along came this little site called MySpace. Now for those of you who are kids in this room, MySpace was like Facebook only before Facebook. And it had all these fun features where you could customize your background and put custom JavaScript code if you are the loneliest person in the world to write a worm, make everybody your friend. So this, I'm not blaming MySpace for the state of web application security today, but I did notice there is a proliferation of security tools that came up after this point in time. So we've got Coverity, Fortify, Metasploit, one of my favorites, Burpsuite, another favorite. And then we have open source tools coming into existence to check to make sure that your dependencies are not vulnerable. So things like Blackduck. One note about this, most of the open source tools that you're going to use to scan your open source code are only going to find known vulnerabilities. They don't necessarily find the zero days. So just word of warning there. And then that's more tools, ecunetics, checkmarks, and veracode are sponsors. Made sure I got them on the slide. OASP-Zap came into existence. WhiteSource, another sponsor, thank you. And Snick, I may be saying that wrong. That's one of those things I've never heard said out loud. But also of note, and I'm not going to talk a lot about this because I haven't actually used these tools, but there is in recent years something called I asked. The I stands for interactive. So it's interactive application security testing. It's another option. Since I have not actually used them myself, I can't speak to it much. And when I googled I asked tools, Google was like, did you mean sast? No, I did not. But thank you for the condescending suggestion. So anyway, so a brief history. Now moving right along, I have a Venn diagram for you because I love Venn diagrams. So there's what they're good at finding. And I have some sast and dast vulnerabilities. So they're both pretty good at finding the OASP top 10. And I like to remind people that it is the top 10. It's not all of the vulnerabilities, just the top ones. So things like PCI compliance. Anybody deals with PCI at work? Yeah, so there's more than 10 vulnerabilities is what I'm saying. So sast tools are somewhat good at finding weak crypto. And I have an asterisk there because it's sort of like, they look for the really obvious things. So are you using DES in your code? It's going to kick out an error message that you're using DES. But it's not going to find implementation flaws. And there was a pretty good talk earlier, which talked a lot about that. So watch the recording or some of you were here. They're pretty good at finding hard coded passwords or keys or things like that because you're scanning the source code. Things like memory leaks that might cause a denial of service. And then dynamic application security testing tools. Find things like poor session management. Missing security headers. Default passwords. And insecure server configurations. So there's a little bit of overlap, but obviously it's good to use both, right? Now you're all here to hear about what they're not good at finding, right? And I have a whole list, not a complete list, but some highlights for you. The first one is business logic flaws. And this can be all sorts of different things. My favorite example though is just a poor password reset logic. So personally I have a grudge against cognitive questions. You know what street did you grow up on? That sort of thing. So weak password reset functionality is something that your tools just aren't going to find, right? And many other things. You could have, if your process for the call center to identify a customer is weak, then they might be vulnerable to social engineering. Things like that you want to keep in mind. So weak or reused passwords. Now fun fact, 75% of internet users admit to reusing passwords across multiple accounts. Now the other 25% don't admit to it, right? So password reuse is a huge problem and that's not something that any one of your tools is going to find, but with all the data breaches lately, credential stuffing is a thing and something that you need to be concerned about. So just keep that in mind. Next up, configuration whoopsies. There was a very recent data breach in the news in the last week or so that was caused by a web application firewall having too high of permissions. Now I don't know of any tool that's going to find that. It might be out there, but that's one of those things that it was just not using the principle of least privilege that led to a huge data breach. So, are you paranoid yet? I have more, okay. Next up, denial of service vulnerabilities. Now with an asterisk. So there are some denial of service type of vulnerabilities that your static analysis and dynamic analysis tools will find. Dynamic analysis tools tend to find them accidentally. Has anybody experienced that? You run a scan and denial of service, yeah. So, but that's not maybe the best methodology for finding these things. So there's all sorts of things that can go wrong. For example, if you have a developer that wrote a very poor SQL query that takes like 17 seconds to run, that's going to bring down your SQL server eventually and then with repeated requests. So anytime you have a server response that takes too long and you can send repeated requests, you can bring down the SQL server and then anything tied to that SQL server will also come down. So that's not something that any tool is going to find, but if you have a good DBA to review all the SQL queries, that will help tremendously. Rogue developers is another one that keeps me awake at night. So, let's see. Oh, so something that you want to consider is that insider threats may actually be coming from nation-state actors. So a lot of times we think about Rogue developers and, oh, well, they're just mad at the company, right? But in fact, if you have a big enough company with valuable enough data, there might be somebody with a Russian accent that applies for a job, say. Not going to pick on the Russians, right? Because I don't want to get hacked. So if there's any Russians in here, you're great, I promise. Okay, anyway. So next up, cryptography disasters. So a very good talk about that earlier. There's so many things that can go wrong with cryptography. So if you're not comfortable with cryptography, you definitely want to have somebody help you out with that. But basically, never write your own crypto. That is rule number one of Crypto Club is never write your own crypto. And rule number two of Crypto Club is never write your own crypto. And even if you're using a standard encryption algorithm, like AES, if you use ECB mode, for example, then your crypto is insecure. So lots of things can go wrong there. Tools may or may not necessarily find them, right? And then good old layer eight vulnerabilities. Things like phishing, social engineering, or your standard ID10T, or... These are all things that are very much... They can affect the application security of... They can affect the security of your application. That's what I was trying to say. So just lots of things can go wrong there. And then secrets stored in weird places. So my favorite and most trendy example of this lately is a JWT payload. So you have a JWT, a JSON web token, and perhaps you have a developer that sees it in the developer tools, oh look, it's obfuscated, it must be encrypted. It's not, it's base64 encoded, and anything you put in there should not be a secret. So that is something that just have to do some manual testing to find, I guess. Things like text files. So I have seen a text file with usernames and passwords in it that was publicly available on the internet. Not great. Config files, sticky notes, still happens, et cetera. So false negatives, when your code is written in other spoken languages. So this is something that may not seem obvious to most, but for static code analysis tools, some of the rules will look for variable names. So if you are looking for, say, a variable called password, but your coder is speaking Spanish, then it's not going to find that. So that's something that I've actually had happen a lot. So you get the scan results back in, and it says all these, it's highlighting all of these password variables. What do I need to do? Should I change the variable name? So you know, you should remove the password from your code. So, sensitive information and error messages, it's fairly self-explanatory. I don't have an example for that. You know what that means. So insecure APIs, there's still a lot that can go wrong with writing a new API. So it's most of the dynamic application security testing tools have a harder time testing APIs without complete documentation. So you can't just spider an API and figure out everything that's there. And even though we've taught developers to tell people, oh, username and password. So when you have an invalid password, you give them an error message back, invalid username or password. So that's pretty standard practice nowadays. But let's say you write an API that checks if a username is taken. So now all of a sudden, the attackers are able to enumerate usernames. So it's a way around that. It was just a very simple example I had. But next up, third-party integration flaws. So let's say, for example, you are integrating with an identity provider. And the identity provider is configured to allow for open redirects. So what can happen in that case is an attacker can use phishing to have a user enter the username and password on the valid identity provider site, and then it redirects to the malicious site with a token, a valid token. So open redirects are dangerous there. There's many other things that can go wrong. Grab a drink of water, hold on. You with me so far? Everybody still awake? Okay. So how to find them? There's the hard way, which I'm sure most of you are very familiar with by now. So, AppSecPin testing, or just eyeballs on code, code review. And I had a very hard time finding a GIF that would appropriately represent code review. I hope you like this one. I spent a long time just reviewing code, and that was my face most of the time. But I have some suggestions to make your life easier. So the easier way, I call this the Lord Varys approach. Now, for those of you who have not seen Game of Thrones, it's okay, I will explain. So Lord Varys was the master of information in Westeros, and his technique for gathering information was he had all of these spies around the city and around the country, really, that were children. Now, I'm not advocating for child labor. Please don't do that. But what you should do is just use the same approach to gather information within your own organization. So you can have an internal bug bounty, and even if you're not ready for an external bug bounty, you can reward your employees to come to you with vulnerabilities. So the idea here is just to make it as painless as possible to report security vulnerabilities to the security team. Now, a lot of us maybe have a reputation for being difficult to work with, highly opinionated. It's okay, it comes with knowing what you're talking about. But just the point here is that you want to reward people for coming to you and give them props. So, yeah. It's my favorite method for gathering information. No child labor, though, right? Okay. Next up is threat modeling, and I'm not going to break down threat modeling because there is another talk tomorrow that you can come and see at 1330. So highly recommend coming to see the introduction to threat modeling, but for those of you that are in here, I'm going to assume that you're already familiar with the process, right? So, I have a tip for you as you're doing threat modeling that if you can really master this principle, it'll change your life. I hope. So, when you're doing threat modeling and you gather everybody in a room, say, and you're asking questions about the architecture diagram, and it's like this and that, this will change your life. The art of asking an open-ended question, okay? So, it's a very simple tweak, but it'll make a huge difference. So, instead of is this connection encrypted? Every developer you talk to is going to say yes to that question. Of course it is. Everything's encrypted. So, instead, ask how is this encrypted and where do you store the encryption key? And if they don't immediately know the answer to that, then that's something that you want to dig deeper into, right? So, that's my big tip for threat modeling. So, there's stride and dread and all the other things that spell six letter words. But, this is what you really want to master. I have another example for you. So, instead of, do you have any sensitive data in this database? You ask, well, what kind of data is in this database? And then they have to answer this question, and sometimes developers may not know what's sensitive and what is, what's not. So, instead of asking them to decide, just give them, just get the schema and review it yourself. That's probably the best method. So, I have another tip. Just the, what I call the full circle development life cycle. So, using feedback from all of these methods. So, you have your tools and you've got pen testing and threat modeling and code review and maybe an internal bug bounty. So, what you want to do is you want to find the bugs. And I know how many of you have experienced a situation where you stop there. And that's the only step. It's just finding the bugs. One person admits to it. You're brave, sir. Thank you. So, the next step is to fix the bug. Right? Seems obvious. Doesn't always happen. And then, what you want to do from there is just integrate that back into your automated test. Right? So, write a new test case that has to do with that particular vulnerability. And then, you write some new code or the developers do. You deploy the new code and you find new bugs. Rents and repeat. So, and then preventative care, of course. Just preventing vulnerabilities from getting there in the first place. Now, there's a few things that you can do. The number one is education. So, that's why I love that you asked that question. How many of you are training your developers? So, it's a really important thing to do. That's why I do it. I'm very passionate. I might be a little bit biased, but I really think that if you train your developers in security, then they're going to have the knowledge to just keep from, you know, writing the vulnerabilities in the code in the first place. And let's see. So, the next recommendation is using secure default configurations. So, for those of you using containers and AWS, just don't let your developers shoot themselves in the foot, right? So, just have secure defaults in a template so that that is set up and it makes it easy for them, right? Configuration management. So, with all of the tools that you can use to run the scans, and then I would really highly recommend setting up some automation to basically fix what you can using microservice... microservices. And this one's really pretty important with the nation-state actors nowadays, but just a comprehensive background check for your employees. So, a lot of times it just... the background check makes sure that they don't have a criminal record, but there might be a little bit more to it. So, I don't know if you have a good rapport with your HR department, but I highly recommend that. I've clearly been talking way too fast, because I'm out of slides, but I will take questions. So, does anybody have any questions? Or certainly, there's probably things that I've missed, but yes, that is a fantastic question. So, for those that didn't hear the question, was how do you do a bug bounty without encouraging developers to write bugs? That's a great question. You... I mean, you do have access to the check-in history to see who wrote what code. No, that's a fantastic question. Well, peer review helps a lot, too. So, it perhaps just changing the process so that when you are doing a peer review, the developer does a peer review and they find a bug, that's the only time that you can report a bug, but then there's a potential for hiding bugs. So, it's a really excellent question. I will think about that. I'll come back to you. So, I have done that, too. Yes, I've definitely looked at old code and I'm like, oh, that's embarrassing. So, knowledge is just a... Knowledge is power. So, but that's why you're here. And I'm glad that you're here, yes. Okay. Oh, an internal education program for the developers. First step, it... Well, I've done that myself. So, the first step was creating about 350 slides. And then the next step is talking to the executives in charge of the developers and getting their buy-in. So, that's really the most important part there, because if they don't think they have to go, they won't go, right? It's not optional. You have to go to training. So, getting executive buy-in. Actually, no, that should be the first step before you waste your time on the 350 slides. Yeah. That's a great question. How do you evaluate competency? I'm still talking too fast, because I'm nervous. After training. So, what I've done in the past is have a quiz before and after to see the effectiveness of your training program. So, yeah, just compare results. It'll go up. And then, another thing that I would recommend for an internal training program is keep it in the developer's face all the time. So, not like an annual training, but to have something at least once a month so that it's in their face, they're always thinking about it. And they're always learning more about security. So, some of my favorite metrics to track for what? The number of vulnerabilities is probably the most important one. So, take all the data from all the tools and put them in the same place, and hopefully that number slowly goes down. So, that's probably the most effective way of tracking how effective your AppSec program is, I think. Does that number always go down? No. Because the developers are writing more code. It does not. So, even if you could stop, there's more code. So, if 10% of the code has bugs, you're going to get more and more bugs in the list. Absolutely. So, you could track vulnerability density as well. So, and at my last job, I actually had a competition between the development groups. So, it was just a scorecard that everybody knew where it was. And you take the vulnerability density and whoever was at the top of the list got a prize for the month. And they got really competitive about it. So, that's combining social engineering with AppSec to manipulate people into fixing the bugs. So, yeah. I like this Q&A session. I'm really, I talk so fast. Oh my gosh. I'm going to have to watch the recording and laugh at myself, but... Yeah, anything else? Definitely. Well, yeah, you do have to kind of start with the basics. So, OOSP top 10, as I say, or just the top 10, that's kind of the bare minimum for web developers. But I definitely would advocate for training in more in-depth and especially just whatever somebody needs to know to do their job. So, AWS is one that's being very widely used and there's so many features and there's so much to learn that, yeah, it's still pretty easy to screw up even if you know what you're doing. So, I definitely advocate for role-based training. And, yeah. That's a good question. I see two hands. I saw you first. Oh, that's how to scale finding business logic flaws. Is that your question? Well, it's probably a slow process, but once you find one, then just write new automated test cases to test the new and existing code. But I'm sorry, I'm not sure if I understood all of your question. Oh, to find them? Learn the business. Learn the ins and outs of it. So, as a security person, you should know more about the business than your boss does. It is. It's very difficult to find, but the more you learn about the business and how it works, the more you realize that you'll find things. That's a great question. I have seen, so in design and story development, have a review process where there's a questionnaire. So, when you write a story, there's also a security questionnaire. So, it's a fairly basic way of doing things, but you can scale things and just by having them ask a few questions. Are you dealing with social security numbers in this story? No. Are you dealing with cryptography? Yes. And so, the things that are kind of the low-hanging fruit will automatically get kicked up to the security team for a review. So, that way you don't have to review thousands upon thousands of tickets. So, does that help? Okay. That is a great question. And if you're in an organization where you're working against reputation, I think the best thing you can do is just... Have you read How to Win Friends and Influence People? Read it again. I read it every year, you know. It's a good review, but it's hard when you're already working at a deficit, but to just try to change the perception of the security team. So, you know, do nice things for them. See, and one recommendation is every time you have a meeting with a developer, just start it out with, I'm here to help. Let me know how I can help you. And then just kind of live with that attitude, I guess, and it'll get there eventually. So... But I've dealt with that a lot of... Especially when a brand-new developer comes in, it's, oh no, you're the security team. You're going to tell me I'm doing things wrong. But you'll win them over eventually. Patience. I know it's hard, but... Yeah. Yeah. Sorry. Yeah. I think you have to understand the principles before you can really exploit them or fix them. But, yeah, for sure. So, I love hands-on training, though. So, because that's the best way to learn by doing. That's how I've learned all of this. I didn't go to school for this. So, good question. That's a big bug. Yeah. Yeah. Okay. Well, I wanted to say thank you to Owen and Adarsh and Vivi for helping me with the writing process of this, and I'm so distracted by the bug. I'm sorry. Do we have any bug hunters in here? Five dollars. Yeah. And then, let's give a big round of applause for all the volunteers and the people who organized this. I also want to thank you for being here and asking such good questions and for all of your hard work in what you do. So, app sex is really important. I'm very passionate about this subject. And I'll be hanging out outside if you want to come talk to me later. But, yeah. Thank you. Thank you for being here.