 How's everybody doing? Happy a good afternoon Was was lunch good you made it back You may have just in time. So the regular announcements, of course Jeopardy tonight Wi-Fi is probably still up at least it was up half an hour ago And there's still lockpicking going on outside in the tents if you want to have a conversation feel free to scoot out into the garden and Remain in here because This is a special occasion Arvind is here for his first talk at a conference for all of you So He does apps application security at doc you sign right he started out in development But moved into the security space after getting a graduate degree So he's got a motivation for some developer friendly things in security And basically taking ways of taking security processes integrating them into the development processes and so right in line with that He's going to present for you static code analysis should be for developers not for you. Thanks Arvind Thank you everyone from coming in it's right up to lunch. I'm pretty sure everybody is super attentive and Thank you to account for giving me this opportunity. This is my first talk in a conference So yeah, I'm excited So today I'll be presenting on something. I'm really passionate about and I've been working on this for past year making security developer friendly as a whole team but in this one mainly making static code analysis work for developers If you don't know what's a static code analysis It's pretty simple you have your code in version control and you use a bunch of tools to scan it to find some obvious box That's somebody might developer might have missed before we go forward a simple warning that all the opinions expressed and the tools I call out are my opinions and Not of docusign in any way About me I go by RV. So it's easy to call and also my Currently I'm an app second junior at docusign. I've been an app second junior at Nike and I was before that I was a software engineer at VMware. I have a master's degree from computer science in computer science at from ASU a special thanks to John Heisman and Skyler Berg who also worked on this project and their Guidance and work has been incredible without that. This could not have been done So today's agenda is pretty simple I'll go through what's what I think is the problem with current static code analysis tools and what was our solution? What did we build and What did I learn in the process and some of the features that I think are important and finally? I'll end with some unsolicited advice as usual, which is more of a call to action, but anyway So problem statement. I'm a I'm a fan of metaphors. So I it's easy to grasp the problem through a metaphor So let's do one That's one and if you're not familiar with it, which I assume most of the room is This is from Hindu mythology Ramayana and the situation is one of the protagonist is hurt in an epic battle and the healer on the ground says Hey, we can fix him But we need a specific medicinal plant called Sanjeevani, which only grows in Himalayas and Since it's far they asked the monkey god Hanuman to bring it Hanuman goes to Himalayas finds the mountain but cannot identify the specific medicinal plant so instead of bringing that he brings whole mountain and Keeps it on the ground so that the healers can find it and Heal the protagonist this is kind of what this static analysis tool does we need specific fix Problems to fix our code base instead of that it brings whole mountain and just keeps it before us The expectation it has is someone in some of the security person knows what to focus on and what to ignore But more often than not it's gonna be a developer who's gonna be looking at it And they have their own mountain to climb and if you keep one more, it's just an obstacle They're gonna avoid to put into words The problem problems are they're monolithic tools trying to scan Every language and every framework that's available. So it's it's gonna be generic It's a huge task to be honest and also it's built for security professional in the sense they have so much information and You know held for security professional with the CVC vs. All things are great But a developer needs to know how to fix it and why not fixing it as a problem That's all they need to know and since they're casting a wide net to catch everything possible in every place It's there are a lot of things that are not bugs. They are mostly You know false positives and if you keep getting in the habit of saying oh ignore that it's a false positive They're gonna come to you three times later. They start ignoring without asking you and It's it's great works great as a silo as its own animal. You can scan whenever it's comfortable for you. It's quick It's it's works great for you, but it's not integrated with the existing engineering, you know systems Some vendors are really trying to get there and they give option, but that's not its primary purpose So it's it seems kind of janky if you try to do that to make my case I borrowed a slide from our friend my friend Clean Gibbler who recently presented in shell con about rolling your own sass tool And he makes an interesting case that when should you buy a sass tool and his observations are Let's say if you have modern frameworks Or if you have if you want to ship it quickly through CI city platform Or if you have invested time in building some safe libraries like we did which are highly recommend Then it might not be for you buying one might not give the best results And also he makes an interesting case that even if you buy an off-the-shelf tool You might end up writing a lot of automation code to work with your system and make it You know it's in black box that you don't know what's happening inside You have to write something else to make it work for you So inevitably like it's not more of an option It's if you want to get some security value out of it We and you have you tend to write your own tool and that's what we did But we did not write another scanner another tool. We wrote a platform And because there are already so many scanners and you know, we have the code and also Grepp is amazing. So we don't have to keep writing new tools again and again. There's so many you know open source one we can use and So we can we used a plug-in model so you can plug in any tool within two to three hours with little coding and It's built for developers Whenever they need and what's the best effective method to give the feedback to them We use that we consult with them and it's not a one one time process. It's a in evolving process where you keep Making it better and better Improving something if something is not working out for you. So it's trained by our app sec team, which I have to thank I'll go through the life cycle of it. I'll go from both developer and app sec perspective it's important to think from developer perspective because if there's a less friction to use a tool or you want So you want them to use something the more adaptability it has if you if you rule with the Handfest and saying no you got to use this they might but they try to ignore you because it if it doesn't fit in your existing system So from a developer perspective everything starts at their source control and we start the process when a new PR is put up Because that's the best place to do it. I'll go into detail later But yeah, once it starts The request comes to CI CD pipeline and the CI CD pipeline Works for us because the teams CI CD team is our best ally instead of you know Going to every team and asking them to involve us or just adding it without telling anyone We we work with CI CD team to make it part of a pipeline. So anyone gets a new pipeline our tool comes with With it. So once CI CD sends a request from a developer perspective our tool our platform scans Scans the code and sends back the relevant result and one more thing we do do it here is we work with the team to To understand what works best for them. What how do they want to see this result? Not every team You know once in the same way if their system is different. They have their own process. So we work with them there as well We also create some security tickets to you know Follow it up or just to have a trail for it But the whole system is supposed to be seamless. So this these tickets are not for developers to look at and close and everything It's for us. Let's say if they fix the Any problem we found in for in next commit then this ticket is automatically closed if not tickets remains and one of our App sec engineers will go in and they open a bug so that they can come back and fix it So trying to do as much as booking bookkeeping for them and just helping them to fix the bug is one of the key for success And from our perspective when our when we get up when we get the request from CI CD pipeline we do multiple things but one of the main things we do is Cloning the cloning the code in our local storage and mounting it to our containers This is the scan important and we scan the files that the PR is touching This is important because I have seen tools which work at the PR level when the PR is put up they again scan the whole You know whole application which I think is a squandered opportunity because you have you have Developers at the right moment when they're fixed working on some particular file So use that opportunity to fix, you know have them fix on that one and Then we'll send it to our containers and for most of you It's pretty obvious why we why we would you know container is all our tools we use multiple scanning tools to scan our code and It's a hard process if we have to keep installing it in every box We're gonna put our platform in it's a it's a lot of work and also Tools tend to behave differently based on what platform you're running it on what OS is it? So containerizing it made us easy And also it's important that we're not gonna use all the scanning tools all the all the tools we have on every PR Because it's not required we were Deciding which tools to use based on initially based on only on the language. Okay, we are scanning this language So we're gonna use these tools, but we realized well some repositories even though they have the same language They have different environment in the sense they might have some safe libraries or they You know, it's a internal one and we do not have to put some tools toolings in there So we added repository as one of the criteria then we realized even inside the repositories some folders and files Might need some additional tooling or some exceptions So we added that like the changes small changes like that make Can make your false positive rate come lower and lower and also Brings the experience better and better for For the developers if if you keep saying how this tool will generate some You know some bugs for you, but it doesn't apply to you just go ahead at some point It might so just making the system work for them is the best and Once everything is done all the details are all the findings are sent back to our platform here We do multiple things one of the important things we do is Filtering of it. So initially when we started what what I did was like if you're touching a file And if you find any finding in that file, we're gonna assign or we're gonna show it to you But we quickly got a pushback a fair foot pushback that hey I'm just changing some part of it code and other things. I'm not even sure I what it is So I'm not responsible for fixing it and also It's not a good practice to you know bunch all the unrelated changes in one PR So that's a fair pushback and we agreed with that who smelted Delta's not a good idea for in these situations and Of course, we write everything to date our database collect as much as data is possible Why did it was even if we currently don't have you know use cases for that at some point? We will and having this data is what lets you decide which tools are useful Which are not and to get some executive buy-in as well I'll go into detail on some of the features we have One of the important thing is giving giving feedback at the CI CD in the code review stage when they put up a PR The developers and engineers are open to you know reviews at that point. They are looking for feedback They want to fix it at that time. So it's better to give at that stage and also It's easier and much cheaper to fix at that stage if you run the tool whenever it's convenient for you and give them a Large amount of findings to fix fix it It's hard for them because they are working on something else the timing not fit and Touching some code is not ideal at that stage. So when they are fixing it That's the best place to give it to them and we have three ways of alerting one Blocking them. Let's say you find something in their PR and they have to fix it unless that their PR is blocked This is more of where we want to get to we for some rep for some teams we do it but For many we don't yet because either They're bigger team and they're faster and we are It needs a lot of mad power because you need an SLA You need a page of duty to do it and we need more resources to do that To this was a initial idea that I was really excited about that Hey, if you find anything we're gonna comment on that PR This seemed like a great idea and even work with smaller teams But it didn't work for bigger teams because they're already too many people reviewing and commenting and an automated tool Just common keep commenting on your PR Seems like a spam and people tend to ignore automated. You know these comments when real people are saying something there So what we ended up with most teams are alerting without blocking so we alert them saying that hey you failed our check But we don't block you yet. Please fix it and I honestly believe no developer wants to write shit code Like they are good people but they are constrained by different things So more than 50% of the time we have seen people fixing it or just at least asking us Hey, what is this about like how can I? Go about it But if they many some don't if they don't we have a trail of tickets where we can go back and open a bug So that it can be fixed later Now there is tool plug-in model This is important because we had a lot of tools we could use and also I strongly believe smaller tools with specific purposes are much more good at finding issues or classes of issues so that you can You can fix them Because we are okay with the fact that we might not catch everything But the things we catch are important and we are sure about it So smaller tools with specific purposes my jam and also there are many You know open source tools like devs came as my colleague talked about It's a no it's a plug-in built-first visual studio code So it's generic, but since it's open source you can change the You can change the rules it's scanning on and you can cut down the rules to bear minimum of what you want it to look for and Write your own custom rules so that you can scan on them Finding normalization. I mean this is specific to our case because it's you we are using multiple tools and getting multiple details from different kind of findings from different tools and Some tools are amazing at description and other things some tools are not I literally have a tool which the name of the finding was excess and Description was cross-site scripting. I mean I like the simplicity of it, but still Some more description a detailed one which can help developers would be useful Key reasons for success if I have to point out there are two of them one It's built for developers at every stage. We thought What would make you know developers life easy here and how can we help them and it's also? useful in the sense of Once you put it we tested with smaller teams got some buy-in and got their feedback of okay This is not working for us or this specific case doesn't work for us And we get we get ambiguous responses from different teams and it's expected because not everyone's the same and We should be okay with rolling rolling with that What works for them as much you work with them? It's better to better get their buy-in and building a great relationship with your developing team and also Accepting for what it is as static analysis tools are nothing but automation to catch low-hanging fruits They're not there to catch amazing bucks. They're not there to you know replace your pen testers If you put all the amazing buzzwords into it like machine learning you put you know artificial intelligence Blockchain for no reason like if you add all those things. Yeah, it might find some cool bugs But that's not the expectation and it's a lot of engineering work So the expectation is to find some of the You know mistakes that that are easy to make by developers that they might ignore and just Help them out and we should be doing that job. Well One more thing that worked for us is writing safe libraries and scanning for them and alerting people on that My my colleague talked about it if you have a more question on that, please talk to Morgan Roman there and this is not unique you know you Many many companies have already done that Because like I showed in the initial slide from Clint that the bar for writing your own or the necessity that comes It's really low. So you you tend to write your own all the bigger companies have done it talk to Google and You know Netflix a snapchat all of their have their own system because To get some security value out of the existing tools is hard These are some of the resources that I highly recommend rolling your own is Clint's presentation in shellcon and lessons lessons from building study analysis core is from wasp Apsic wasp in California last early this year my boss talked there. It's amazing and Oh, sorry, that's this one the lessons from building at at Google. That's a That's a paper that Google released which I learned amazing things from it's like a gold standard I at least in my mind and I learned a lot of things from there and the last tool like while our tool is We started it as a POC. So it's still not a open source tool. We're working on it It's highly embedded to our this thing our environment But one of the similar tools is Husky CI who steam I met in Defcon. They're amazing They're all the features I mentioned are not there But they are doing amazing work and they're pretty cool to talk to just check it out if you're interested in using one of these tools And so I said advice Well, I'm kind of ending it in an anti climactic, you know message here because as much as a lot building it And enjoyed the process and we are getting some value out of it I don't think it's the good way for whole industry to keep writing in every company We go to because it's it's not feasible. It's not scalable and huge waste of resource we should be working towards this goal and We are doing all this after I mean all the vendors have put a lot of work into it. They're smart people But I think they are focusing on wrong things And also trying to fit in the existing ecosystem of developers What's useful for them what we can how we can help them out fixing fitting in that context is very important And also we're also pretty good at finding bugs, you know, it's amazing like we love finding bugs It's fun, you know, you can we can just call out people. That's why we are in security anyway But like let's we have to you know build the process to fix them It's it's time for us to focus on and helping in enabling all the developers who are doing amazing things with empathy to fix it Because again, I strongly believe coming from a development background. I don't think developers want to write bad code They're just not have the resources and the support to do it And since I started with the mythological metaphor, it's only fair if I end with the some shit metaphor So this whole situation is like diapers You know diapers are for kids, but it's bought by their parents and The marketing is always towards, you know aimed towards parents So as an industry we have as an industry we we do the same thing It's used by developers the static analysis tools, but bought by security professionals as an industry We have done a great job of marketing it to us security professionals and making us Feel it's important, but now it's time to catch the shit kids made. So let's focus on that Thank you so much Anybody have any questions? Hey, great talk. Thank you. So I get that like Trying to do this QA process for catching bugs earlier on In the development process is good because You can you know get kind of earlier access the things that go wrong, but In environments that have a whole bunch of different development teams working adding code into the same code base How do you? How do you reconcile like maybe bug fixes that cause more bugs? And different situations like that or development teams that are from disparate Parts of the world and work differently together. How do you what's your advice for that? Yeah, I really think in doing whatever you can as much fancy tools you have The most important thing is to build a really solid relationship with the developers with the security team I do think and also In one of the slides I show can share I'll share it in LinkedIn later that We don't tell them like when when they go to see our the details of our finding We don't tell them that oh, this is this as CVS a score of this that this and everything we say specific things like what's the problem and how to fix it and For some of them. Yes, I agree like there are no Universal fixes and in those cases we always encourage like we have the link to directly contact us there I know it's a complex problem. I don't have a perfect answer for that But at least what has worked for us is a long-standing Relationship with developing teams where they can come to us at any time and we will Work with their situation rather than saying now. This is a bug. This is how you fix it This is this is the ultimate rule. So yeah, that has worked great for us Thank you. Thank you. So we're actually gonna shuffle outside for additional questions. So thank you very much, Arvind And thank you. Give a round of applause