 Morning, guys. Can everyone hear me in the back? Okay. So my name is Magna Logan. I'm gonna be your host for the rest of the day, just introducing the speakers here. It's an honor to be here and be volunteering and helping this village. So yeah, without further ado, I'd like to introduce Yangyang Wang. She's a cloud security architect leading implementation of security tests at high trust. She has led integration security into various CICD pipelines and created successful security programs and culture. And she's going to be talking about automated pain testing in dockerized CICD environments. Give her a round of applause. Thank you. Good morning. Thanks for coming. And I hope you're happy by the end of the presentation. So today's topic is about automation, automating dynamic application security testing in CICD pipelines. So before I start, how many people are actually very familiar with the CICD pipelines? Okay. How many people here are security professional? Okay. I see some overlapping here. All right. That's good. So before I start, I wanted to say that I'm here on my personal capacity. And what I say, what I express in this presentation does not represent my organization. So the outline of the presentation is that, well, you know, in the me and I age, I have to introduce myself, obviously. And then I'm gonna briefly talk about the CICD pipelines because I know some people probably are not I mean, probably know something or not a lot or some people don't know. So I'm just going to cover it quickly. And then I'm going to talk about some characteristics of CICD that's relevant to pain testing. And then I'm going to talk about the reasons of this project, which is the challenges that I encountered, trying to automate the dynamic testing or dust in CICD environment. And then I'm going to introduce the approach that we did and then how we integrated with the Selenium test to make it even more awesome. So who I am? I'm a cloud application security architect. And so it's a month full, I know. And it means a lot of different things to different organizations. But I live in San Francisco in Bay Area. I think generally architect, particularly in startups, it means you do everything, really. So I want to talk about the CICD. So CICD stands for Continuous Integration, which means you commit your code multiple times a day into a centralized repo that is shared with your team. And then continuous delivery, that's what CD stands for. That just means every code should be releaseable and that you can deploy a code on the push of a button. And the pipeline means from the automation of the entire process, starting from when developers commit their code and then build tests and deployment. All of this is automated, meaning everything is powered by code. There is no physical environment that you have to deal with anymore. So why does it matter? So I wanted to just compare what software integration is before the CI and what software delivery is before continuous delivery. And so that way it will be clear why the existing tools, the pen testing tools are not suitable for the environment. So before continuous integration, when engineers write their code on their own and they test in their own local environment and then they integrate. Sometimes it's a week, sometimes it's a month, and then that big bang happens and things will fail in avoidability. And so you might, you're all software engineers, you know how meet the deadline for delivery is hard, especially in the past. And I don't know how many of you are actually in the ICD environment, but if you're not in this environment, it's pretty hard. And if you are, it's still hard, but slightly easier. So when continuous integration comes and engineers have controls of almost everything, they write code, they write the tests and all the tests are automated happening as they're writing the code, merge the code, and then the build happens right there after they merge the code automatically. And so it's all together. By the time the codes get to integration stage, the test will happen automatically too. And if there is any issue against the commit fails and they have to fix issues, whatever that is. And then for software delivery, before the continuous delivery, and you have to have someone to do the test after integration, and you have to have a team to do the deployment. And obviously there are human interactions there to get it to make it happen. And with continuous delivery, and it's same thing when engineers commit their code or the test or the release test deployment test, they all happen there. And so theoretically when you have, when your code is done, it's committed and it's merged into the master branch. And sometimes it's even in local branch and your code is, everything is okay. It's ready to be deployed supposedly. However, when it comes to implementation, there are shades of grade obviously. And most of the time, there is still some kind of human interaction for deployment. It really depends on the company. So this is something I found online from Instagram, Instagram website. I don't work for Instagram. And that's just to illustrate how fast things are moving today. And they deploy their back end the code 30 to 50 times a day. And oftentimes most of the time it was no human interaction at all. And what they're saying is that their engineering moves fast. So that's one of the key words for CI-CD fast. And then make it easier to find your bad commits and fix them. So that's another keyword too, then particularly in the sense of finding your security box. Well, I will come to it later. So I just touched the characteristics of CI-CD pipelines in terms of dust testing. I haven't explained what dust is. And I assume most people here already know. And if you don't know, can you raise your hand? Okay. So dynamic application security testing basically means you have to have a running app. And then you either manually or use a tool or both to point it to your running app and look for security vulnerabilities. So anyway, the characteristics that's relevant to automate your dust is the build are repeatable and deterministic. And then also that everything is automated on all the release, all the testing, testing, different testing. And also they're really fast. So the reason that's relevant is because when the build is, when the entire process is repeatable and deterministic, obviously, if you think about it, if you can automate your security testing here, then you have a repeatable and deterministic way to do that as well. So that's why it's relevant here. And then that's automated. That means, you know how oftentimes security teams would chase after the engineering team. And oftentimes it would say, that's deployed, I don't know. Oh, that's what happens. No one told me. I mean, every one of you said that. I guarantee you, if you didn't say that, you're not a security person. I've been in this field long enough, I know. So if you think about the automation and it's deterministic, then that means you can automate your security steps and do the same thing, which is a plus sign. And that to me, it's a good thing. And then also testing environment, in the past, you have to set up an environment specifically for testing. And you have to wait for people to tell you which was going to be released, and you're going to deploy into the testing environment, and you're going to test. None of them is necessary anymore for CICD pipeline. So challenges. If you look at the existing DAS tools, oftentimes it needs some kind of human interactions. They want you to put a URL in there. They want you to put the credential in there. They want you to put parameters there, what to exclude, what to include this and that. It almost seems like it's even worse to automate that because some tools does offer API tools to let you automate that. But it's just like this still involves a lot of human steps. And then also, this is irrelevant if you're running in a containerized environment, you have to have a running app. And I'll touch that later a little more. And also oftentimes with the existing DAS tools, you have to have a separate reporting system where someone else, where whoever is doing the test, have to log into the UI, find the vulnerabilities, and generally you create a Jira ticket or some other system, whatever. You know, if you wanted to make an engineer do that, you know what they're saying because I haven't taught that. I have too many tools to run. I don't want to log into another tool. So if you think about all the CICD and the state and the state of the DAS tool, you would be thinking about panicking, what do I do? Because it's a black box. Code gets into the pipeline and production website comes out of that pipeline. So fear not. So what we can do is unpack one of this CICD pipeline, one of the processes from beginning to end. So oftentimes you would always have SCM, let it be GitHub, GitHub, whatever you're using. And then you always have to have some kind of built testing. And in CICD environment, it's automated. But most of the time, built testing are automated to some degree today anyway. And then you generally have some kind of integration test. And then in a very automated environment, the built test and integration tests are usually combined. And then you have production. So built server, what is the built server? It's nothing but a server that you can do anything in it, run anything in it. So awesome. You can run any kind of security testing there too. And then so that means you have two places. It depends on your environment. You may have one or two places that you can do security testing. And that's during built tests or integration tests. So with that knowledge and the knowledge of, well, with that knowledge, so I know my requirements for a Duster tool is that this tool needs to be completely automated. And no matter which environment I'm testing, I have to be able to integrate it easily. And then it should produce a report within the CICD pipelines, whichever tool that you're using. And then it should be relatively fast because fast is one of the characteristics of CICD pipelines. And it needs to be scalable. So before I get into the details of how we implemented it, I want to talk about a little bit of container images or containers, just in case that any one of you are not familiar with it. Container, basically, think about the OS in the old time. Basically, everything, you have everything in code of that OS. So it's lightweight, standalone, executable, package of software that includes your code library and configurations, everything you need. So no matter where you have this container, you just start it and run it. So that's the idea. And then with container, oftentimes, companies would do orchestration, meaning manage the life cycle of the container. So you can think about that, I say, abstraction of network layer. So you know how you have to set up, you used to have to work with the DevOps engineers to set up the environment and the network interface or how the servers talk to each other, how the servers can be accessed internally, all of those can be taken care of, can be taken care by orchestration layer. And that's code as well. So that's what I was trying to say. There's no physical servers anymore. Everything's code. I know those of you who were here for the last presentation and Jim talked about how he thinks serverless is the future and then dockerization and orchestration is going to die. Well, I don't know if it's going to die in the future. I know that's what's happening right now where I am. I live in San Francisco and in the area, containerization is huge. So this is just some statistics I found on Datadog. So this was, this data was like March last year and they were saying docker runs on 20% of all the servers they monitor. And then there was a 75% increase of using docker. It's like March the year prior to that. So you can imagine it's a lot more now. Actually where I am, it feels like it's 100%. I mean, I'm biased stats obviously. And also another thing to be noticed is when companies use orchestration, servers goes up and down all the time. That's another challenge for dust testing, obviously. So now I know what we want from a dust tour in dust environment. I know where we can integrate it. And I start looking for tools. And I know I want to dockerize the solution because that's what works best for us. And then I know there's a plus thing. I mean, it's a nice to have thing which is integrated with the Selenium test automation. And because when you integrate with the Selenium test, then you test exactly what's released, no more, no less. And that's, to me, that's a somewhat important in CICD pipelines because we start the tour often time. You have to crawl the entire website and you have to test them more than what you need to. And obviously that slows down the tests. So, and then you wanted the resort to be available in the engineering CICD pipeline. To me, it's Jenkins UI. And then the tour should find meaningful vulnerabilities out of the box. So that leave me with two choices, zap and burp. And so there's an existing darker image and it's well maintained with a zap image and it's well maintained by OSP. But the findings are kind of limited. And so I looked into burp and there is some kind of open source, the darker images, but usually it's only suitable for interactive use. It's not for automation. And so what that makes me think that I can automate it with the headless, headless extension anyway. And the important thing is burp out of box produces a lot of meaningful findings. And to me that is important. If I'm going to do something, if I'm going to spend my time, I want it to count, right? I don't want to just check another box and that's not what I'm here for. So burp, I know you can integrate with the Selenium test. And zap, I looked for information, I didn't find any, so I'm not sure. And adding that, I think the findings was insufficient, so I decided to go with burp. So here's, I guess it's unfortunate for now. I got this acceptance for the talk like two, three weeks ago. I didn't have enough time to work with my organization to open source it. But we're working on that. I work in a highly regulated environment. We have, we have HEPA data. It's very restrictive. So I have to go through a lot of hoops to get it open source, but I'm getting there. So with that being said, I can talk to you on what I did, but I won't be able to show you the source code. Not today. So headless burp extension, if you haven't used it, it's a command line interface that you can use with burp suite and then it's usually used for automation. And one of the, another, some of the good things that when you actually integrate with the Selenium tests is beyond the test, exactly what you release no more or less. And you can pass the URL and you can pass the TLS cert wall and credentials. Pass. I mean, you don't have to do anything. Basically, Selenium is a headless browser and you just cross your website and the QAs have to do that anyway. So they have that part set up and you set up burp to listen to the Selenium traffic and it's passive listen. It captures all the credentials, URLs, and then the search is set up from QAs ID. You don't have to worry about anything. And imagine if you were going to actually deal with any other. Those are the things you have to do manually. So integration with the Selenium helps tremendously when it comes to automation. And then with the headless burp, it also produces JUnit and HTML reports and also I might integrate that with Jenkins UI. And also the headless burp has a feature that you can tune the fast positive off and that will just produce actionable findings. So the steps of Dockerize headless burp is this, that's what I did. And there are about like four, five different open source burp dog images on GitHub. You can find them but this particular one I found it most useful. If you follow this guy's interaction, you will get to the UI phase. And then after you have the dog image created locally, then you have to build the proxy and scanner, the headless burp proxy and scanner. And then copy that into the dog image you have. Sounds straightforward enough, right? There are a lot of bumps, sweet bumps that I had to go through and I'm sharing it with you guys. Hopefully you don't have to do it. Actually, if you knew exactly what I'm talking about here, your speed of Dockerizing headless burp is going to be a lot shorter than mine. And so basically the only way to active burp license is through the UI. There is no other way. And then use Docker volume option to persist your data, whatever you need to persist. And then this reference is actually from Burp Suite. It's on their support page and they will tell you, they will help you to Dockerize their burp. And so if you have questions, I suggest you to go there, check it out there, and you will find some valuable information. And then you have to build the headless proxy and scanner because the app in the Burp B app does not work. It didn't work with me. I don't know why. And so I built it myself and it worked. So another thing is if you want your Burp to work with Selenium, you have to set the listening mode to all interface because you have to listen externally versus internally. And then, yeah, I can't stress enough. And so I followed the disguises in instruction and then I forgot to restart my computer and it didn't work. And I stuck it and I was like, what's going on? And I had to post the issue on his GitHub repo. And turns out I just need to restart my computer. I just read the instructions more carefully. Problem solved. So any questions about this? I think I'm using 1717 or 177, something like that. Yeah, the newest way I tested, it didn't work with the headless proxy and scanner, unfortunately. This is kind of a boring question, so I apologize guys. How does the licensing model for Burp work for if you're spending on multiple Docker containers? Yeah, I think that's the question you should talk to Portzweiger. I practiced like a hundred times. So the Docker license is different than the Burp suite for a license. You need a separate license for Docker version? I checked online. I don't think they have that model right now. So if I have a Burp suite for a license, I can get the help check. Yeah, so yeah, license question, you should all talk to the company because I have no idea how they do it. The reason I don't know either yet, I'm going to talk about it in a bit. So once you have a Docker and if you work in a company, you're doing this project for your company and this is the time you hand the Docker repo to your DevOps guy and then tell them what you want and the magic will happen. So basically that's what happened to me. And so what happened is I checked in the Docker image and it immediately tricked a build in our build server and then the image got pushed to our image registry. And then the rest I pure seeded out and worked with a QA team and we tested it out with Selenium and everything worked out like much better than I thought. I can't complain really. I thought it was better than I ever asked. So unfortunately, like I said, I'm here on my personal capacity. I couldn't show you live demo. There's no way I'm going to show live demo in DevCon anyway. But I recorded my screens to go through the entire process in this image to just give you an idea of what happened and what the end result is. So before I move on to that demo, I just want to talk about it. So we actually integrated with the integration tests and so we haven't done for like all the engineers when they run their tests that we haven't done there yet because right now we have other issues to solve before we can move there. So that's another reason I didn't know. So okay, I'm going to do the demo. Oh, I guess it's going to go through very fast. I can pause it looks like. So this is a, I'm showing you, I'm just trying to show that I made changes to the Git repo. And it's going to show up and will trigger a CICD, will trigger a build in the CI server. So I made changes in the README file and then I had to commit it. So now the code is committed, Git push made it happen. So I'm going to refresh the page and this wasn't there, the DevCon demo online wasn't there and now it's there. So commit was successful and then I go to our CI server. So those are the steps that happened. I wish I could pause it. Maybe I could, yeah. So those are the steps that happened after the build was triggered. So I think this is okay to talk about it. We actually use the SoCoCI. And in SoCoCI they spin up a Docker image, they set up the environment, they do whatever task you ask it to do. So in this case build the Docker image, inner Docker image and then it pushed the image to our image registry. So this is where the integration happens in our environment. We're still trying to move it into one environment but suffice it to say once it gets into integration environment it triggered a burp proxy. It started it up. All of this is automated. No one did and I mean no humans is doing anything behind. Go ahead. You committed to your own branch. I mean it depends on how your organization is set up but unless you're doing it yourself I would imagine you have to have your own branch. So I had my own branch and it got pushed to the master because that yeah Exactly. Everything is done and I'm finished working on this project. I'm sure they will be maintenance but right now it's pretty much done. So that's why I was committed to master. So this test took 23 minutes. It's relatively long but I also had a test that took about five minutes. So this is another thing to be considered when you want to do integration. How long it takes to actually do the test for your environment. If it's too long you probably won't be able to integrate with the development pipeline. So that's what happened in Jenkins. All the reports was generated and then they will actually post it into Jenkins and then there is a stack message sent to me so I know that a report is done because it's still kind of early for me. I'm still manually tuning the tool so that way I can feel comfortable handed it to everyone. Just say hey go ahead and take it and everything's okay. There's very few fast positive because with all the security testing tool I mean I'm sure you all know it always has the fast positive and then a lot of things is environmental we have to tune them off. Then we haven't done it yet because like I said I don't feel 100% comfortable that I can just feel built yet because I haven't finished tuning the reports yet like the fast positive. Good question so right now it's an integration test it's the UI. It's actually powered by the Selenium test. So I have thought about the UI. I thought that I can probably integrate it with a REST API tool. That's my next step. Oh and so you see that the step here published HTML report so once this gets done it produces a report. So when the report is produced and in Jenkins you can actually post the using code to post it into the UI. Actually it's not webhook. We use I don't know if I can talk about our environment yeah but we use a specific kind of way to post the things in Jenkins but it's code it's not webhote. It's customer code. It's not that customer yeah you can find the sample code so I think that part I should be able to open source it once I go through all the hook and I'll post the sample code online you should be able to see it and you can just use it. Yeah yeah that's the idea. Yeah you know it's a QA tool it does all the function tests for the particular feature you're going to release and Burp it just out of box. I didn't do anything special. I haven't started tuning it yet but I'm curious to see what I can find once I start tuning it. So theoretically you could replace Burp with a commercial availability list. That's what I'm thinking. Yeah that's what I think I hope the commercial tours if there are any representatives here they're listening. Yeah because so I don't have to do this I just spend my time doing other things. Yeah so so I can see that on so I did the integration phase so it's a little bit later already and usually QA that's QA's group and usually they're tested take an hour to anyway so they don't mind but I know what you're saying so what I was just thinking is to make it happen we see the depth environment I might try I might try out this app that's what I was thinking for what I'm going to do next. Like remember the engineers develop their code and then they trigger the build and it goes through all kinds of tests sometimes coverage and quality tests all of those you can add zap there that's my next step. Yeah if the test or if the code is the same the findings are the same no difference at all it works pretty well. Yeah so that's the part of where I said it works pretty amazingly when you use the Selenium. Initially I thought I have to set up a verb but it turns out none of those it's necessary because the Selenium has to handle the credential itself and once it locks in the tokens get carried out in the subsequent requests so there you have it you don't have to deal with the credentials. Yes all right no problem so that's the report out of box that's the findings I have I would say it's pretty good and for how cheap a verb suite is I really can't complain so yeah and that's the end of the presentation any any other questions. Yeah so with headless verb that's the part where I was hoping that there does the representatives here that they can come up with the commercial tool you have there's a configuration file you can add specific findings there as a filter but you have to do it manually you can do it but it's kind of there's a lot of work that's finding type yeah unfortunately so right yeah yeah have you given any consideration to figuring out a way just to test what's changed to make it even faster? You mean when you said test do you mean in the CI server? Well when you run the verb task kind of thing oh you could think your verb so say it knows from a manifest or something you know that it's five changes ended up with so so I okay so I know where the confusion comes from so the committing part and then trigger a test in CI and commit the and then push the image to image registry that's for the test is not for verb I use that I put the image down so if I show you this diagram what I did is I commit the code that triggered this entire process and then the image got pushed here and this part doesn't have anything to do with how I create the verb image anymore yeah I just put the image what do you mean when you say it doesn't know how so like you know man that's not correct but when you're testing on the death side does it know anything about that? Yeah well so in this step update deployment if you're using some kind of orchestration orchestration yeah the orchestration actually sends a request to your environment so if you're in a Kubernetes environment Kubernetes receives the request and it updates your environment by pulling the newest image to here so if you make changes and another environment that's using the image it will get updated but we don't have to worry about it because all those are set up via the orchestration yeah yeah yeah the infrastructure team infrastructure code takes care of it yeah yeah only if there are changes to the verb suite image that I made it will the information will get updated via the update deployment part like the orchestration part takes care of that to pull the new image and know when to pull new image it only pulls the new image if there are changes if there's no changes obviously it doesn't pull new images yes but um so what my username is pink gladiator on github actually none of us once it's set up it's none of us does anything so selenium sends all the requests to the web website you're testing and verb proxy passively listens and record all the traffic and then take the information the data that it recorded from verb proxy and then and then I when that's done when selenium test is done I kill um verb proxy set up a verb scanner to scan the information I just collected from selenium the app inside the container you could test whether you can break out of the container or whether you can achieve root ownership of the container which of those three did you test um this is a test a application outside of the container so they basically it's nothing fancy I didn't try to to you're saying if I tried to escape the privilege to get the root privilege yeah none of those yeah we have hack one for that we have okay sure yeah those things and you can't find them via this kind of automated testing yeah it's a lot more complicated yeah yeah that needs human involvement that's why we always have jobs yeah I think that's that's it my time's up thank you