 I'm going to introduce you, uh, uh, not, uh, next speaker. Um, he's currently working as application security engineer at Global.com, which is a, uh, news media and TV network from Brazil. Uh, he has a OSCP and OSC certifications jumped into software engineering to help develop teams improve their quality of their code, uh, by finding vulnerabilities as much as possible. He's very interested in development, uh, exploit development and building security tools to automate the process of, uh, security issues during the SDOC or CI CD, you choose. Uh, his presentation is going to be about Husky CI finding security flaws in, in CI before deploying them. Uh, give a round of applause to Rafael Santos. Hello. Well, thank you very much. Uh, I'm very excited to be here. It's first time here speaking Vegas. So today I'm going to talk and present a tool that I have been working on with my security team. And it's called Husky CI. So who I am, who I am. I, I have already been presented, but, uh, as a Brazilian, I play it off a lot of soccer. And when I play soccer, I usually play as a left winger. And as, and I, for, uh, uh, uh, uh, last year I, I, I was working a lot of in the pain testing field. And this is something that I did, some sorts of certifications in this field. And currently I'm working as a security engineer at global.com. And global.com is the largest media company in Brazil and in Latin America. We, it's, it's, it's current, it's actually a TV channel. We will host some, some famous shows like big brother Brazil and the voice Brazil. And, and my passion is about writing security tools and trying to understand how the development part, how the process of exploit development is. And this is some of my passions. So today I'm going to talk about a story about a day in the life of a development team. So, uh, let's imagine that there is this development team that, that they are, have their project. They have that, they, they have that repository that is hosting in the, in GitLab. And they're doing the process of developing their project. They're, they, they actually use a first, for example, to lose our platform as a service that is open source, but you can also use Hiroko on AWS to deploy our projects. And later on we have our products, our products in production. So this is basically the overview of a day in the life of a development team. So let's imagine that these developers want to develop, develop his, his features and try to do his, his commit to the repository and all the tasks are being used in the, in his CI. So we already have these configurations being made, the build, the iOS and write and linked and the tasks. So later on we, at global.com, we ask it to another developer to do this manual called review. So to, to check if there are some flaws and, and, and something that was not seen by the, these automated tasks. And if that, that is correct, this, this, this commit is, is merged to master. And we do, we deploy it to our platform as a service. And later on the, the project is, is in production. So as a security team, the, this process, we, we see there's something is missing. I mean, I mean, no, no security test has been made here. So as, as, as some security teams, we see, wow, you are actually coding in Go. Have you ever heard about this static analytics called GoSec that is open source? The, it's very good. So you, you can add it into your CI so we can fight eventually security flaws before deploying it. So the developer actually, the development team actually include this in their tests. And this is something that we, we, as a security team thinks that is the, is very good. So, but later on, another developer that is working in another language, that is Ruby, the same process. We as a security team comes to the team saying, hey, have, have you ever heard about breakman? That is also an open source tool that do the same thing. And the same process goes. And the team will eventually put that in their CI as well. But what, what happens if there are a lot of other language that we do not know? For example, which static analysis tool does these developers must use if, if she use Java or JavaScript? So we as a security team could eventually ask them, hey, what about using a tool that we could eventually create that will, that developer may abstract and will not have to worry about which security to issue or he would eventually try to use. And this is, is what we dream. So this would be great if we were able to do that. So a day in the life of a bigger organization, as I was saying, we are actually talking about one security as one development team. So global.com is a huge organization. We have a lot of developer teams. So if we try to scale this scenario, we have a lot of development teams. We have a lot of pull requests. We have a lot of commits. And each one uses its own. It, it won't repost or get hub, get lab. So everyone is deploying. Everyone is doing this thing. Unfortunately, our security can't be inside all of the development teams. So all this commit is being made and is being deployed. And later on, our security team will, will do some tests, some manual tests, some pen tests on, on, on, on this project that is already in production. So this is something that is very bad in our perspective because there's a lot of, of coding involved. So at global.com, we have this project that is called the hack day. There is actually my t-shirt that I'm using that we, that is, is, is a hackathon that we, we take a day and day or two to do a lot of everything that we, we want. And this was a great opportunity to us as a security team to try to develop in this security tool. So what we, what would be our dream? We have all these development teams, each one using its own repository. We would eventually add this security tool in, in, in every CI. And eventually some may eventually pass and some will, will eventually fail. And for, for those that fail, we will eventually receive this response because we, we, we can eventually have this emailed and, and, and that is good to us to know what, what, what vulnerability is being found. And for example, for the second development team, it will eventually see, oh, we have found this one bit. All right, let's correct it and send another request and we will eventually mitigate it. But for some development teams, this may not be so easy. So by, by using this tool, we can eventually put, put this, this security analyst inside that team to understand that vulnerability. For example, we can't, we, we don't know, we do not know how to mitigate this SQL injection. Would you help us? So our security may eventually be there and try to, to help them. And now this is the plucking we made. So this is our dream. And later on, our security team will eventually continues doing these pen tassins and manual, and manual tassins in our productions. Okay, so what are we quoting? At global.com, we quote in a lot of languages. So we use a lot of Go lang, use a lot of Python. You, you use a lot of Ruby. You use a lot of, a lot of languages. And this is maybe, and this is the eventually meet the, the same scenario as you face every day. And I even challenge you to, to check this slide and see, you, you may eventually miss some language that you work on. And that shows how challenging is to us to, to, to security team to, to handle this. So as we are working in the hack day, we were just trying to focus on these three languages at first. We're trying to use Go lang and Python and Ruby. And there are free open source tools that are being done. And this is another important thing. We were using open source tool only. So any other, other companies could eventually use this tool as well. So Go sack, ban it, and break me was our starting security task tools that will eventually do. So how would we build these two? So we have this development team, it will eventually send this pull request. DCI will, will start all the security tasks. And Husky side will eventually fail if some vulnerability is found. So we are using some, so three components here. We use an API, a Docker API, and now our database that we are using Mongo. And for example, if we're using this Go language repository, when this pull request is being made, and Husky CI will receive that request, it will eventually send this request to a Docker API to start a container that will, for the first thing that we'll do, it will use Henry. Henry, that is actually another open source tool that can check which language are in those repository. If you go to GitHub or GitLab, you will eventually see, oh, this repository is 80% go and 20% make fire, for example. This is what he do. So he does, sorry. So and he will eventually see that that is a goal and we'll return it to the Husky CI API and say, hey, this is a Docker API. This is a Go language repository. So what are the security tools, the open source tool that we have in Go language that we can eventually try to scan it and find vulnerabilities? At this one, we only have Go second, but as we are building these two, we are actually trying to make it possible so everyone could send plugins. So I've got these new security tools in Go language. And so this was eventually built a lot of containers and tried to run all of them and eventually failed it. So this is the same process. And later on, the security we eventually see these results that is being stored in our Mongo database. So what happens if another language is inside this repository? So now Python is being aided, some scripting Python is being aided in that repository. So the same process will flow. And he will check that now we have both Go language and Python. And later on, we will create two containers or more if there are more security tests related to Python in parallel. So we can run all these security tests. And later on, if there are vulnerabilities, we will fail with the CI. So this is our dream. And what we would like to see, what are the matters that are important to us? The first thing that it would be good to see is the commit outer, not to blame the developers or nothing like that just to have these matters. So we could, for example, if we find that these top 10 developers are committing a lot of SQL and Jackson, we could eventually get our security to train them in this specific vulnerability. So Git commit outer is something that we think it's good to us to see. The repository landed for metrics. It's very good. Also the files that were found in that repository so we can have, for example, in this pull request, we have these files. And later on, a new file has come and a new vulnerability was found. So this is something that would be good to us to see. Also the branch name. And the most important to us are the vulnerabilities that are being found. So this is an example of the two. And also the mitigation. So if the vulnerability was mitigated, it's a very important metric is to us to see. So this is how our MongoDB looks like. And there is this repository, the containers that were used. And the results that we're actually trying to separate here. We currently are trying to fail the CI when meeting and high vulnerabilities are found. So this is something that we start with. And this is how we also are bringing it in some JSON so that we can consume this output in other tools. For example, if we use a front end later on. So I've got a, and this is another thing. So what the developers must do now. The developers must only add at his configuration, the CI, these simple requests that we will eventually download this client, this Husky CI client. And the client will be executed inside the container and the container will run that container. So we'll run the security test. So the developers do not need to check which are the security tools that he or she needs to use. Husky CI will do that for him or her. So I've got a video here to show the tool working. I've created an open, a repository that is open. That is public. And this is the repository that is in GitLab. I have the reference for the links. And let's check the GitLab CI configuration. We'll see that there are a lot of stage data already being used. And now I'm using this Husky CI that we have built this external environment so we can test. And now we're going to add some vulnerabilities here just to check how the tool works. We're adding some, a new function here that Git, MD5, that is something that, there is a vulnerability in Golang that we are using some wiki function here. So we're actually checking out, creating a new branch to push it. And at this moment of this repository only has one language that is Golang. And after committing and pushing it into GitLab, all these tests will be, we'll start. And we are not using GolSack, we're using only Husky CI. And Husky CI will be, will be able to understand that repository is a Golang. And later on, it will eventually start all the security tests related to Gol. So I have added the video so that we, as we are short in time, so the pipeline has started. And all this security, all this pipeline, this pipeline has a lot of jobs. And the last one is Husky CI. And when we click it on Husky CI, we will see it running. And the container will run. And we'll eventually download the Husky CI client and start the analysis. So after a few moments and a few minutes, Husky CI will eventually find those vulnerabilities on the CI that is medium-high. We're using a weak cryptography primitive. And it shows the file that, that vulnerability was find. It finds the line of it, the code itself, and, and give it some summary over the line, over there so developers could see what's going on. So for GoSack, it has found some four or three vulnerabilities. And as there are some vulnerabilities that are medium, the CI will eventually fail. So that is the CI failure. So what if a new language is added into the project? As I mentioned before, as what if a new script is added? So I'm creating a new branch here and I'm adding this Python script that is using some, that has some intended vulnerabilities here. So we can check if it, Husky CI will find that we are storing some hard-coded credentials on it. And eventually calling a function that could cause a remote code execution, for example, that is, that we consider insecure. So the same process is being done. Now we are committing and pushing it once again to, to GitLab. And let's see what Husky CI will find. And the new pipeline is, is created. And the jobs will be run. I have added the radio so we can check just Husky CI. And the runner will start. Now the Husky CI will eventually see that a new language is being set. And it will add, and it, and it, and it will run all the security is related to Python. So as we already had the vulnerabilities in GoSack, we still can see them there. And also as we are using now Python, currently in Python, we have two security tasks that we use that is banded that do some static analysis and safety that checks for vulnerable, vulnerable dependencies. So this is what it, it found. So I want to show you some statistics that we have in global.com so far. So we have talked to some development teams. So if, if they wanted to have Husky CI in their, in their CI as a better testers. So as I said, this is our, some of the products that we, we have in Brazil, in global.com. And some of them are already using Husky CI. And we have a 65 unique repository that was already analyzed. And 108 unique range that was analyzed. So 20% of the, of our CI has failed. It's causing, showing us that some vulnerability was found. And 52 passed. And the others just have, the developers are still understanding how the, the tool works and some errors has been found. For example, they forgot to, to set their environment variable that Husky CI needs to, needs to run. So this is our, our first statistic that we have. And the, the, the metrics of the seconds that each secured tool takes to, to, to run completely. GoSec takes 14 seconds. Breakment as well. For Python, we have these both secured tests that we are running that take this long, 730 seconds. And for JavaScript, we have these two, these two tools. We are NPM, we are using NPM audits that we have built our own containers to do this check. And Retire.js is another security that we use. The Retire.js takes a little longer because we are, we need to do some NPM installed before using the, the tool. So this process of NPM installing is taking a little longer. So, and all the process is being taken seven minutes, our average, because a lot of projects use multi-languages. And eventually all these containers must be created. And this is our average so far. For vulnerability, the most, most vulnerabilities that we are facing there is some error are not being handled. For example, in GoLang, we, we are able to return error when we, when you're using our function. And we do not want to use this error. We would eventually use, use the, I forgot the wording, this is that, that, I guess. And this is some security vulnerability for us. And all the others are vulnerable dependencies. This is the top one for all of them in languages. And what are our next steps? As I said, this is an open source tool that everyone could eventually try to use inside your organizations and try to understand how this could help you. So as I said, we are at this moment, we, at this moment, we have these three components. And our security team is working on checking all the, all this data from our MongoDB. And, and some feedback that we are having from our developers is that we need to have a front-end so everyone could check in in a more, more friendly way to see this vulnerability. And to collect these metrics, there are already some open source tools that are very, are very good to do this. That's our, our tree sack. And eventually, SonarCube, that can use it as inputted to, to use as a front-end. So we are, we are not using none, none of them at this moment, but we could eventually use them. And there are a lot of other languages that we are not facing. We are not handling at this moment, but it would be great if we, if we could. And next step is also to contribute to this, to these awesome open source tools. Then they are all, they are all open source tool that we, they are being updated daily. And it would be great to us as a HSCI community also contribute to them. And add more security tasks. For example, there are some, these are some open source tools that could, that we could add. And there are some GitHub repositories that have a huge list of security tasks as, of security static analytics tools that can be, can be used. And this is something that we have to, in mind. And many more, we are trying to open some missions to handle how, how, how the future of the tool is going, on, on how we, we could do some more features and, and now, and everything. And I would, if you would like to give HSCI a chance, we are trying to build some documentation so it's, it turns, it's so it's easier to other, to other companies to use it as well. And as I said, it's open source. These are my reference. I'm not sure if I was too fast, but I'm open to questions if anyone has one. Yes. Yes. We, yes, we are, we are scanning each push. If when a, when a new push is made, we're, we're getting that range and, and, and using that commit from, until that commit, that range. And we clone all those repositories based on that branch. We, we do some git clone, a single branch, and the following branch that we're working on. And later on, we start all the security networks based on, on, on, on that code. So, yeah, it's the entire repo start. Not sure if that is optimal, but this is something that we're doing at this moment. We, we are trying to use some goal routines to start all the containers in parallel and trying to make it possible to scale it. So it's easier when a lot of repositories try to use it we, it, it, it's smoother to our environments to run it. So we're using goal routines to start them in parallel. All the security tools. Yes. Parallel. They're working parallel. Yes. Yeah, sure. We are using a lot of GitLab because we use a lot of GitLab in, in our, in global.com, but we also use some GitHub and also SQLCI and, and, and other, and other CIs. So, basically we, what, what we could do is, is just add it. You have to build this environment, the SQLCI invite, and put that line inside your container that will be run. So, I think it, it would be easier to use in GANKs and any other CIs. Yes, nice. Some, some security tools, uh, permit us to add this, this comment in the line that we, that developer and the security team think that is a false positives. For example, GoSec allows us to put some comment in you, uh, uh, using NoSec. So that GoSec will eventually not consider that as a, as a, as a, as a vulnerability. But false positives, some of our challenges, in, because if some developers also, uh, ask us how we could not fail that CI if this false positives is shown. But we are, at this moment, we rely on how the security tools are, are, are, are allowing us to, to handle these false positives. So, for Bennett, we, we, our safety we need to check if then can allow us to, to add this tool. But it's something that we could eventually do in Husky CI client. We could do some feature to try to allow it us to not face this problem. Nice. Uh, well, when, when, when the first talked to developers seems to, to use this tool, it, it's something like, it, it's a culture. So each team has its own, uh, autonomous, uh, way of handling the those repositories. And as, unfortunately we are not able to be inside all of their, our development teams, but on, on the, the developers, on the developers that we do have a, a security analyst. We ask, we ask them to gently add this, the security tool in their CI. And, and our, our most problem is, is basically finding that the repository using vulnerable dependencies. So this is something that we are trying to do. Developers are, are, are, are, are, are, are generally saying that those dependencies are from the vendors, not of their project. But we actually see as, well, it, it, it is indeed vulnerability. So we should not use this, this, this, this dependence. So this is something that we're trying to, to handle. Uh, I see secured more, more like a people. So we have to talk with them and try to increment, input some security culture inside our team. But this is something that we're trying to, to work on. Currently, our, our, our SKCI team is, is, is free, two and three developers as well. But it's something that we could eventually add in the future, even trying to integrate to, to these tools or service now and, and any other tools. Okay. Thank you very much.