 Welcome to the Blue Team Village for a year, got a treat for you, Chris and Plug going to talk about how to not suck at vulnerability management. If you do vulnerability management, you know that this is going to be a really good talk and you're going to be talking about how to not suck at scale. So how about starting off with some applause for these folks? Yeah Testing, testing. Hey, thank you for coming by. Beth gone. It's awesome And thank you for stopping by at the Blue Team Village. It's the first time so it's nice to have a lot of people here I go by Plug and this is Chris So the title of the talk as you see is how not to suck at vulnerability management So really quick we work for oath, which is a new company that is owned by Verizon We officially come from a CDN portion of that Verizon digital media So we do a lot of stuff at scale, and I think this is the reason why we decided to do that So let's let's just go for it So to kind of set the stage we need to talk about the current landscape and the reason why we decided to go about this talk We're not gonna go a lot of you we're gonna talk a lot of the technical portions of the of the day-to-day operations But we're gonna ask you why you should consider the way you're running your vulnerability management program Learning on the spot so In this case What you see in the screen is a lot of different companies and what they all have in common is they haven't recently been breached What you can see in there is that these are the reasons why they got broken into or the or the reasons why the data was Exfoliated or anything that happened to them that got our data exposed So you have a patchy vulnerability you have backends that are exposure to the internet on secure servers software bugs and all of these things are fairly easy and they could have been Proactively monitored by having a really good vulnerability management program In addition to that do a labs release these Reports basically talk about the cloud and all the findings that they have so if you have the cloud Well, that ability management applies to you and you should be aware of that These sets the stage and the reason why we're doing this because we do security and is very important that will net Ability management gets the right resources and the attention that it requires so we don't suck at it now The one thing I really want to show you here is that Management is not a check mark. It's not a compliance check mark. It does what I think it is You're doing it wrong like really really wrong. It's not about that. It's a lot more Now the management is not easy It's really not easy. It takes time It takes a lot of iterations and it's important. You understand that because you're gonna go for a journey Now before we go with that, we need to set some goals before you begin a bullet up Well, let's actually ask some questions. Who's doing vulnerability management here? All right, that's fair. Who does incident response or blue team? Who's in the blue team here? Cool So we got a lot of people and and they kind of blend together So if you're already doing a program or you're gonna begin with a vulnerability management program What you want to do is set some goals and you should be very simple with these goals and we're gonna define them This is what we implemented So our first goal was to have real-time identification. That was the goal. How soon can we identified a vulnerability? And how fast can we to react that vulnerability the sooner that you know that there's a vulnerability The faster you're gonna react to it and the better decisions you're gonna make. So that was goal number one Second way is fast 3-edge What teams did you have an incident you have to make decisions you have to make decisions very quick? So vulnerability management teams should do it too and it's very important that you work really close to address these This is the portion where you're gonna invest most of your time Finally mitigation and remediation. What's the point on when all of this work if we cannot mitigate or remediate vulnerabilities? Again, these are the three goals and these are the three goals that you should consider and you should work on your program for our recommendations Alright, so now a vulnerability management comes with some challenges and we need to address those and the first one is There are a lot of resources From which you get vulnerability intelligence. What is vulnerability intelligence? Well, that's a thing that we use internally, but basically is any source where you can get information about a vulnerability blog posts, Twitter Mainlist And anything else that will give you that the information about a vulnerability So there's a lot of sources lots of information and to cope with that information is difficult So that's the one challenge you have to address the second one is There will be patches that might not be available when a vulnerability comes out or you might not be able to patch So you have to have any strategy to how to deal with that and you have to compensate for that So it's an important challenge and we're gonna try to give you tips on how to do that Now before we go any further there is one thing I really want to address in here, which is something called common vulnerability It's calling system also known as CDSS Now this is what you see in the news all the time. Oh my god a vulnerability just came out 10 The thing is you should not trust that number and you should do a little more than trust that number because it's just that It's a number and what you need is context Context is gonna come from a lot of the things you need to understand in your Organization, so you know the number wisely, but don't go by it Here's some context Hardblade On version two of this corn system got a 5.0 Yeah, most of us traded like a 10 or more depending if you were to go by this core system You're gonna would have done wrong other that got a lot of hype But there's a lot of other vulnerabilities that are on five that are very important that you might not be paying attention second vulnerability Which one that maybe That's eternal blue. That's the one that basically allow you to Use SMB, right? That's the one to wanna cry use There are two score systems because there are two score systems. You can use version two or three Mission two is nine point three on this core system version three eight point one Which one do you choose? Do you go by the number and what are the conditions that you say? Hey, if I pass this number, I'm gonna do something Here's another example of another vulnerability that affects In the system in Linux now there are no scores This information of about the vulnerability. What do you do? Are you waiting for the scores to come in and then do something about it? So these are examples and you can find a lot more so please if you've done vulnerability management Take in consideration this core system But use it as a miracle value to ADU and provide context if you're just you need that number to drive your program You are gonna do a lot of things wrong Now in any program you have some Perquisites right things that you need to kind of have or you should have to be able to do your program better But one of the ones is to be able to understand your your assets. What are your assets? You seem to be now Moving on is if you are one person shop Spreadsheet is king and you should use it and maximize it and take advantage of that If you can there's a lot of open source tools that you can use to your advantage and you should look for them Now I think that is really important especially at scale is to keep track of your IP addresses You don't know that you might find an IP address you didn't know you own or a block and you might be surprised So it's very important that you understand you keep a really good type control on your blocks Well, they're internal or external. It's very important Well, if you're doing a cloud When you're doing below management, you have to ask a few questions. So first of all, is the cloud a plate? Do I have cloud devices? Well, you might don't think you have but then maybe someone in your organization that might have an account You don't know about and it's using the cloud Which providers will use assure Google? Which environments are on the cloud because they matter is it the development the production? And then what are the accounts? This is a similar person if you have cloud You need to keep track of the accounts associated with you Because you might get surprises and you might find an account that you didn't belong to you But one of your developers created for you. So very important information No, I'm not doing this extremely important is attribution you find a vulnerability You never know how to mitigate that but I'm sure a lot of you in here had to find out who owns what and oh my god Who's gonna take care of this? So attribution is super important, right? So who owns and who have contact? So it's very important you keep really close Attention to your records and as you're improving your program you're actually getting them better So yeah, so just going over some of the theory of what you're gonna What you're gonna try to do when you're we're gonna build it out your system So you can kind of break down the things that you need to know into two categories external intelligence and internal intelligence Hello, so yeah, you can break down so it's kind of Two categories of things like internal intelligence and external intelligence so external intelligence is generally going to be like public stuff So like us n's or redhead security advisories Security bulletins for Microsoft things of that nature You're gonna need to parse them to make sense in your environment And you're having internal intelligence which is essentially going to be you looking at your assets and Making profiles upon them in some manner to make sense to you that we can relate the two together to figure out What which ones are your things are vulnerable and which ones aren't? And then based on that data you or your tool or whatever you're going to do you're gonna you're gonna try to go out and drive some remediations The the important thing though is that I was remembered that you shouldn't you shouldn't get bogged down and trying to relate things together Instead you should try to like automate as much as that as possible to make it make it easier So into external intelligence. It's kind of a buzzword out there. You'll hear it other places It's it's nice Be be picky about the external intelligences that you're gonna look at like use the ones that are between parsable ideally and that are Our most most useful your environment. We're in a boot to shop So we used a boot to security notices and we we parsed them out and we were able to like pull some really good data out of it And then like compare against our environment They're almost all gonna require parsing of some sort But it leads to you know relate it to how it makes sense to your environment So like if you're looking at you know like a boot to security notices You might want to you know also parse various bits of pieces of it Yeah, now there's a bunch of vulnerability intelligence feeds that you can use a bunch of tools I can go direct to the source to like red hats security API or you can you can use these tools And then you can also there's there's a bunch of places you can get them So, you know take a look at them peruse them find the ones that make the most sense to what you're trying to do and And and you know pick one and start with it and then add to it as you go on Now internal intelligence isn't a buzzword. It's something we just made up Essentially the most important part is that you're you're being accurate with the data that you're collecting from your own assets and that You know that you're you're collecting as much as you can about it You don't want to have bad data or you don't want to be incomplete You're always gonna be a little incomplete because that's just the way wife is but you want to be as complete as possible And this is really we're gonna build a lot of your integrations with other tools Like if you're if you got if you went on you bought a tool for this or whether you're like you're trying to integrate with Another team or something this is this is where like all the pieces are probably gonna end up coming together Now there's there's a bunch of different ways to get internal intelligence to it's not just like I'm looking at all my servers You can grab like stuff from Windows and like your network devices your domain records Things things of that nature a flow data is a big one that you can look at one of our one of the guys on our team There's not in this talk. He does a bunch of great work with flow data And you can you can really learn a lot about what's going on your network that way so You know just just things to keep in mind Once you once you have these metrics, you're gonna you want to create metrics and data based on it It's gonna help you figure out what you're doing. We're kind of visual creatures. They're nice graphs are great And then when you when you have that data and you're trying to make an argument to someone to do x or y or z having the Graphs and having the metrics are gonna be nice and you know graphs are nice Everybody likes graphs Do try to keep in mind who you're looking at like this graph here is an example of one that doesn't Help you make your point because there's too much there So try to try to make it look nice because it makes it easier to sell your stuff upwards Something something more like this actually one of our coworkers who's in the former coworkers is in the back actually made this the set of graphs It was actually really helpful and and and getting some things actioned by the business when they could look at You know a breakdown of what these vulnerabilities were or whether it came or their categories and things of that nature So, you know simple is good Yeah So I actually ask something else to that because it is this is Management your audience Charts and electrics that are gonna go to your whatever here. You have different years You have to make them count a lot. So let's just kind of revisit there's a moment This chart you have a lot of information and it is that information meaningful. I Don't think so But this is a common mistake that I see in different places So don't don't do it. There's no reason to have that information if someone asked you for detailed information break it down But it will be much more easy to summarize. This is you know, I was and that's it Maybe like the significant version numbers now take a look at these other screen It's super easy to read a lot of the important things. It's just plain as simple What are your vulnerabilities and what everything like the most significance that are right there? How many of them do you have and also? What is what is what is the risk on them that we pre-score and it's super simple for someone to see oh my god I need to do something or I'm in good shape So please do yourself a favor and they're really good graphs or pay attention to your audience who are gonna read them These are the key for you to have people remediate things if you don't do them right You're not gonna get to remediate things properly All right, so let's talk about the tools We have these beautiful trusty tool, which is the spreadsheet and it might sound crazy But you can actually use the spreadsheet to run a lot of the stuff on your program or your program So one of the estimate the spreadsheet because a lot of stuff you're gonna be doing comes in the spreadsheets So it's a very useful tool maximize it use it and abuse it now we're getting to this interesting thing which is scanning and I was not gonna include this in the talk, but after having conversation with our peers I figure this is one of those where a lot of people have a lot of questions and it's very important to address them So what are some things about tips about scanning First of all, you should start with discovery scans. You should do them very small Take a small subnet do as most can Use very simple ports. There is no reason for you to scan 65,000 ports TCP UDP Don't do that. It's gonna take a long time. It's gonna fail You're not you're not doing any favor. So it's most cans small ports if you if you pass that then use the common ports that come on end map and use them, right then What you really want to do the discovery is you want to use the results to figure out What is your exposure? What do you have and then hopefully you can use that data to add it to your inventory So you know that you're kind of different the data to make sure that you're discovering the things that you should have That you have to come into your inventory or not but the most important thing is The ability scans the first time don't do that. You're gonna break stuff Especially if you don't know that exist in there. So do yourself a favor. Let's go risk hands. We want to blend ability scans then I want to come out as firewalls and it's a tricky one But it's a very important one firewalls will get in your way and you have to make decisions about how you're gonna go about that so a tip for you is So your infrastructure and you can set a certain amount of scanners that you have to be allowed to go anywhere on your infrastructure Then so the other sort of scan is that I basically facing the firewalls now you might be asking Why would I want to do that? Well, it's kind of gives a lot of other Visibility of things that you might not know if you set up your scanners to go through a firewall that allows you to go through And out of the blue you don't get data Then something is wrong with the firewalls something change If you have rules that say I'm gonna block this traffic, but you're able to see on Something else change and that might be bad So it gives you this extra intelligence and you can play with that and you can pass that intelligence to your response team Because out of the building we have an ACL and access list that just dropped so very important thing now I took that from the phone Reason that he did attack a blockhead and I'd really liked it because if you're doing scanners You sometimes are praying and waiting for the scan results. So I feel like it was important to have it Here's a really important thing to keep in mind But this is a very important question I'm sure that all of you are thinking should I do authenticated scans or not maybe PCI says that I should do them We should really really ask yourself do I need to do authenticated scans? But I'm gonna get the authenticated scan data the other way What are plans where you cannot do that? It's not possible But that doesn't mean I can't actually do a really good program and manage vulnerabilities. It's a very important thing you need to ask yourself Finally, if you're gonna implement any security scanning infrastructure, make sure you secure that Because that can be used for bad So if you're gonna have an infrastructure to cure it control it We can talk about IPv6. It comes with a lot of challenges But there's a link in there that you can use to understand how do you can scan IPv6 and we can talk about it after the fact But those can IPv6 and it's a lot of them. So I can use some strategies So before you start your program or as you study a program you need to ask yourself why you need this tool and if so why? Now the only thing you need to ask yourself is, maybe you have tools in your infrastructure that you can just use Just use the data. You're going to go and buy something. You have whatever you have and maximize the data to your favor Finally, there are no tools that do everything. There is no magic tool, but there is no magic tool There is no magic tool that doesn't exist. It's not like you're gonna go buy one tool And that's gonna be your vulnerability management program. That does not exist. If there are any vendors out there that tell you that that's the case And they're saying that is not true. You have a combination of tools and The best way to approach it is use one tool, maximize it and enhance that with other tools But there is no magic tool Finally you need to invent the world. There's a lot of open source tools We're going to show them that we made with you guys today and hopefully you can use that Hey, so when we were doing our role management process, one of our former coworkers he built a tool called Jellyfish internally Not at all Hello? Oh, hey A lot of people say At least So we built a tool called Jellyfish internally. Externally, we released it as Man-O-Rour Because there's another Python tool called Jellyfish, so we didn't want to encourage them to name A number of tools that you could use to kind of manage like your internal intelligence side of the house Where you're grabbing like your profiles of assets and things of that nature and storing them somewhere to analyze against The open source version that we made is missing some of the helper tools because it has some business logic in there I'm trying to get those open source maybe later this year You know when time allows Under the home, we're essentially reaching out to all of our servers And we're grabbing a bunch of data back about them and we're storing it in a database to be analyzed So the top one there is an example of sort of the data that we're collecting from a particular host The bottom one is an example of what we call an audit that's based on a USN there And then there's some graphs there that don't pay attention to the download worth the end that doesn't actually make any sense But essentially we're going out to the USN site and we pulled down the data and we parsed it and then we compared it Against our environment of profiles that we had grabbed Under the hood I say vectors a lot when I talk about it You can just treat that as a crazy alien guy But I wanted to shout out Baron Swartz of Vivid Cortex Back in 2015 he gave a talk on his tool Vivid Cortex And it gave me the inspiration to write this part And he used vectors in his thing, so Vectors So the getting data part of it, I call it profiling We store a bunch of data about servers in like a three tuple So we keep collection type, subtype and value Then we record when we first saw it and we've most recently seen it So we'll have like essentially a first state scene and the last state scene For this tool it's important not to record like performance metrics that change all the time Because otherwise you'll just be storing a bunch of different vectors and you won't get any of the benefit of storing the dates We also store IP data Our network uses a lot of VIPs So it's very nice to be able to store just a bunch of VIPs that are associated with it And then we have the ability to like customize the stuff that we're pulling back So we have a bunch of essentially behind the scenes is just a bunch of commands that are Bash commands that pull out a piece of data that makes up our collections You can see kind of an example of some of this stuff That cpu-info is a cpu-info command and like boot time is just It's a parsed date command that has like the time that one was booted And then like we pull it back into our interface and you can see the bottom one on the right there And we format it in a nice table so if you want to go look at the stuff And then we build a scheduler to go out in SSH into a bunch of hosts We also have an agent still very experimental but it's actually in the code So you can take a look at it and improve upon it I take pull requests so that's nice On the audit side of the house this is where we're grabbing data from our external sources And then this audit part is what's kind of merging the two and doing the comparisons Essentially I'm breaking all my hosts out into a group or a bucket So I have a series of comparisons that said which hosts And then I do a comparison against them to actually see like if it passed or failed the audit itself That's sort of the idea ideology I followed when I built Jellyfish or Manowar And it works pretty well. I haven't open sourced the part that automatically pulls stuff down So this is actually a live audit and I had to boil some of the stuff because it really veals stuff But it should be alright. You can see that it's essentially the top one there is Pulling out 14.04 hosts and the bottom one there is doing a comparison against a recent vulnerability And making sure that the version is higher So you can also do other things in the audits like you can tell to do regexes Or whitelists or blacklists and things of that nature So we have a few of those in there and then we display the stuff And like I said the USN scraper is still coming Also we built some APIs. There's a lot of APIs in there so you can integrate other tools Which we've been working on. We have one that's trying to statistically model the differences in our servers Trying to get that open sourced too but that's still coming And then there's web pages to allow you to do simple searches You can go like hunt for servers that have git installed on them Or hunt for servers that have a package or have a particular CPU type or things of that nature And then Oh yeah so there's a lot of that Ubuntu specific logic and our environment Specific logic that needs to be generalized before I can release some of the helper tools But effectively it should work if you write your own audits Talk to me afterwards if you want to see more of that type of stuff And then if I were a smarter man I wouldn't actually be presenting that award to you I'd be presenting one of these tools Specifically Hubble staff would have been a great fix for what I was trying to do If it existed when I started working on this tool But there's a lot of great tools that will help give you data about your environment Catello if you're a red hat person like that's built into like red hat satellite Or the open source version Spacewalk OS query is a great one. Faceback makes it and uses it and it goes out and grabs a lot of things Linus, yes at Zeus and if you're a Windows person WSUS should give you quite a bit of data You'll probably have to pull things out if you want to do more of the threat hunting type stuff But you know it's there You're always going to need to customize whether you buy or you build or whatever the case is And there's no one size solution to fit all so just you know make sure that you You're willing to do a little coding on the side or scripting at least to make it all integrate nicely Alright so that was the tool that we created in-house Now we're going to share the tool because it really allows to do vulnerability management at scale If you do try it it will really make sense. A lot of the stuff that we're telling you Those goals the tool allows to do that So one of the things that is not very clear in here is that with that tool One of the things that we can do is an inspector vulnerability just came out We can actually check our CPU and decide is that impacted and so how many of the servers that we have are impacted We can do it almost like this So that is really good because we can actually decide how we're going to mitigate that And a patch of vulnerability comes out no problem we can actually do that So find a tool you're welcome to use this tool hopefully will be as useful to us To you than it's to us And if you have any concerns questions or so let us know in the back But I want to also ask you something else Early on we mentioned that there are all these companies that had compromises And all of them had also something in common They used the cloud and some other services So one of the things that you should consider is Vulnerability management 2.0 which is go beyond the scanning Do you have S3 buckets? Well go and find them Do you have elastic search open on AWS? Go and look for it You use GitHub and you might have credentials but go and find them So there's a lot of stuff that is out there that the bug mounted people do Or attackers do and it's information that you should implement in your Vulnerability management program So one of the most important data sets that you can have is open source intelligence What are a lot of people that are scanning the internet? Right you have sensors, Shodan, you don't have the resources to scan your assets externally Get one of these data sets or get an account in Shodan and use that to check what your... What is the word? How visible you are, are you exposed to the internet? Has anyone scanned you and they know something about you? We're doing it, we use that data so you should implement this in your program It's free, it doesn't cost anything, so please do consider that Now we're getting to this important topic which is triage I'm going to run real fast from here If you want to be effective with triage you really need to know your software stack This is a challenge for everyone Are you using Windows, Linux? Fine, now you know that But what about the software that is installed in there? What about all the software that the developers are using? What about all the software that people are using because they're taking their laptops because they use it at home and they become an extension of their lives? What about all the libraries? You guys know that when you're doing an management sometimes there are not CVEs for libraries but it doesn't exist, so it's very important to know what you have out there Again, I'm going to keep going at those CVS score It's only good as a numerical value, value data, your stuff A lot of these vulnerability reports tell you that an attacker could possibly do something So when you read these advisories as yourself, is there an exploit out there? If it is, how complex is it? Does it matter to me? Am I supposed externally? How difficult will it be for an attacker to use that? Do you have that information to then account for the score of that vulnerability and make decisions based on that? Not only on the score Ask yourself the questions that we provide you here Now, it's very important to do this This is another common mistake You get a tool, it's beautiful You got an assist in the book, all those, any of those They give you a report, you got a list of vulnerabilities Oh my God, you got a load test But then you don't know about the data then About the data findings Is it really a vulnerability? Is it really a tent? What if it's a tool but it's even more important? That happens often And it's very important that you validate before you go to the owners to tell them to really get stuff Because how are they going to get something that they're vulnerable So it's very important that you find a way to validate One of the most common mistakes is someone finds a vulnerability on a poor or a sales And they don't even connect to check that that service actually exists Or it's a valid vulnerability Super easy to do, please do it This is a really important thing When you work in your garage You need to build a partnership with your teams The people that you work with You need to be friends with them You're not the enemy and you should not be one Another very important thing that you have to keep in mind is When you're without, you should ask questions Whenever they are not asking questions Is the worst thing you can do You're a secure professional There's a lot of stuff, but not everything And the people that build applications that are in your organization They know more than you And you are doing the service if you didn't ask them for questions If you didn't ask things If you didn't confirm with them something Please, don't do that mistake Go and ask both relationships Your litigation will go way better And you'll be much more effective Especially if you're doing this at scale We're getting into the renovation and mitigation This is a difficult subject Yeah, so sometimes you've got to interact with your organization To get things fixed And you have to pass I make you a ticket I throw it over the wall And sometimes you want to You want to shame people who don't know your tickets Try to avoid that Because then you incentivize them to avoid the shame And not necessarily to fix the issue Sometimes you do have to be the bad guy And call people out for not doing your stuff If you're using the ticketing method Try to avoid it if you can And then the other path is sort of self-service Oftentimes if you give your org the ability To look at some of your vulnerability data And look at some of the findings Going back to the metrics and stuff we were talking about If your graphs and things are understandable They'll go up And they'll actually go fix them for you And they will preemptively Before you get to the point where you have to go after them They will go and they will attempt to fix it Keeping that in mind though Accuracy is important in these things If you have a bunch of false positives And they go out and they put a lot of effort in To take a look at four or five issues And try to go fix them And then they find out they're all false positives They're probably going to stop looking at your stuff So if you want to go to self-service You have to make sure that your accuracy is right And actionable things And then so Some of the questions are like should you remediate it Like going fixing the root cause Or should you do some mitigation Sometimes it comes down to like what can you do Oftentimes when you're doing phone management The answer is going to be patch Nine times out of ten If you come across a vulnerability The answer is going to be patch But sometimes you can't patch for various reasons In that case you're going to want to look in the remediations And then document your decisions Not just because you can like hand that To an auditor Although auditors do like it But also because when you add new teammates You can give them your List of your repository of documentation And then when they're wondering about How you did X in the past They can go just search and look at it And it makes it easier to kind of record What you did in the past And how well you did And then sort of kind of iterate upon it And get better each time you go through it I think this one here So this one is very important And it happens to all of us in security And the reality is that one day One day things go horribly You know, things go bad Really, really bad And the advice that we have for you Is very simple If things go bad First, don't panic You're not the only one It's bad Two, don't shame Don't blame anyone Don't play that game It doesn't work The most important part is that Find out what was the cause And do lessons learned That is critical for you When you're doing your lessons learned It's very important that again You don't panic and you don't shame If you know that You're gonna learn from the mistakes And you're gonna be able to get better At what you do or what we do Which is securing things Again, that applies to everything But one day One thing is gonna go horribly wrong And just keep this advice in mind Now there are some next level ideas Let's say that you Many of you maybe are doing already All of this and you feel like Well, this is not the new I already am doing this Well, we're gonna give you some next ideas What can you do to increment And make your vulnerability management program better? Well, why don't you gamify Your program? I mean, we have CTFs, right? People are capturing the fly Well, why don't you capture the vulnerability If you remediate it? There's things that you can actually do You can create scores for teams The ones that are doing better Everyone loves swag, right? I mean, some of you got swag in here Don't you think that the people that are remediating The vulnerability is one swag too? Well, they do And if you do it, they're gonna love you And they're gonna want you to patch faster So, if you can, I'd recommend it Now, let's say that you're already doing that as well You really wanna get into the automation business Is really gonna make your life super, super easy Not all the time, but it's gonna make it fairly easy So, look into orchestration tools That can actually help you and do that If you don't have orchestration tools And if you work at a scale It's very likely that you already have some of that So, take it back on that Build your own stuff Just use what they have already in place And use it to your advantage It really pays off And you can use the bandwidth That you have to do something else More meaningful Now, what about buys? For orchestration, you have to secure your pipeline If you don't do that, it could be misuse If you use Jenkins and some other tools There's a lot of vulnerabilities So, make sure that when you're doing this In vulnerabilities, you address your own And that environment that you're using Is actually secure So, keep that in mind If you buy that and you're ready to go For the next level So, you can actually develop bounty Now, the best thing here is this These are the basics Which we cover early on, right? So, it's here on the base And the basic way to attach certain things And to respond to any game But I'll do bug bounty It's not for you You're not ready Wait for it It's not going to be painful And it's not going to be a good business decision Now, if you're ready You're ready to get a lot of work And you're ready to take some cycles Of your regular pipeline To address all the vulnerabilities That will come out from these reports Because you're going to get lots of reports So, you're not ready to make a decision But if you're ready Bug bounty is highly recommended We're about to finish this talk And this is one of those things That comes often Which is How should I not make sure My vulnerability management program? Now, this is a difficult one Because there's no magic bullet There's nothing that tells you How you should do that But every now and then We'll talk about the three primary goals That you should set Is how fast can I respond to something My triage and my mitigation You should use those three goals And any of the data you have to drive You to score how good you're doing Anything else doesn't matter I would say that the most important thing To make sure your program will be How good are you at triaging things This will come up easy Triage is very important Is the key to you to be successful To other things So, normally, there's no magic bullet In how to make sure you have an ability point But you should finally do that And you should set goals To improve, improve, and improve And with that, we're going to get to the end of the talk Sorry, so yeah So these are the final takeaways If you just kind of spaced out For the rest of the talk These are the things we think you should use So try to avoid shaming Work with your orbs when you're in doubt Go ask them Don't blindly trust upstream scoring If you are going to rely on CVSS scores Please make sure to use the temporal And the environmental scores And those scores will change over time As time changes and as your environment changes Validate your data Make sure it's good And improve incrementally It's okay to not be great the first time around Just get better every time you do stuff And the most important part is that If you're doing a very learning management Like you are one of the most important assets So don't get bogged down And looking at one bone That you can't quite figure out You got to stick and move sometime And get a little Muhammad Ali going So I think that's it We're going to be in the back Taking questions I believe we'll probably be out by the bar So meet us there Thanks