 Okay. Thank you so much. Can you hear me okay? Perfect. So I'm here to talk to you about bugs in the system addressing the vulnerability problem. We're going to solve it in about 20 minutes. Are you ready? Okay. So for those of you who don't follow me on Twitter, this is a word cloud of things that are important to me. Things that I tweet about in order of how important they are. But these are a few of the things that I've done in my career and that I've known for. Obviously, I'm the founder of a new security company founded about a year ago. And before that, you know, I was with a startup that was a bug bounty company. But I changed course because it's not necessarily all about bug bounties. How many of you have heard of a bug bounty? That's okay. I'm going to explain it. Oh, most of you have. Last year at this conference when I asked that same question, hardly a hand was raised. So it's an interesting phenomenon and we all think that bug bounties are all the rage and they're going to solve all of our bug problems and security problems. But this list of things that I am and things that I've done, not too bad for a high school dropout misfit hacker. Okay. Before there was North Korea, there was Katie. So this is an actual photo of me having been hired by Sony to hack one of their online games over a decade ago. It's been over a decade since I was a professional hacker for hire. And I thought for sure we would have solved the vulnerability problem by now. I mean, problem solved forever. It can't be that we're still running into the same types of vulnerabilities. That we still haven't solved cross-site scripting and SQL injection with training, with better coding tools, with better frameworks, with better testing methodology and security awareness for developers. It's amazing to me. So I got really sad about it. And I stopped being a professional penetration tester thinking, well, maybe there's something else that I could do to be useful in this world and hopefully help to solve this vulnerability problem once and for all. Okay. We only have about 12 minutes left. Let's see. So, of course, Twitter is the source of all kinds of wisdom these days. This is some guy on Twitter's idea of how to solve security problems. You want things to be secure, fast, cheap. That intersection, if you can read that small, it says not going to happen. Not going to happen. So he's on to something. This guy is on to something. But we started talking about bug bounties briefly. So this is an arc of the progression of bug bounties. And for those of you who didn't raise your hands, bug bounties are simply paying money in exchange for vulnerability information, in exchange for somebody reporting a security bug to you. So back in the mid-90s, Netscape was the first to offer a bug bounty program or offer a bug bounty $500 in exchange for reporting a security bug. Now, how many of you have heard about the millions that are spent on unlocking the most secure devices? Like 1.3 million was a number that was thrown out there for the Apple versus FBI, the purchase of that exploit. Now, what's missing in that context is that it's not usually for a single bug. It's for an exploit, which in modern systems usually contains multiple vulnerabilities and utilizes a complex exploitation technique. It is not as simple as one bug gets you full access to most modern systems. Now, that being said, when I talk about cross-site scripting and SQL injection, those are very simple types of bugs to find and exploit, generally speaking. When we're talking about the high end of the vulnerability market, that is where we're talking about the most complex exploitation techniques. But from this chart, you'll also notice that there are non-technology companies that have joined the bug bounty bandwagon in the last few years, like an automobile manufacturer, Tesla, Western Union, a shipping company, and then you also see a nice little seal in the corner, the Department of Defense. How many of you have heard of Hack the Pentagon? Okay, great. Well, we're going to talk about it in a minute. Even some of the camera guys haven't heard of it. That's great. Okay. So, when I talk about, I've got 10 minutes to solve this problem for you guys. Okay, let's speed it up. So, when I talk about vulnerability disclosure versus bug bounty programs, it's been trendy to talk about buying bugs. But what is the foundational element that you need in order to support a bug bounty program? Well, you need a process to fix it. Finding the bugs is only half the equation. How are you actually going to address these things? And the foundation is vulnerability disclosure. So there's a couple of ISO standards that I've been the co-author for. ISO 29147 vulnerability disclosure. And ISO 30111, vulnerability handling processes. How many of you heard of those? Okay, only the rest of my new American fellows. That's it. And the reason for this is that it's not popular to have to actually follow some sort of regimented process for handling your bugs. It's very popular to talk about how you're finding more bugs, but not necessarily how you're fixing them. Bug bounty programs are simply an incentive towards getting more bugs into your system to be able to fix it. But if you have no mechanism for fixing, you're dead in the water. So how did we get from Microsoft who had historically said they would never pay for vulnerability information? How did I as a security strategist there get them to go ahead and start paying among the highest bug bounties in the industry? Hmm, okay. Oops. So these were the three original bug bounty programs at Microsoft. $100,000 for a brand new exploitation technique. Remember I said earlier, it's not a per bug exploit, a one bug exploit that usually compromises most modern systems. It takes a chain of vulnerabilities strung together in a particular way to compromise the most complex modern systems that have had security hardening over the years like the most recent Microsoft operating systems. So to Microsoft, it was worth $100,000 for a new technique. The only per bug bounty was the IE11 beta bug bounty. And that, this slide, is how I convinced IE to start paying for vulnerabilities. So what you'll see in white there is actual data. So there were very few vulnerability reports during the beta period of IE10. There was no bug bounty. But we had over 200,000 non-spam email messages a year coming into secure at Microsoft. So over 200,000 times that helpful hackers were trying to warn Microsoft about potential security issues. So why would Microsoft then start paying them? Well, the purple is a projection. And I said we can shape the traffic because we're getting all of our bugs reported after the beta period is over, after the code is released to manufacturing and the code is somewhat stable. Why is that? It's because we had created a perverse incentive. And the perverse incentive was the only thing a hacker could get in exchange for reporting a vulnerability to Microsoft before the bug bounties was 12-point aerial font of their name in a bulletin. If it was fixed during the beta period, there was no bulletin. So there was no incentive for them to report bugs early in the beta section of the release of the software. So we projected that we would actually move the spike, move the traffic if we did a bug bounty in the first 30 days of the IE11 beta period. And the actual results, 18 serious vulnerabilities were reported. They were bulletin class vulnerabilities. Each of them could have fetched on their own over $100,000 on the offense market. However, the offense market wasn't buying during that time because who would buy a bug that could disappear during the beta period? So anyone want to guess how much money I spent to get 18 bulletin class vulnerabilities in the first 30 days of the IE11 beta period? Okay, I'll tell you, about $27,000 total. That was it. So let's move on to hack the Pentagon. We were hoping in the participation of this program that we would get a few hundred people to come forward and want to participate to help secure the Pentagon in a first ever pilot program. The reason this was so significant was not just because this was the first time that the U.S. government was going to pay hackers to hack them, but it was the first time it had been legal since the Computer Fraud and Abuse Act for anyone who saw something to say something to the United States government to help secure our systems. So this was very significant, but we got more than a few hundred. We got 1,400 eligible participants registering over 1,000 reports in a 21-day period narrowing down to about 138 valid security issues. And the time it took for them to start 13 minutes after midnight, note to you, don't start a bug bounty program at midnight. So then what happened? In November, we launched the next bug bounty program of the U.S. government. This is Hack the Army. By the way, the coins that the participants get for participating in these programs are they have little binary encoded messages on them for the hackers. For hack the Pentagon, it's translated to I hacked the Pentagon. For hack the Army, it translated to I hacked the Army, beat Navy. So, and apparently it worked. So, what did we learn from this and how did we scope this down? Remember, we got 1,400 eligible participants in the first round of this pilot. Well, we scoped it down to limiting the participation to 400. The total reports received, the signal to noise was much improved. Still, only took five minutes to get that first report. Thankfully, we didn't start it at midnight this time. But learning from this, it's that, yes, you can receive vulnerability reports if you entice them with a nifty coin for love of God in country and a little bit of money, but what are you going to do on the back end? How are you going to process all of these things? So, what's coming next? Are we going to be overwhelmed with a tidal wave of bug bounty programs in 2017? It sure looks like it. However, I mean, if you want to get a flood of vulnerabilities that you may or may not be able to process, you could certainly start with a bug bounty program. You would get a lot of bugs. They may not be the bugs you're most interested in. They may be a whole bunch of duplicate bugs that you could have found in a more cost-effective way, either by hiring your own folks or running some tools yourselves or essentially not offering a bug bounty, but just opening the front door to receive vulnerability reports. I don't know if you caught it on one of my slides earlier, 94% of the Forbes Global 2000 lack the official means to report a vulnerability to them. These are folks that are spending millions on security, but they simply lack a front door and a process by which to receive their bugs. So, what are we going to do in order to solve this problem, not just for right now, but for the internet of the future? Supply chain, internet of things, cars, pacemakers. Well, these are complex vulnerability disclosure scenarios. Often a car is not just a car. It is a collection of intellectual property that works together. It is not a simple thing to just dangle some money in front of a car and say, go ahead and hack this. You have to have the mechanisms and the communication channels to actually coordinate those bugs to resolution in a fully functional device. We have had an internet of thing for a very long time that has been quite difficult to secure and these are our mobile phones. We haven't been able to solve this problem thus far. So, what is the answer? Well, we hear about maturity models all the time in terms of secure code development. We hear about maturity models when it comes to dealing with breaches and your readiness. We have not up until this last year heard about maturity models when it comes to receiving incoming vulnerability reports and your ability to get through that process, understand it, triage it and decide on a fix and release it. So, I do have a vulnerability coordination maturity model, which I'm not going to go through in two minutes and 30 seconds, but avoiding the pitfalls of jumping on to a bug bounty bandwagon before you are ready to actually fix any of the bugs that you have and that you could find out about through other means is basically planning on running a marathon immediately following a coma. I mean, this is a bad idea. So, what's next? I just came back from the UK and went through my very, very first secondary screening in immigration. No relation to the fact that I was just at GCHQ. I think that's fine. So, what we just launched together, what we just announced is a vulnerability coordination pilot for the UK government. This is a pilot to refine their process maturity with an invited set of trusted security researchers because they know they have bugs. They know they want to receive bugs from the outside, but what they want is to understand where their process breaks down so that they can scale it in a mature way. Leave it to the Queen to think of a way to get through this problem in a much more dignified manner. They did, by the way, say that the most important thing that people can do, this was the CEO of NCSC, the National Cybersecurity Center in the UK, did say on stage at his own conference, patch your damn bugs. That was his advice to everyone to secure things, but they understand that this is a process and it is a growth of maturity that will lead the way. So, someone is eventually going to disclose a bug to you or try. It's a matter of being prepared that's more important than paying for vulnerability. So, building a process for mature vulnerability coordination and fixing the bugs that are reported is more important than any kind of monetary incentive you can think of, no matter how trendy it sounds. So, bug bounties are not going to replace other security testing, and I would venture to say that we don't have a bug problem. We have a patching problem. To open it up to hackers, helpful hackers who want to tell you about your bugs is only part of the equation. The other part is dealing with it. And the last thing I'll say is that it's wonderful to be here at New America, and we are thinking about solutions to bring the future to cybersecurity in America, but the whole Internet is a place where we have to operate as defenders without borders. Hackers come from all over the place, from all different backgrounds, and we have to be prepared to hear from them no matter who they are and where they come from. Thank you.