 So, welcome. Thank you for joining us this morning at super not late or early, but... Somebody followed the 3-2-1 rule to the letter last night, didn't they? Sorry, my bad. Hey, but at least I got three. So, in case you don't know where you are, we're doing voles 101 in here. Hopefully everyone can learn a little bit of something. We all wanted to ask a couple of questions before we jump in further. Number one, like, how many people here is their first DEF CON? Wow. All right, nice. That's pretty impressive. So, how many... What was the other question we wanted to ask? So, how many of you here are here for this talk? You've just started out in vulnerability research and you want to learn how to improve your game. Okay, and how many of you are not familiar with research, but you're curious about it and so that's why you came here. That's about even. Okay. Good. That doesn't help us any, but thanks. Yeah, what we realized a little bit earlier this week as we were practicing was that we were trying to target two audiences, newbies and then people who were curious about vulnerability research but didn't know much about it. And so, sometimes for those who are curious about it, we might go into some terminology you might not be fully familiar with. We're just going to move on. We'll be glad to follow up with you afterwards as well. Yeah, and neither of us hide on the Internet. We're pretty approachable, so don't be afraid of us. So, introduction. I'm Josh Jduck. I go by on the Internet. That's how you find me there. I've been doing VR for 20 years and so this 20 years actually includes several years as a hobbyist. And at one point, I ran the high defense vulnerability contributor program where we would actually buy vulnerabilities from researchers and get them fixed and coordinate and all that stuff, so that's how I met Steve and we did a fair amount of work together at that point. And we met through the CVE program which I was a co-founder of and led from 1999 until the end of last year basically. There's a thing called Responsible Disclosure. I am a survivor of the Responsible Disclosure Wars. I now call them Coordinated Disclosure. I coined the Responsible Disclosure term for which I will be eternally sorry, but it served its purpose. I started getting into classification of vulnerabilities as well which is where CWE comes from and I was also a participant in the development of CVSS version 2. One quick question. How many people in here have a CVE to their name? Alright, let's fix that. That looks like a dozen, I think. That's pretty good. So why are we doing this? We want to fix that, right? We want more people out there doing research into vulnerabilities of software and hardware into all kinds of systems because as we've seen, there's lots of crazy things possible. It was very interesting to see the previous talk, the guy hacking the loyalty program. That's good stuff. It's fun. So what else? We have this little tiny picture on the slide on here. You guys see the slides better than we do. So we're flying a little bit blind. If anybody works at Google Slides, you might need to work on this. We can move on. Alright, yeah. So just to get people involved. So disclaimer up front, right? This is our opinion. We did a lot of stuff with Volums over the years. Find them, analyze them, all that stuff. We just formulated these opinions. So they may not be right for you. Take that with a grain of salt and remember that you're your own person and you've got to find your own path. We'll just try to help you see some of the stuff. So lastly, there's no new exploits here. Who came here to see new exploits? Okay, good. Thank you. Okay, so first of all, there's a question about what is a vulnerability in the first place? We'll start with what a vulnerability is not. One of the most commonly confused terms is exploit versus vulnerability. And a lot of people think that exploit and vulnerability actually mean the same thing. However, they don't. And exploit is really a sequence of steps that's used to take advantage of a vulnerability. A vulnerability is a problem within the code itself that just kind of more or less sits there waiting to be exploited. These are almost circular sounding terms, but we face a number of difficulties in actually really defining these a little bit more carefully. And I think that's a reflection partially of the relative immaturity of the vulnerability research specialty. Yeah, I wish I could expand this picture for you, but I think what is Taylor Swift say here? To be loved is to be vulnerable. To love is to be vulnerable and to be loved is the greatest exploit. So we go back and forth with this definition of what is a vulnerability. And there's so many different ways to define it. I think Steve did a great job already saying it. One of the biggest things really is that you have some kind of impact on a system. If you don't have some kind of impact or you're not changing the way things are working, it's pretty much not a vulnerability. But in mind, Greg McManus at iDefense taught me through heavy abuse of him asking me this question every time I thought of him when I thought I found something. He's like, well, what do you have and what do you get? If what you are getting from this vulnerability, if you manage to exploit it, it's not better in some way than what you started with than you do not have a vulnerability. It's more like a bug or something really annoying. And this is one of the common problems we run across with new researchers who try and report CVEs or something like that. They find something that may be a bug or may actually be a feature, but it's something that is legitimately already allowed. Or with the privileges you already have, you can already go legitimately through some other route to obtain those additional capabilities. And so that's one particular point of confusion. Another important point to make about vulnerabilities is, as Steve knows well with this classification work in CWE, that there are many, many properties of vulnerabilities. Like I mentioned before, the impact is a really important one, but also the user interaction is an interesting one. These things are used by the defensive side to prioritize patching and strategies around defense. So these are just a couple of the important properties. Another interesting one these days, which is getting more and more interesting, is how hard is this thing actually to exploit? And I think as things have improved over the years, it's gotten much harder to do that. Yes, it's become a lot more difficult to do that. And that's thanks to the defensive work and the build-up of many different kinds of protection mechanisms. And so there's almost like a Heisenbergian approach to interpreting vulnerabilities these days. And where something was clearly exploitable perhaps 10 years ago, it may wind up taking a whole lot of work. And that's one of the great things about the defensive side of understanding vulnerabilities, which is to really build in these systematic defenses. I have a whole rant on this. We can do some other time, but not right now. We'll save that for later. All right. So finally, we get to what is vulnerability research. And in this case, what we're saying is the process of analyzing a product protocol or algorithm, you guys already read this. Or a specification to basically to try and find vulnerabilities or understand them, one or more vulnerabilities basically. So there are different kind of approaches, different kinds of products or specifications and so on that you may decide to look at. It's all more or less falls under the umbrella term of vulnerability research. So the term itself is treated and interpreted a little bit differently. You might sometimes hear the term vulnerability discovery. And that's really intended by, used by people who want to distinguish from let's say academic strength research versus going and doing bug hunting and so on. So some people use that term. That distinction does wind up being important, I think sometimes. But again, the terminology is still kind of emerging and we don't have a lot of agreement. Well again, we're not doing exploits here and I personally think exploits and exploit development and stuff like that falls into VR. As do I. But it's not really our focus here. So let's keep going. It's really about solving puzzles where you don't even know what the puzzle is in the first place. You don't even know if you'll find a puzzle. Then maybe you find a puzzle which leads you to other puzzles and so on. That's one of the big attractions to me for vulnerability research. Alright, so why do it? You know, in case you're very new and you're just curious, these are some reasons why you might want to get motivated to do the hard work that is vulnerability research. Hooray, Google Slides. I'm serious, it's so tiny. I hope there aren't any Google... The speaker notes are the whole screen and the slide is like a little tiny square. I hope there aren't any Google people in the room, but... I hope there are some to fix this stuff. So one of the big points to note here as you look at this nice little word cloud is... Go ahead. Ah, that's okay. Let's just go to the next slide. The main takeaway from that previous slide is that there are many different motivations that different researchers have. And your motivations may not be the same as others. And in addition, when you're dealing with vendors, vendors may have only experienced or they may only assume certain kinds of motivations from you. And so that potentially causes certain difficulties when interacting with vendors. I personally like just about all these words on here. So there's a lot of different careers. Steve, you want to talk to them? Yeah, so there's a number of different careers, but it's not like there's a career shop that you can go to. This is still a new field. And to my way of thinking, we're like kind of entering the second generation and we're sort of the first generation. We're trying not to skip one. We're trying. And it's really good to see a full room here, actually, because we need a lot more people doing vulnerability research because we've only seen the tip of the iceberg. But what's called vulnerability research, it may vary, but there's a lot of different things that one can do. Yes, you can go and you can hunt bugs and hunt vulnerabilities. Other people may really like building up exploits against those vulnerabilities. And I agree with Josh that that's an aspect of vulnerability research. Then there's other stuff that's maybe a little bit more adjacent to vulnerabilities, but that still involves a lot of analysis. So I previously mentioned things like building defensive protection mechanisms to help protect against entire classes of vulnerabilities. You know, for CVE, a lot of what we did was to catalog those vulnerabilities and in a way try and figure out how the CVE identifier could be used to help people to coordinate and so on. And another option is to really work on fixing them. You could be working at a developer somewhere and see a vulnerability that's been reported and figure out actually how to fix the code. And I guess one note that just popped in my mind here is that there are a lot of vulnerability researchers who discover vulnerabilities but have no idea actually how to fix them. It's a completely different mindset. You can have a really solid development background. You need a solid development background to build a good fix, but that's not necessarily the kind of skill you need to do vulnerability research. Yeah, that's right. One thing I will add is that if you do start doing any of this stuff or find a job doing this stuff, you basically have unlimited job security for the foreseeable future. So there's a lot of different employers that you basically could have. I mentioned you could work at a software vendor. You could work for a government organization or a cert coordinator. You could work at a commercial enterprise, whether it develops security products or whether it does consulting services. However, these days pretty much every business that's out there is more or less a software developer. Think about Target. Think about other kinds of brick-and-mortar companies. Those all develop software, either in-house to help them manage their operations or externally with respect to websites and so on. And as you all have probably heard, there is a huge demand. And so these are some of the options that you could look at and you would be most likely welcomed with open arms by someone somewhere, because we need a lot more researchers. I'm a little partial to the bounty programs. They're really fun. And if you have a good employer like I'm fortunate enough to do, then you can kind of keep that bonus money and throw parties at b-sides and stuff like that. Cheers. So, yeah, let's do the next one, huh? So these next couple slides, these are really kind of a little bit more of a disclaimer than normal in terms of our opinion. But based on our personal experiences, and I've actually done some vulnerability research myself. It was a while ago. But based on our personal experiences and interacting with other researchers, there are a number of personality traits that generally seem to be useful for longer-term success within vulnerability research. So for example, a willingness to work independently, a willingness to learn, being very critical thinking. You always have to be more or less questioning your own assumptions. That's a good point. I don't even think that's in the slides, but that's a really good point. And really, it's primarily a solitary effort. You need to be diligent. And you see some of the other features there, basically. But two of the biggest personality traits that we believe are important are patience and persistence. Patience is essential not only with yourself and with the process of discovering and investigating these vulnerabilities, but patience when dealing with others, especially when dealing with, say, vendors that might not necessarily be behaving exactly the way that you would want to when you're trying to communicate with them. So those are some of the should-have personality traits. These are some of the ones that we think are nice to have. Still a greater formula for success here. To really be able to be focused, to seek to improve software, which is a common thing. The ability to collaborate, whether that's to collaborate and work with other people is something that we believe is important. There can be many rock stars and not-so-rock stars that don't work well with other people. But that oftentimes, especially if you're just starting out, I think is probably a career limiting kind of attitude that one would take. And we also have here a notion of having kind of an addictive personality. So for example, at CVE, I stayed at CVE for 16 years through 70,000 vulnerabilities. Now, I didn't investigate and look at all of them, but you could say that that might be kind of indicative of an addictive sort of personality. And Josh, how many days, weeks, or months have you spent on, say, a single bug? I don't know. I stayed for, I spent goings for a long time. I think it's at like maybe one year, but yeah, it's not all at the same time. So, you know, none of these personality traits here that we're talking about is absolutely essential. Each of you will find your own path. But if you feel that you have some of these personality traits, then you might find vulnerability research enjoyable. So I think we're going to be totally screwed on this slide because it's a small one over there. We have a number of different... You guys could probably read it fine. We can't read it at all. We have different skills that we've sort of listed here for long-term success, but I would say probably that some of the biggest ones, one is about analysis tools and findings. Not this one, is it? Yep. We can skip it on the next one too. All right. I think the big one we wanted to say here was about communication. I think we made that pretty clear. Yeah. All right. So here's another awesome wall of text slide that we put together. And we don't want to read it to you, but these are some of the key terms that we feel in vulnerability research. Of course, the slides will be available. You know, if you hear us, you probably already heard us use some of these terms. But when it really comes into doing analysis and deeper research, like some of the stuff like root cause analysis and vulnerability chains and classes, and especially proof of concept code, become more important. I think one of the key terms here that I guess we touch on it a little bit later as well, though, is the notion of root cause analysis. This is where diligence and critical thinking comes into play. You might discover something that's like a symptom of a problem. And it's really when you become tenacious and dig deeper and deeper into it to find out what's really causing the problem in the first place, where you may find some significant success. All right. So in the industry, many of you probably if you're interested in vulnerability research already know about this thing we call the fire hose. And that's basically just a steady stream of information about vulnerabilities that's coming from all angles. Includes stuff like some of my favorite stuff like CTFs in war games, where you can learn, excuse me, at your own pace. And just lots of aggregation in other places. If you want to learn more about violence, look at these things for sure. There are a couple items that are not on that list there that I think that came up during this weekend. One of them actually is the Pony Awards, because the Pony Awards often talk about, you know, individual bugs. And typically those additional bugs have additional details. And then another area is bug bounty programs, which can help you to learn and interact with others. Actually, by a show of hands, how many people are in or have participated in bug bounty programs and gotten some kind of reward? It's better not to be too much more than how many people have CBEs. Moving on. I guess there is that rule about CBEs and websites, right? So, um, wow. I'm going to bend over and stare really kindly at it. So selecting your target, there's a lot of choices if you want to find bugs somewhere. This is kind of on the vulnerability discovery side. You can go deep or you can go broad, and what we mean by that is you can pick one particular type of vulnerability or something and go look at every software you can find to see if it's vulnerable, or you could pick one particular software and just drill down until you find something. There is a lot of software that has more or less low-hanging fruit, and if you want to expand on that a little. I will on that. I don't know if it's on this side or the next one. So, uh, another big point I wanted to make here is if you do do some research and you find nothing, it's actually quite useful for people to know that somebody looked, even if you found nothing. So that's one point. And again, low-hanging fruit, a lot of older code is buggy, complex or overly complex stuff is very interesting to look at, although, you know, a lot of times you just get lost in it just like the developer did. Large attack surfaces like web browsers are always fun to play with. You've got a lot of possibility for things to go wrong there. Software popularity matters. So if you're going to try to become super famous and you want to go find some bone and something, it's probably better to not pick a random personal website project off of SourceForge or something like that. But on the other hand, if you want to find something and, you know, a super popular product like Microsoft Windows Server or something, it's probably going to not be anywhere near as easy. Not necessarily anywhere as easy because that software, you know, the really popular software has already been pounded on by many people, by elite researchers and so on. So the lower-hanging fruit, the kind of software that doesn't necessarily have that doesn't necessarily have any vulnerability history at all or that no one's really looked at before, that's often an area where you can find some success very quickly. One thing I like to do sometimes when I get super stuck is to go and pick on somebody lame. I think this is kind of popular in the VR industry where we just need that redemption, we feel good about ourselves again. But the problem with that is, you know, a good example is like open office or something, it's pretty easy to fuzz that and it's full of bugs and nobody really cares about them too much. And so you can go find bugs there but then you deal with the secondary problem as nobody caring too much. So brand new emerging technologies is always a great place to look. Many people in bone research like to wait until a thing becomes very popular and therefore when things are emerging, nobody's really paying attention. I think we can say that about IPv6, I think there's like maybe a handful of IPv6 researchers around even though that's sort of slowly becoming a norm. Mobile and IoT are definitely guilty of this because as they try to hurry up and get to market really fast, they didn't invest in security and we're hoping that we don't repeat that mistake with IoT but we'll see. One suggestion we do have which would be very useful for the entire community and for contributing to the body of knowledge is that you have access to software or products that are very difficult for the everyday person to get access to. Say, you know, multi-million software or expensive medical devices or other kind of physical devices you know, those aren't things that just everybody can go and grab and look at. So not only might you have some good chances of success in finding vulnerabilities in those kind of products, not a lot of people have access. Who knows how to do that like magnifying glass thing on OSX? Anyone? Nobody? You want coffee or something? Well, I'll just stare at it really small again. So, something that I've seen a lot and that Steve kind of coined a term here on this pig pile effect. It's pretty interesting. That's one way to select your target. You see people beating up on something through advisories getting published and you're like, well hey, maybe I feel like there might be more there. I should go take a look there and maybe do some follow-on work. I encourage the community with this one on stage right heavily. It's good to have more people looking. So tools and techniques there's kind of like these two main ones that are really kind of high level design review and threat modeling. These are I think really important for anyone who's developing software to have this as part of the cycle of figuring out how to stay secure or how to basically stop having alarm bells ringing all the time. Dynamic versus static analysis is very important to differentiate depending on what kind of stuff you're going to do. Like on the malware side, static analysis is a lot more popular with vulnerability research a lot of more dynamic analysis seems to be more popular but I think the real power here is when you have both together one of my personal bug hunting processes is to start writing a fuzzer and then just let it run while I read the code and as soon as I learn something more about the code that will help the fuzzer be good then I'll add it to the fuzzer and I'll just keep doing that back and forth. So code auditing in some of these other automated tools like static code and analyzers, they're great but a lot of times they have false positives or they have other issues and so it's just important to be aware of the trade-offs of kind of all of the tools and techniques when you start getting into them. I really think that a tool in this industry is the embodiment of a technique someone developed to a large degree. And I agree with that. While we have a number of tools and techniques listed here that doesn't mean that you have to know all of them and be expert in all of them in order to find any kind of success. This is part of your path but we do recommend I would say to at least investigate and look into each of these. Everyone kind of has their own sort of favorite techniques that they like to do. This one's you man. So as the field becomes a little bit more mature and ties in obviously with vulnerability management overall, there's a number of different relevant standards that you should familiarize yourself with and utilize wherever you can. One of the main ones is the common identification scheme for vulnerability is called CVE and for those of you who've had certain questions about CVE especially in the last year or so with concerns about coverage and what MITRE is doing while I did leave CVE last year, I'm still at MITRE and we do have one of my colleagues here who is carrying the torch as it were and would love to talk to you and so I want him to stand up here. That's Dan. He will be Hey, Stan. We need to talk buddy. So he will be available and he would love to talk to you. Not all of you at once but a few at a time. Another effort is the common weakness enumeration. This is when you have these different vulnerabilities and different products well it turns out that programmers make the same mistakes and many different programmers make the same mistakes and so CVE is effectively a classification scheme for how programmers wind up making these kinds of mistakes. It's useful in two different ways. One as sort of a common identifier for characterizing what the mistake is that you found but it also winds up being very useful as a dictionary or as something to educate you. So for example CVE there's 800 different kinds of mistakes that programmers can make and as much as you think you might know about everything I guarantee you there's one or two things in there that might surprise you or that you might not have expected and if you're even just starting out you get good information from things such as OWASP but CVEs as well for stuff like SQL injection and cross-site scripting are also pretty mature. Equivalent for attacks is called KPEC and then CVSS is a way of being able to consistently apply a risk-related score to a particular vulnerability that you found so it may be your favorite vulnerability you might be in love with it you might have worked really hard but you need the cold objective or reasonably objective eye of CVSS or something like that so that you can communicate its importance effectively. Thank you. Hey sorry about messing with the slides I was trying to zoom in on this little tiny thing and it zoomed everything so that doesn't work. So disclosure models disclosure, disclosure, disclosure we're not going to go into specific details but there are a number of different models that you could consider and figure out more or less what works for you I do and I think Josh agrees that we both suggest using a coordinated disclosure model which involves really working with the vendor in order to try and reach some resolution but there are other models as well such as full disclosure as soon as you find it you sort of put it out independent of whether or not the vendor has been given a chance to patch and then there's also non-disclosure some people may choose simply not to disclose the vulnerabilities or to only provide them or in some cases sell them limited markets but this is these are different things that you're going to need to consider as you move more into vulnerability research. You may have any number of different approaches and beliefs in why it's important to do public disclosure but I think the more we know the better we know all of us collectively and finally there are a couple different standards or standards like documents that have some guidance with respect to coordinated disclosure or equivalent models that you can follow and that you can provide or give advice to vendors who may not be used to handling vulnerabilities. Most importantly is International Standards Organization ISO document number 29147 which was done by Katie Masaurus and others International Standard it is something that is directed towards researchers which explains to them how to build up a process for responding to vulnerability reports and for interacting with the researchers so as a survivor of the disclosure wars I'm very very happy to see standards like that 29147 come out and yes it did take me six months before I could start rattling off that number if you start to get deeper into building your vulnerability career so to speak then you may have different kinds of considerations for building your own kind of public your own disclosure policy based on your own experiences in your own beliefs you want to start evolving certain kind of considerations for what you're going to do in certain kind of circumstances so what would you do if you try and contact a vendor and you can't find any contact information or what happens if you're working through a process and then suddenly the vulnerability is released by somebody else as a zero day or something like that there's a lot of debate about how long to actually give the vendor before they fix the vulnerability and push it out push out a patch some say 30 days some say 60 days there's 90 days or however long it takes these are some of the questions you need to ask yourself yeah what I had a point I forgot what it was let's skip this one I think I was just going to say that sometimes the disclosure process you choose will even vary based on individual vulnerabilities some people decide not to disclose things that are not super awesome alright so we got 10 minutes we lost a little time to the difficulty let's we can move on skip that one alright so let's talk a little bit about advisory structure and contents I'm so not going to read these bullets to you structured content is very useful I think Steve and I collectively well we probably read a lot of the same advisories but collectively he's probably read thousands and thousands I agree with that some of them were really horrible there's this really offensive group and when I say offensive I mean when we read it we get offended I don't mean they use bad words or anything not to name names but I'm not naming any names if anybody reads advisories they'll figure it out so these are just some fields and some guidance that we have for making advisories and of course there's some more here one of the big ones is proof concept code I think it's a really important sort of thing to prove your case you know when you when you disclose a vulnerability to a vendor a lot of times you get pushed back like there's not even a real issue here and of course on the metasploit side of things it is a little bit hard to argue with a shell but you don't necessarily have to give somebody a shell as your proof of concept it could be whatever you choose it could be a sequence of steps that they manage to follow to verify it it could be a kind of level of maturity proof of concept code crash proof of concepts but do remember like the more sort of detailed information you can learn and extract and provide to them the easier it's going to be for them to deal with that information so one of the reasons that we have these particular details here about advisory contents is that a lot of researchers especially beginning researchers don't necessarily know what information to provide or you might submit a bug report to a bug bounty and it comes back and says you're not providing enough information or you're not communicating clearly so those fields that we have listed and you know are on the slides and will be on an updated version of this slide we encourage you to really look at those and consider seriously capturing all of that information there's some pros and cons that we came up with this basically just Steve and I kind of ranting on all the stuff we didn't like about various advisories you know we want people to do simpler simple stuff plain text is real easy it's very portable it has a very low attack surface so as opposed to say pdf yeah which is like a web browser basically and some people do like to do videos and we think that's kind of cool but I have a couple suggestions here one of the main ones being is you know respect the viewer of the videos you don't want to make your videos too long but you don't want to go too quickly either and there's a couple other considerations up there as well so you want to be mindful even of the format in which your advisory goes out so what to expect from vendors I already mentioned some of this stuff you can expect total cluelessness you can expect in some cases for people to threaten you with their legal teams I don't know why they do this I think they're confused but these are just a bunch of possibilities basically every most good companies these days especially the bigger ones they're very open to work with and actually amazingly even sometimes a new vendor that's never had to deal with these problems comes out actually understanding it quite easily and being very good to work with as well but one thing to keep in mind is that it's not like one size fits all all the time and every disclosure winds up being some kind of unique snowflake so you need to be able to be patient as mentioned before and also be able to be flexible keep an open mind so I think the first bullet like one time at iDefense we were trying to report this phone and I tried phone calls I tried email well on the other order of course tried emails and then phone calls and finally I got a response from a 30 guy there when I faxed them the advisory and what year was that it was probably 2007 so they were like oh this came out of the fax machine let's call them okay so where do we disclose publicly what we like to see is people to disclose publicly in places that are archived forever so that becomes mailing lists basically and these other things on the list like exploit db's and vulnerability databases those sites are great and we'll hope they'll live forever but in some cases in the past they have not lived forever and ultimately those sites generally are pulling from more public sites that are archived forever anyway so this is just our preference if you want to put your stuff on your blog to gain readership that's great but maybe also throw a note up on one of these lists and get the traffic to your blog as well so common mistakes to avoid number one is don't test other people's stuff unless they let you I think there was one case with facebook bug bounty program where a guy like owned a hell out of them basically and then tried to do a bug bounty with them and that I don't think worked out really well for that guy we have here on this slide in the next one we aren't going to go into details especially because we're running low on time but there are a lot of common mistakes that many researchers make including ourselves actually and so why don't we move ahead if you guys want to hear some more stories just hit us up we have some real meat to the at the tail end of this presentation okay so this is one of the one of the main ones here this is just sort of our own model first crack at this as far as we know nobody else has really sort of started on this but we're trying to outline different kinds of stages of growth that you might encounter in your career or in your technical abilities when you're doing vulnerability research so when you're just starting out you're starting at more or less a newbie phase you might have you might know just like one crude technique that you apply against software and maybe you make a lot of different mistakes at some point once you become more familiar with things you may reach what we call a work horse stage where you know a number of different kinds of basic vulnerability types you can generally find multiple issues and then you start to really more or less get a hang of certain kind of processes then when you start to move more towards the subject matter expert these are the these are the times when you're really looking for the newest and latest techniques that other people develop or you might you might go and extend those techniques that have already been reported and you're more or less pretty much treated or assumed as being reliable by people like me and so on and at some point you start to really have a clear sense of like what your own disclosure policy is and you can be relied on to find a lot of things and to write a really good solid quality report and finally upon reaching the elite stage which not everyone needs to reach the elite stage and not everyone has to and not everyone wants to and that's perfectly cool because there are way more vulnerabilities out there than there are vulnerability researchers to handle them but what I think of a little bit of as an elite researcher is you discover or invent new ability classes or you develop entirely new techniques or you know you give conference talks and so on you really sort of push the industry forward and that can take a number of years you're not just going to read a book or look at a couple of blog posts and be elite tomorrow or next year so that's something to keep in mind as well yeah it takes time for sure I'll read this again that's got my name on it anyway I think we got only a minute or something like that so one thing about the notion of growth is that there is a book by Malcolm Gladwell called Outliers which basically says it takes around 10,000 hours of focused effective practice in order to reach a level of expertise and so you can do the math and it can be questionable but that's something really to keep in mind but there are a couple different ways that you could progress a little bit further if you want I think we have like three slides left do you want to do this one or do you want me to do it it's got your name on it I'll do it if you want are you getting tired no we're getting an X from the goons get out of here guys you talk too much so we just wanted to leave on this note about feelings and fails we mentioned we are not perfect I believe there's this thing this is what I call the human condition which basically means you always make mistakes and have to deal with things that your body tells you and such but feelings are definitely a part of that so remember feelings are okay there are a number of times when you're doing some deep research and you get very discouraged you might want to find something easy to do you may want to go at it in a different way and maybe just go to the beach for a while another one of these things is you feel like you really got to keep going and you want to work really hard because you're addicted to something but it's been like 17 hours that you've been working on it it might be a good time to sleep so yeah feelings are okay and failures are okay too here we go thank you everyone and we are available to talk to anybody afterwards we're going to go down to escalator somewhere there