 I'm Greg Conti, and I teach computer science and computer security at West Point. And I hope to do several things with this talk. One is to explore the process of conducting research, because I think a lot of people in the community are doing exceptional work, and it's something we as a community need to think about how we do it, and maybe there's some lessons learned we can learn from each other. For those new or thinking about it, hopefully this will inspire you to conduct research. And for those of you that have a great deal of experience in conducting research, hopefully there's some things in here that are new to you, many of which came through hard knocks. And while I've been doing security research for the past decade, I'm not claiming to have all the answers, but I do claim that I've spent a good bit of time thinking about this and helping other people do it as well. So again, I put a lot of effort into this, and hopefully you'll leave with some tools and techniques that you can apply to your own work. I'm here as a free citizen and not a representative of any government. So what is research? Research to me is the search for knowledge, trying to find something new, something that hasn't been discovered before, maybe to solve a problem. Sometimes it's done rigorously using the scientific method. Sometimes it's done informally. And here's how I like to think about it. If you look at the gray sphere in the lower left, that line, I think of that as a present. And then you can look out 10, 50 years. And going back a little into the past, things that we've already discovered, we have things like books. It's well-synthesized information, exists in libraries, and then college courses. And then as you approach the edge of human knowledge, you hit graduate courses. You hit talks at Black Hat or DEF CON, and they appear at the edge. And those could also be research papers. And the idea then is to not rediscover fire. The idea is to find something someone else hasn't done and fill in the blank and push out collectively the edge of human knowledge. Unfortunately, there are pockets of knowledge out there that we don't have access to. It could be hidden behind paywalls. It could be in classified archives. You may have heard of GCHQs, claimed to have discovered public key encryption. It could be proprietary information that isn't shared. But the idea, though, is if you'd make a discovery and you're at the edge, you have this ability to look outward and predict, and you see the problems. Look farther than anyone else has seen, and identify other problems that can be solved, other questions that can be answered. And then even farther out, you have science fiction authors, maybe 50, 100, 1,000 years out that are just speculating about the future. And I think there are many science fiction fans in the audience. I know I am. And you can draw great inspiration from those authors. So why conduct research at all? Well, I mean, it's in many ways. To me, it's about advancing human knowledge. You're doing something that we benefit from others' work, and we're giving something back. And one of the most pleasurable experiences is doing something and seeing someone else build upon it. And along the way, you can make yourself an expert in a given area because you've specialized in it. And you pick up new skills, and frankly, it's fun. Hackers in particular have a great deal to offer when conducting research. First, let me say it's great to have resources like Wikimedia Commons that you can search on Coloring Book and find pictures of someone who's uploaded their child's photo of coloring outside the lines royalty-free. So it's pretty cool that they make that available. But what hackers bring to the table is this native curiosity. Folks are clever in the hacker community. They like to color outside the lines. Often, they do excellent work. And importantly, they have less constraints. If you think of industry, they have to really show value to the bottom line. In academia, they tend to be like dogs on the hunt, sniffing for the latest research grants. But the hacker community can choose things that are fun and interesting and just important to them. So this sounds pretentious, the idea of seeking to be the world expert, but it really isn't. I'd argue that we've got probably 100 world experts sitting in this room in some particular area and many well on their way. And those are the people that are at the edge. They can see important problems and either work on themselves or help tip off somebody else that maybe you want to take a look at this problem. And Malcolm Gladwell in Outliers came up, he did a spouted research that said there's like a magic number of how much effort you need to put in to become world class in something. And the magic amount of time was 10,000 hours. And there's people in this room that have put 10,000 hours in and probably some with 20 and some on that path. Because people, you know, if you get it, you get it and you're into it, you put a tremendous amount of time in and you gain momentum. As a Dungeons and Dragons player, I can't help but think in terms of experience points. That you can single class and put all your experience points into one area to get depth. Or you can multi-class and put your experience points into many different areas. So you progress farther, but you gain breath. And in research, you're focusing in one particular area or a small group of areas, but it's also important to balance that with breath. So you understand the context of what you're doing. And if you get a little bit off, in academia, I see that if you get a little bit off someone's specialty area, they don't know what they're talking about because they lack breath. So it's a balance that we seek. So let's look at some strategies for finding problems. Because sometimes that's the hardest thing. How do you find an important problem or something that you think you can get traction on? One strategy is challenging assumptions. And I would say the hacker community is very good at that. There's things that we think about or don't think about that oftentimes should be challenged. One is the idea, perhaps, of a hard drive wipe. It's conventional wisdom that you wipe a hard drive and all of a sudden, if you have a malware infestation, it's gone. Well, maybe it'd be interesting to dig more deeply and say, well, what can live past a hard drive wipe somewhere in the system of the computer? Another is to think very big. And you've seen things like Dan Kaminski will scan the entire internet. And this is Cooperative Association for Internet Data Analysis. They did an entire two month survey of the entire IPv4 space. You can also think very small. And these are showing the individualizations of individual bytes within files. And thinking very, very big, thinking very, very small are interesting ways to approach problems. And a classic technique. Irritating software, hardware, protocols, and people in ways that they weren't designed to handle. And that's often a fruitful response. You can try and detect patterns. We all have news sources, information flows that we bring in, books that we read, people that we talk to, that we, over time, though, you sense patterns. You sense things, emerging patterns that might be important to look at. For example, one might be Bitcoin. You might be watching it and you see enough samples come by in the news sources and you'd say, this is important for me to take a look at. Another is to try and sense a need. I wanted to buy a book, I wanted to buy this book. And I went on and found out that it's no longer in print. And that the two copies available on Amazon started at $679. So there's some used booksellers making some money if anyone buys books at that price. But the idea is you can sense a need and maybe that's an opportunity of where you can explore some more. Another powerful technique is to look at the intersection of your interest areas, your expertise areas. For me, HCI and security, I took a look at that intersection. And you can see things that popped out were malicious interface design. Because your average HCI person doesn't think about an adversary or the designer as an adversary. And so it's useful to sketch out Venn diagrams of what you're interested in and look at the intersections. Another would be just crazy intersections. For example, I have a friend who has carpal tunnel and also likes to use nunchucks. This crazy combination, but there's something here that she told me that when she uses nunchucks, her carpal tunnel gets better. I don't know, it's a data point of one. But it might be worth a little bit of exploration to see if that's actually a technique to help improve your carpal tunnel. It may be illegal in the state of New York and other states, but it might be something worth exploring. Or maybe it's like medical use marijuana. You could have nunchucks if you had carpal tunnel syndrome. Another strategy is what makes you mad? Do ice cubes fall on the floor when you try and get a cup of ice out of your refrigerator? Maybe you can look at doing that better. For me, a tipping point in that malicious interface research was when a flying vodka bottle appeared flying over the weather when I was trying to just see what the weather was. And a man came out at the bottom and toasted. And then that tips you off. Or maybe it's a flicker in the status bar of your browser. And you see these third-party sites. And maybe that tips you off to something that you can explore more deeply. So what makes you mad? Certain technologies, you just have to ask what could possibly go wrong? This is a self-wiping hard drive from Toshiba. I don't know if anything's wrong with it, but it just sure sounds like something bad could happen. Another is a new file format from Wolfram Research that claims it's as every day as a document, but as interactive as an application. To me, that's a combination that might merit further exploration. Because what could possibly go wrong? And if you're short of ideas on what could possibly go wrong, you could go to slash. And people will tag things for you to give you ideas. Another common strategy, powerful strategy, is just looking under rocks, poking in corners where people haven't looked before. And we've seen that with the GPS phone tracking databases. People lifted a lid, took a peek, and saw things that largely weren't well known. Here's an example of a multi-function printer. What goes into it? A network cable and a phone cable into the same device. Has anyone thought, can those two be bridged? I don't know, but that's a rock you might want to take a look at. It took a look under. Looking at something old, the DNS, so you can go back if you see old technologies. That's usually a sign that maybe you can dig a little more deeply. Or look at something new. Again, don't know if there's a vulnerability here, but Google makes web pages load instantly. The Chrome browser will soon silently fetch pages as you scan search results so that they will load without delay. I don't know about you, but I hesitate before clicking on certain links and don't really want my browser preloading them for me because what could possibly go wrong? So you see these things. And assuredly, people are looking at the security of this. But it just tips you off that maybe it's an area for exploration. You can also extend or generalize work that's already been done. There's been work on fingerprinting the sensors within digital cameras. And here's a paper from it. And they say, well, how could I extend that? Well, you could look or generalize it. Well, think of all sensors. What types of sensors broadly could be fingerprinted? Can microphones be fingerprinted, such that this recording here could be an MP3 tied back to this specific microphone? Or what other sensors leave unique noise or some aspect that can be fingerprinted? So you can extend, you can generalize. And of course, science fiction provides ample ideas. And this is the Diamond Age. And remember the Young Ladies Illustrated Primer, which is basically a tablet computer with artificial intelligence designed to teach children. You can draw tremendous inspiration from these. My personal favorite probably is assuming the worst in people, that you need to look at the capabilities of what a given company person or government has. And it's just a useful thought experiment to think about problems. We've seen examples in the past. And to look at the incentives. And you've seen real player software or spyware, the Sony root kit, and then some really crazy Facebook privacy interfaces. Another thought experiment is to think like a nation state. What would you do if you had god-like access to the network? What would you do if you had near unlimited processing power? That's just useful ways to think about problems. There's calls for papers for various events. And they'll list ideas. So if you're looking for something, you can go out and find them. And this is from the Usenix Leap workshop. But DEF CON, all the major events have calls for papers with ideas. And then the research themselves. If you're in a talk here, or you're in Black Hat, or you look at academic papers, they'll often tell you, here's where they've seen where to go next. They'll tell you where to go next if you listen. And the same thing in the written documents. And finally, a good survey article. If you're still interested in general area, but you don't want to get too deep into one thing, you could do a survey article. And by survey article, I mean you survey the current state of the art. And that might be for security visualization. It would be to go out and look at all the security visualization tools out there, compare and contrast them. Just that alone, a current version of that, is something you could publish and would be of interest. And at the same time, it gives you a foundation to build upon. If you might have identified a gap or an interesting area, and you can move forward. OK. So with research, though, it helps to have a system, a strategy, a time, a place. If you have a question, or a question you're looking at, or a topic you're trying to delve more deeply into. So one is constantly feeding your mind and choosing your news sources and getting things away from the keyboard, analog hobbies. And it's, again, creating those crazy intersections. Many people have different paths to learning. And I think ultimately, and this is kind of a holy war, but certificates, training, college, self-taught, each is a path to learning. And you have to find the combination that works for you. And along the way, it allows you to build up your toolset. Maybe you learn advanced data mining techniques that have never been applied before against a given problem. So you have new tools that you can combine to solve a given problem in a different way. It's very important, I think, to write down your ideas. I tend to do this obsessively. And when I was talking with friends before this talk, they said, oh, we remember everything. Like, really? I mean, I feel like me, I think writing it down, it puts a timestamp on your record. It allows you to recreate what you've done. And that moment of brilliance, it doesn't just dissipate. You've got it, you've captured it. And if you ever have to prove at a certain point in time you came up with an idea, this can be very helpful for doing so. And it just, it's a good way to brainstorm and record ideas in a formal way. And then there's other techniques you can use. One, I like giant pads of paper. I think those giant post-it notes all around someone's office would be very cool. And but there's many tools out there. But amidst the space, you've got these problems now, things that you've identified. But, and life is short though, as is time. And you have to find something that the time is right, something that you can get traction on, that you can make a dent in. I have a friend who developed a meme streams, which was a site, a social, it was a site kind of like slash dot. But if I talked to him about it, he said, you know, I just did it a little too early. And so ideas can be too early, they can be too late. So you have to time it a bit and wait for things to mature. You can't really think about solving a problem matter of a few days. Oftentimes, these harder problems you have to chip away at. And at first you don't feel like you're making any progress. But over a month, over time you'll, even if you just write a paragraph or a sentence and make it a little contribution, it adds up over time towards a final goal. And it's important as you're trying to do this to build on what others have done. You don't want to rediscover fire. And this is a great picture of standing on shoulders of giants that I went out and there's some great stuff out there. But it allows you to avoid duplicating what others have done and use that to energize what you're doing. You may encounter paywalls along the way where like various digital libraries but oftentimes the authors themselves or either have it posted online are happy to share if you send them a note. And as you collect pieces of information you can quickly become overwhelmed. Again, you may think, oh, I've got 10, I've got 50, it's not a big deal. But over time you can get this huge mass, it's unmanageable. So there are various tools out there probably the most famous is EndNote that allow you to organize and add keywords and create a searchable database. And even some of them allow you to push it into a word processor if you want to create a citation for some reason. So being able to keep found things found is the purpose of these tools. Organizing your data. I did some research, a forensic research of a million binary fragments. I didn't use the best naming scheme and every time I go in I have to spend half an hour trying to figure out what I did. So putting thought into how you organize your data at the start is very helpful. The target may move and that's okay. You're gonna circle in toward a goal and you may find out someone's done it or it's not achievable and you slide, you're flexible and you slide to something else and that's part of the process. You may find that it's a blind alley. Well, that's research is search and search again. So you went down an alley, you come back, you go down another path and knowing that something is not a fruitful approach, that is knowledge unto itself. Finding other people to help you along the way. From my experience, it's better to have people close, people in your local community that you can work with. You can still do it distributed but I would avoid against cold calling people and just sending them papers and the like to say, would you mind reading this? That generally doesn't go over well, particularly William Gibson has a great note on his site where people say, how can I get in touch with you? And Gibson says, you can write me in care of my publishers. They will then compost your letter, allow it to ferment for several months and eventually send it to me. I will then neglect to reply. So you probably don't wanna collaborate with him. But who can you collaborate with? Well, there's DC groups, 2,600 meetings, hacker spaces, industry groups, user groups, colleges all in your local community and who all can share depending on what your topic is share a passion. And at a minimum they can provide feedback and it's very powerful way to refine your idea and move it forward. And I mentioned before the idea of this isn't something, I've had cadets like we'll write something one day and turn it in the next. They don't even read it again. They just turn it in sometimes. And it's much better if you chip away and you put something down for a few days, pick it up a few days later and really refine something into work that you're proud of. If you're thinking about publishing in a forum like or almost any forum like a magazine, they'll have author guidelines. It's usually, I'd recommend reading them because they tell you what they want and they get overwhelmed. So it's easier if you provide something that tries to match what they're looking for and falls their norms. If you have the opportunity to work with an editor, pay attention to what they change. For me, this is from the security visualization book I wrote and the author or the editor, marked it up again and again, but I took a look at each change and asked myself why or even talked to the editor. Why did you do this? Because they're specialists in this type of thing. So you can learn a lot. It's like a free course in English, which if you're like me, you didn't pay attention in high school. Research requires time, often unbroken chunks of time. If you think if you're a programmer and you're trying to program and your manager comes in and interrupts what you're doing with a meeting, you're cruising altitude and all of a sudden you're crashed and you may or may not be able to get back up to where you're productive again. And Neil Stevenson in his Why I'm a Bad Correspondent has some great quotes on this. Writing novels is hard and requires vast unbroken slabs of time. Four quiet hours is a resource that I can put to good use. Or if I know that I'm going to be interrupted, I can't concentrate. And if I suspect that I might be interrupted, I can't do anything at all. Now life tends to intrude on our large chunks of time. And G.H. Hardy says, no mathematician should ever allow himself to forget that mathematics more than any other art or science is a young man's game. And why you ask? Well, you may pick up one of these, either husband or wife, and you'll find that life or time starts to evaporate or one of these. And again, you have a baby time evaporate. So it's all about keeping balance in your life but still finding chunks of time. And that might mean finding a place where you're creative. For example, interesting meetings, classes or talks will spark your creativity. Or for me, boring meetings, classes and talks, that I just bring a book and pretend I'm paying attention. I'm often most productive there. For others, it may be a bar or a food court or an airplane. And I'd encourage people to think beyond just a single problem. Once you kind of have a lane in mind, think about a series of things you're trying to accomplish. For me, in online privacy, which is an area I'm very passionate about, I think not just what do I do now, but what do I do next? Maybe I write a paper for a technical audience and then I write a white paper for decision makers and that they can understand and maybe affect policy in a positive way. And then I have strategies along the way, like three or four objectives and maybe what to do if it's something's not accepted, the plan B. Now I have a friend who's a military history buff who, when I showed him this, said that really wasn't the best one campaign graphic to use because that was the British and they lost in the, and in many ways our ability to do research is an economic problem. If you have to have a day job to work in a band at night, it's a problem, right? It cuts down on your time. And as Mudge mentioned at Black Hat for those that caught it, there are some programs out there that can provide funding, but typically they require a lot of meta work, a lot of writing giant proposals, lots of strings attached and lots of competition. I'm optimistic about what Mudge announces DARPA Cyber Fast Track that may be a way to bring some dollars in. I typically, what I do is at night and on weekends on my own time. So it's good to see things like that out there, particularly the focus on hacker spaces. All right, so what are some methodologies? Of course, the scientific method, if you haven't looked recently, it's a good way to think about problems, the idea, ask a question, do your background research, find out what other people have done and then construct a hypothesis and test it and then analyze your results and report on them in some way. It's useful to think about scientific method when conducting research. Another place to look is, I pulled this from the National Science Foundation, but it's a good way to think about, they're talking about conserving their dollars and choosing the right problems. Same thing with all of us. We have a very limited amount of time. What problems should we work on? And what they use are criteria called intellectual merit and broader impacts. What is this a transformative technology? Does it explore something creative and new? And what is its impact on society? Does it have a major long-standing impact? And not everything you do has to have that, but those are just good things to keep in mind when you're trying to balance the few projects you can actually accomplish in your lifetime. You may run into, I collide with others in idea space. It happens all the time. I mean, there's even a Wikipedia article on multiple discoveries, but it happens all the time. And you can either, maybe if they're user-friendly, you can bring them into your group and you all work together. Or you may find that you, even though you did due diligence and tried to research, but you missed someone's work, what can you do? You did due diligence. In the future, you note that that occurred, acknowledge that person's work, maybe build upon it and cite it and move on. And this is a holy war. Again, it's beyond the scope of the talk today, but when you do security research, you have to think about the ways that you disclose it. And there's various strategies for doing so. And if you search online, plenty of good discussions illustrating both sides of it, of the debate. And I'm not a lawyer, but keeping your personal research distinct from your employer is probably a good idea. And the advice I had been given in the past is not to let your personal work touch the time or resources like computing resources of your employer because they may gain some rights over it. So being very distinct in protecting what you're working on as a personal project. Although I think smart employers recognize that if they want talented people, they have to give them, create an environment where they can have their own IP or at least a share of their own IP. And in some cases, if for example, when I wrote the books, I had to go to my employer, tell them what I was doing so they weren't surprised and then go through these guidelines of keeping work distinct from my personal work. And then you've done this research and you say, well, what type of outputs can I have for maximum effect? I like to think of in terms of artifacts, things that you leave behind that others can use to build upon. And a video of a talk, a set of PowerPoint slides is good, but the idea is, can someone else reproduce what you've done? And by publishing, it allows you in some fashion, it allows you to contribute to the body of knowledge, but also to get feedback on what you've done, maybe build your reputation or find others interested in the problem. And importantly, it puts a timestamp on your work. You've got out there, because I think a lot of hacker community research just is not well known in certain circles and it's incredibly powerful and it's almost dismissed or it's pulled into research without properly citing it. Getting things out there puts a timestamp on it and will help prevent that. And the idea of reproducibility, creating things that you leave behind and histories of replete with technologies that we no longer know how to do them. The Strativarius violins are probably the best example. Just those technologies are lost and we're trying to avoid that. Research papers provide a useful way to think about problems, whether you're reading them or trying to write them or even thinking about a way to structure a talk on a subject. It's a time-tested way to think about problems and to create artifacts. Whether you write them or not, it's just a useful outline though. And it starts off with the background and motivation. That's the who cares, why and so what of the problem. And it gets into related work. What other people have done? So in this one package, this capsule, they're telling you what other people have done and why theirs is different. And then the core of it, the design, implementation, evaluation of some activity, system or experiment. And it wraps up with future work. Again, they're pointing you toward the future. And importantly, references. So you can go back, as you're trying to dig deeply into an area, you can find out what references they use to build theirs. You might find new ones. And you'll find as you explore an area, eventually you'll find everything that's been done. And it might just be 10 or 20 different little papers or articles, and then all of a sudden, you've got a pretty good understanding of the state of the art. And as you seek to publish, there's many different forums. I think great places to start are magazines, maybe writing a book chapter, serving as a technical reviewer for a book even, serving on panels and talks or writing white papers. Those are all things that are accessible. And in academia, there's a hierarchy, poster sessions. These are you create a poster, you stand in a room and a bunch of people who have interest in the area, wander through reading a cookie or something, giving you free advice on your work. And you kind of see some of that going on here. There's an analog here. And the idea then is you go up this hierarchy, you refine the idea more and more. And then it goes to say a technical report that self-published, maybe a workshop where people dig into a particular problem, and then more general conferences and then a journal which is supposed to be this archival thing. So that's the academic model. If you're thinking about writing academic papers, I strongly, or like a research paper in a workshop or something like that that's academic, I strongly encourage you to work with someone like a grad student because in academia, there's almost like this code, there's this writing style, this way of discussing things. It's not necessarily intuitive, but that's what they're learning. So it can be frustrating if you don't have that. Work can be awesome, but sometimes you'll just get with viewers that don't, it's foreign to them and they don't understand. Just trust me. And then at the bottom right, I've got just a few venues that I respect and I think might be open to hackerish type research. Usenik's Leet, the first Monday, the open publication format journal, IEEE Security and Privacy Magazine, Make Magazine, and then the whole hacker scene magazines like 2600 and many others. And as you think about, say academic security conferences, there's about a hundred of them and they vary in quality. And probably the best example of that are who's familiar with the MIT students that built the automatic paper generator? All right, yeah, that was a pretty good hack there. So they wrote a paper that they had a database of papers and this thing would just randomly take snippets and create using the exact right format and structure, just gibberish papers. And they got router accepted at WMSCI 2005. And then I thought that was awesome because there's some forums out there. You really don't want your name associated with it. And regardless of where you're trying to publish, it's good to do some reconnaissance or some research. Imagine trying to come to an event like DEF CON for the first time and then speaking at it. It could be, if you don't understand the context. So it's useful to do some research on where you're thinking about publishing, either online if it's or off or in person. You can even self publish. There's a great talk, self publishing in the underground from DEF CON 15 that talks about self publishing strategies. Or you might want to work with a publisher on a book at some point. And a question that comes up is, well, how are the calculations for royalties made? And I thought I'd just include an example in here. And this is Ballpark, but I checked with the publisher two days ago just to double check my math and agreed with me. So if the cover of a book is $50, the publisher may sell that book typically for $25 or so to a company like Amazon, to a distributor. Then you're given a percentage of what the publisher makes, which is Ballpark 12% of the $25. So on a $50 book, the author may make $3 per book. Then the question is, are you going to sell 100,000 copies and pay off your mortgage? Or are you going to sell 3,000 copies and buy a Stairmaster? Well, frankly, the advice I got from the publisher I was talking to said, you don't expect to get rich. I mean, 3,000, 5,000 copies, a pretty good run for a book. People have exceeded that and there are 100,000, I think, steal this computer book, for example, went over 100,000. But it's just good to know, $50 book, $3. But there's other benefits, right, by doing this. You create a long-term artifact. You gain some, perhaps some reputation, some exposure that'll bring in consulting gigs or media opportunities. And ultimately though, you have to do a lot of marketing yourself if you want to really push it up and over. For me, it took 14 months of free time for each book. Your mileage may vary. So some parting thoughts. Don't self-censor. Good research is often disruptive to the status quo. And I think the hacker community excels at that. So don't be afraid to choose something that's controversial. Be fearless when you can. Develop a sense for open problems. And that comes by knowing a space. And many in the hacker community do that. You're already doing research, right? So you find, develop a sense for open problems, where the holes are, and you can share them with others or work on them yourself. Also develop a sense for important problems. So now you know the open problems, but how do you prioritize them? And Richard Hamming wrote a very, or believe it was a talk called You and Your Research. And he was talking about Nobel Prize-winning work, but I think it still applies for everybody to think about, what are the most important problems in your field? Are you working on one of them? If not, why not? So it's a good way to think about it. It's a bit pretentious, but it's a good way to think about these things. And my most favorite quote from this talk is from Paul Graham about procrastination. The most impressive people I know are all procrastinators. They put off working on small stuff to work on big stuff. And I think that's important. When you go back to that chunks of time, you can actually get anything done. And we all, if you just stay on the email treadmill all day, you'll accomplish nothing. So I know programmers that shut down their email and their instant messaging or whatever to be able to focus on a problem, maybe unplug the phone. But we have to ignore the silly stuff, otherwise I'll eat all of our time. Of course the silly stuff may be important to our employers and you can't be totally ignored, but it's important to think about. It's also, balance is important. How much do you input? How much knowledge do you absorb? Some people just absorb, absorb, absorb, absorb and they're brilliant, but they don't output anything. And what type of, how much coding do you do? How much processing do you do of the data? So finding a balance that you learn things, you do something with them and output things is a balance you have to seek. And this was suggested by my friend Sergei that DEFCON, BlackHack, all the major hacker conferences, textfiles.com, have tremendous archives available. But it's almost to the point now, it's so rich, there's so much information out there that it's hard to find. You say, what was that talk on, I tried for this, there was this great talk on media relations that I was trying to find to mention. And I was at Hope, I think, and I don't remember and it's like every other year and you're trying to find this. So as we look to the future, maybe we can see a search engine created or like a library that points to these documents. It has this tag with metadata that's searchable in some way. Also playing to your strengths, right? So academia follows the grant money like dogs on a hunt sometimes. Industry, you have to make a case for the bottom line. But hackers, you've got many advantages. Passion, you know the real world problems, right? Oftentimes, that aren't known in academia. The, you have perhaps time, you may be fearless. You may be immune to the law in certain ways. You may be obsessed or you have a diverse background or diverse friends or diverse access to data. Those are all strengths that can be leveraged. It's also important to fight uninformed law and this comes from FX and some of his colleagues, Fina Lit website, it says honored visitor. Much to our regret, this site is no longer available in the form it has been since the late 1990s. It became illegal. So we have to, if you're in the security research, you got to watch the law in your country to make sure it's not going and do what you can to fight back because there's a lot of benefits to hacker research. A strategy can be finding inspiration others you respect. Whoever they may be. Oftentimes, sometimes they'll have an online presence and you can look and say, how did this person do these things? And you can follow what they've done or just people who don't have online presence but you just respect them. You can talk to them about how they did what they did. So not my car, I wish it was. Wikimedia Commons again has pictures of Ferraris but the license plate was actually real. I saw it at a conference in Washington, a security conference in Washington and I was in the parking garage. Didn't know if I was in the right place because I was a little lost and I pulled in the parking garage and parked next to the knob sled. I knew I was in the right place. And the journey itself has many dividends, right? Whether you find the cure for cancer or not, you learn so much along the way and maybe make some contributions. And Donald Rumsfeld very famous for his quotes on knowing what you know and don't know and the like but the key idea is some of the smartest people I know aren't afraid to ask questions. And I try to do the same and I encourage you to ask questions too. And this picture, you may wonder why there's a picture of cats and ducks but that's what Google thinks happy is. So I searched on happy and then this was one of the top ones and I thought it's fine. There were some other happy things in there that I didn't wanna put in to the slides. But kittens and ducks is probably safe. But here the idea is don't let this search for novelty, the critics, the reviewers that may give you a hard time cause they don't get what you're doing or some frustrations in publishing. Get in the way of good hacking and fun, right? You know, that's what it's all about. Life's short. So the final thing is this is doable. The research space isn't as crowded as you'd think and there's incredibly talented people here. So just encouraging you to go out and dig in and this is a doable thing. Don't be like me and think you can't do it. Lots of people to thank. And I've also, these will be posted on the website but, and this is a list of papers on that if you're thinking about publishing there's some ideas here and some books on problem solving that I've looked at or read and recommend. Okay, so I can't see. No one's giving me, I think I've got about seven minutes. So any questions? Oh, my website is, it's gregconzi.com. It's cheesy, yes, gregconzi.com. But it'll all speed up on the Black Hat website. I'm sorry, the Def Con website. Good, whoever turned down the lights, thank you. I'm not blind anymore. Are there microphones or are there microphones? So if anyone has questions, feel free to ask. Okay.