 So, I want to tell you a story today about intelligence and intelligence collection. And when you talk about intelligence, unless there's been a well-known leak in the press or something, there aren't a whole lot of examples to talk about. And we'll certainly talk about those. But I was really wanting to do today is talk about some actual projects. And most of the time, like my intelligence clients, they don't want me to talk about what I'm doing for them, for obvious reasons. Sometimes I get lucky like two years ago and I had a client where we did a project maybe like six, seven years prior. I was able to talk about that, but normally that's not the case. So a couple years ago, I decided that I wanted to become my own intelligence client, okay? So what I ended up doing is I wanted a business with my girlfriend, and we have an online retail business. And it's doing pretty well. We're in our second year, and it's growing. And I like to think the reason that it's growing is not because we work hard, but we do work hard. I like to think it's because we have got these little hacker brains, and we just have better intelligence than everybody else. Or we create better intelligence, I should say. So I'm going to show you some absolute concrete examples of how we create and apply intelligence in our own business. In addition to the other examples I'm going to show. So I'm going to show you how we do intelligence campaigns not only on our competitors, but also on the sales channels that we use. And I'll describe why that's important in a bit here. The other thing that's important for retail businesses is you want to know exactly what kind of inventory to buy before you buy it. I'll show you the intelligence that we use to do that. And I'm also going to show you how we manipulate markets. Or as I tell my mother, how we protect our investments. All right, so when people hear the word intelligence, usually they think immediately military intelligence, right? Well, there's not a whole lot of that that applies to what we do. You know, the other side of it is either business intelligence or competitive intelligence. And those two are really pretty different. But I think it's important that we describe the differences between the two. Business intelligence is what's happening within your business. You use internal data and you focus on knowing your operations and you focus on knowing your resources. And what you're really trying to do is make your whole operation just more effective and more efficient and everything. And that's all very useful. But it's also very different than competitive intelligence. Because with competitive intelligence, you're trying to find what's happening outside your organization. And it doesn't matter if you are a business or a political organization, or if you are a investigative journalist trying to collect information. This is all more competitive intelligence. You're using external data and your focus is on knowing your competitors and you're knowing your markets. So this is the area where we're gonna live today. We're gonna live in the competitive intelligence area. So a lot of people, they think competitive intelligence, that's kind of, that's kind of murky. But it's important to know that competitive intelligence is taught in every top business school. In fact, if you are running a business, it's your responsibility to do this kind of stuff. And it's also important to know that there is actually a professional association of people who do competitive intelligence. With newsletters and conferences and a whole shot. So I wanna talk about the title a little bit, Applied Intelligence. So when I'm talking about applied intelligence, I'm talking about intelligence that's actionable, right? Something that's gonna change the way you're doing things. The problem is, is that most of the intelligence that's collected for the organizations that do collect it, it's not actionable. It won't actually change what they're doing. And the other problem is, most organizations tend to over collect intelligence when they do collect it. So when you do that, it comes at a higher cost and you've got more exposure. The other thing is that a lot of the intelligence that's collected is done because people just feel obligated to collect it. Not with any real reason. So for example, I don't know how many of you saw this, this is something I had up, I was tweeting about it. I came up with a little scraper that every hour would go off to the DEF CON site and it would make a printable, mobile friendly version of the DEF CON speaker schedule. And it's still up there if you're looking for it. And if you go through my Twitter account, you can find it. So I'm doing this and it's cool and it's useful. And I know a lot of people were using it because I was collecting analytics on this stuff, right? So I got the access, time, IP addresses, I wrote a cookie so I could tell people they were looking at it more than once. But this is more analytics, it's not really intelligence. And the reason I don't really consider this to be good intelligence is because there is absolutely nothing in these logs that would have changed anything that I was doing. In fact, the only thing that I found out that was really interesting is probably about 90% of the traffic was bots, it wasn't even human activity. But again, that's not gonna really affect anything that I was doing. Now, if I wanted to make this applied intelligence, what I could have done is instead of having just a static table with all the schedule on it, I could have given people the option to click on the talks they wanted to go to and I could have generated a nice little schedule for people to take along, it's like, oh, you know, four o'clock, I need to be on this track, you know, that would have been handy. And it also potentially would have created some actionable intelligence. Because if I had this information and if I had it in aggregate, I could make some projections as to which talks are gonna be really popular, which ones are gonna run out of seating and it could have been useful for people doing speaker ops. I didn't do any of this kind of stuff, but I see the potential use that could have changed the way things were done. Okay, the second part of the title, information that's not there or isn't there, that is a direct reference to metadata. Now, for most of us, we had no idea what metadata was until the Edward Snowden disclosures came across. And then suddenly, you know, we heard the word surveillance. You know, we all know what that is and we knew that things that we were doing online, on our phones, even at the library, were now subject to surveillance. And there was something called metadata that was involved in this. And people in the United States, we were pretty much confused. But fortunately, we have public officials that can help us in this area. So Diane Feinstein comes up and says, this was a very hastily called meeting from the Intelligence Committee. And she goes, as you know, this is just metadata. There's no content involved. Okay, so as a citizen of the United States, this made me feel pretty damn good, all right? It's just, okay, so you know I went to the library, but you don't know what I looked at. Okay, okay, that's pretty good. So then our own president got involved. So it's nobody's listening to your phone calls. That's not what this program's about. They're just looking at names, or they're not looking at names and they're not looking at content, but sifting through this so-called metadata. Okay, they're sifting through the data, okay? And I think I'm pretty familiar with data. I'm not familiar with a SIFT command. You know? So this was good. This was very good. This was very good because now I know more about metadata. You can SIFT it, okay? So I'm starting to feel pretty good about metadata, but I really got confused then when Michael Hayden, who was the former NSA boss chimed in, and he said, oh, we kill people based on metadata. So okay, I need to know a little more now. I need to know just a little more. Okay, so metadata. Metadata is so important, and it's largely misunderstood. And we probably all know that metadata is data that describes other data, right? But more importantly than that, it's data that provides context for information. And usually the very best metadata doesn't even really exist. It needs to be created, all right? And that's where the fun is. That's where the data hacking happens. So if you study metadata, you know, academically in college, they've got this whole plethora of categories for metadata. I like to keep it simple and say there's two kinds, basically. There's parametric data, and that's data that has to be collected or created. And then there's embedded data, and this is probably where most people got familiar with metadata initially, like in the 90s. And that's generally user-created. It tends to be kind of stuff like the embedded headers in a digital photograph that describes the camera that was used when the picture was taken, the F stops, Geocodes. In fact, this stuff has become like a whole branch of law now, because this is all submissible in courts. So I've got a couple of examples of these and some of the data that's leaked this way. This is one I actually used last year if you're at my talk, and I'm using it again because I just love it. This is a selfie of a Russian soldier that was uploaded to Instagram. And unfortunately, before he uploaded it, he didn't remove the Geocodes. So it revealed the fact that while his government said they had no troops in Ukraine, he was in Ukraine. Another example, another war-related example, this is kind of a famous one. This is the Tony Blair memo. This was the memo that was the justification for the invasion of Iraq in what, 2002, 2003, something like that, right after 9-11. And it was actually used by Colin Powell when he went to the United Nations to make his justification. And the assumption was that this was done by area specialists, and it was all original work, well researched. What we find out, it was actually plagiarized. And not just plagiarized, but it was plagiarized by a graduate student someplace in the UK. Oh, that was the original document, was a graduate student's paper. And they know it was plagiarized because not only was the content the same, but so were the grammatical errors. So where this gets interesting though is in the metadata. You know, Microsoft Office files are full of metadata. And some of it is for version control and all kinds of stuff. But if you look at the metadata here, we can see that the people who did the plagiarizing, we can figure out who they were. And we can see that they were not area specialists, but they were actually political advisors to Tony Blair. So it's kind of a nasty thing. This kind of stuff happens a lot though. The CEO of Google, Eric Schmidt, a couple years ago, he wanted to just to put a PowerPoint presentation online. That was just kind of a general status report for how things were happening at Google. What he neglected to do though is that when he made that report, he actually made a copy of a very confidential PowerPoint presentation that was for internal use only. And he did a lot of editing of the slides. What he neglected to do was edit the speaker notes. And within the speaker notes, he leaked the existence of a brand new project called Google Drive. And that was well before it ever was officially announced. So watch what you put in your speaker notes because there's a lot of, you know, it's out there. Okay, so let's talk, that was embedded metadata. Let's look a little bit about parametric metadata and see how that's used. So according to the NSA, this is their own admission. This is what they collect. They collect phone numbers of parties making phone calls. They collect the time a call was placed. They collect the duration of a phone call and also who initiated the call. Okay, that's kind of invasive, right? But you know, it's actually, they're not doing anything that any Android app isn't making available to the developer, right? I mean, it's, stuff's in there. It's in all your phones if you have an Android. But with this metadata, you can get a lot of really interesting stuff. What they're really after is the relationships between callers, okay? Because the metadata creates the context within the phone calls, how they were placed. And within that, you can figure out the relationships of the callers. And these relationships can then be profiled. And once you've profiled them, you can pick out the anomalies and outliers. And at that point, it becomes possible to differentiate between a phone call being made by a parent at work to their child, making sure they got home from school, okay? That phone call is gonna look very different than some criminal operative calling in for instructions. It's gonna look very different. And they're also able to identify things like burner phones because those things are big outliers that kind of stick up when all this stuff gets profiled. The other thing is you can look at these, the patterns that people are calling and the relationships, and you can tie them to other events. And if other events affect the way people relate, there may be a connection between that and those events. So these are the kinds of things that the NSA looks for. These are the real red flags. And they'll go three hops, okay? So what that means is if you have a phone call unknowingly with somebody who has had a phone call with somebody who is a person of interest, when all suddenly you're kind of a person of interest too. So that's how that all works. So you could look at this and say, well, yeah, it makes sense that they would look at metadata because it's the easy way out. Otherwise they'd have to look at all these conversations and they'd probably have to do some kind of speech to text conversion and deal with languages and they'd have to deal with dialects and all that kind of stuff. But it's just better data. Just flat out better data. It's already digital. It can be processed, but it needs to be created, okay? And that's the important thing about metadata. All these relationships and stuff need to be figured out. It's kind of like, how long do you need to play the game of Clue before you know it's Mrs. Peacock killing Colonel Mustard in the library with a candlestick, right? If you're good at this, you're gonna figure it out long before the other people that are playing the game. So that's where the mind games come in with metadata. Okay, so how do we do metadata? How do we process things? Well, I'm not the NSA. I don't have unlimited resources and stuff. So the area where I live is called OPSEC, or Operational Security, and that is a military term. And what that refers to is looking at your day-to-day operations to see what kind of intelligence you're leaking. And if you had an adversary looking at your day-to-day operations, what kind of actionable intelligence could they collect? Okay, well, I do this online. The internet is my theater. So I look at things like employment postings. Last year, I gave examples about how companies leak strategic plans through their employment postings. And I think you can kind of figure out how that could probably happen, right? If you think individuals leak information about themselves and social media, wait till you get a whole organization together leaking information on social media. A lot of information gets leaked this way. The way orders are fulfilled, so if you go online, place an order on a website, and if you watch the process, you see what kind of emails you get back, you look at order numbers, all this kind of stuff, you can learn a lot about the way a company fulfills items, where things are shipped from, all that kind of stuff. If they have a store, and if they maintain a store, you can learn all kinds of stuff about pricing strategies, the products they choose to stock, the products that they choose not to stock, very rich source of intelligence. If you sell something to a company online, the way they buy things can be very revealing. In fact, you can tell a lot about an organization's financial health just by looking at the check numbers, just little things like that. And then there's the whole regulatory area, and this is largely things that are done for transparency, financial filings, things that get filed with the courts, variants, licenses, all that kind of stuff. Together, this is where I get competitive intelligence from. This is where I live. And for my counter-intelligence clients, this is what we protect, okay? So the first area I'm gonna talk about is for things that we really have been able to capitalize on is sequential numbers. And the privacy leak, the data leaks that happen with sequential numbers, it's really staggering. Sequential numbers are used everywhere. I mean, absolutely everywhere. They're on your vehicle identification number, social security numbers, ticket numbers, order numbers. I mean, they are literally everywhere. And from a data standpoint, you need unique identifiers to represent things like orders, people, all that kind of stuff. But you don't need sequential numbers. And typically, the only reason these numbers become sequential is because it's sloppy programming. Somebody programs a online app, and instead of providing a unique order number, they basically reveal the index for that order and at some table someplace, right? So they end up being sequential. Well, unless you've got charged backs and weird things going on, but you can pretty much count on these things being sequential. So I'm gonna show you a little story, I'm gonna tell you a little story to show the power of sequential numbers and how the US government almost let an entire generation be susceptible to identity fraud, all right? So here's how it happened. Social security cards, they all have numbers on them. And the first three are an area number, the middle two are a group number. I really don't care too much about those. They basically kind of identify the regions you're from and that kind of thing and where things were processed. But the last four digits are interesting because those were serial numbers. And from 1935 up until 1972 when they changed the law, they were truly sequential numbers. But it wasn't a big deal because, I don't know about you guys, but I grew up like in the 70s and I was probably 14, 15 years old when I got my social security card. And if I had gone with a buddy or if there had been somebody else applying for the number at the same time, we would have got sequential social security numbers. Not a big deal because there wouldn't be a whole lot to connect these two people together. So if you knew one number, you probably couldn't guess the other one because you wouldn't know who the other person was, right? Well, this all changed in 1972. These numbers became no longer sequential. And I don't know why they did that, but I know the way the government works. And I know it wasn't because they had a lot of foresight and were thinking about the Tax Reform Act of 1986. Because in 1986, the IRS said, if you are going to declare a dependent on your taxes, that dependent needs to have a social security number. And I think it was probably done more for tax reform to weed out fraudulent activity on taxes because the year they did this, there were 7 million fewer dependents claimed on tax forms. But again, I believe it was an accident that in 72 they stopped doing the sequential numbers. It wasn't because of this, I can't believe it was. But if they hadn't changed the law in 72, this would have been the scenario. You've got three kids back in the late 80s. You need to declare them as dependents on your tax forms. You've got kids that are maybe ages five, three and one or something. They don't have social security numbers, so what do you do? You go to the post office, you go to the Social Security Administration, and you file for social security cards for all of your children at once. So the bigger the family, the bigger the threat because each of these siblings will have sequential social security numbers. And it's pretty easy to pick up a social security number, right? If you work in HR, if you do credit kind of stuff, or if somebody dies, you're all familiar with the Social Security death list, right? As soon as you die, if you're an American citizen, your social security number gets published. It becomes public, so nobody uses it. But if you know one of these numbers and you know they have siblings, it would be trivial to guess what their social security numbers are. So I'm very thankful that they changed the law, but it would have been catastrophic otherwise if they hadn't. Okay, so how do we use sequential numbers in our business? And sequential numbers, again, are usually not thought of as being metadata, but they really are. So my girlfriend and I, we started our business and we had tried other things before and nothing really caught on. But we got excited about this one because our month-to-month sales were pretty good. We were having like 150% increase in sales pretty much pretty consistently every month that we were going on this. And I was excited. I mean, this was growing a lot faster than my consulting practice. And I think probably in the next year or two, it's going to surpass my consulting practice, I think. But we were getting very confident at this point, right? You know, along by August and September, we were plowing money into the business. Everything that we were making, we were putting back in plus some. So we were investing in an inventory, we were investing in space, shelving, fulfillment equipment like scales and all kinds of stuff. And we were doing it because we had some confidence. We had about six months of consecutive growth. We're in this now, okay? And then came October. You know, sales just went through the floor. So what you need to remember here is that we sell on a number of different channels and probably 80% of our business comes from this one channel. And our business on that channel was really, really off. And we were getting very concerned because we were still plowing money into this business, right? So we're thinking, you know, we don't know this business. We've only been at this for a few months. Did we enter at a bubble? You know, did we hit the peak of the bubble? Now this is, it's going to be this from now on? You know, we didn't know. So we were trying to figure out what was going on. We're pulling up spreadsheets. We're looking at old orders. We were trying to get a clue, right? So my girlfriend's reading off order numbers and amounts and stuff and I'm plugging things into a spreadsheet. And we both noticed at about the same time that the order numbers were getting bigger. And then it was like, okay, we need to find two orders that were close together. And we found two that were placed almost at the same time and the order numbers that we were getting from this channel were incremental. Well they weren't incremental, they were sequential, right? It's like, okay, we got something now. So here's what we did. We took the last order number that we had to fulfill in October and we subtracted the last order number we had from September. And this gave us an estimate of the number of orders that were fulfilled through the sales channel by all of our competitors for the month of October. Pretty cool, huh? We knew that they fulfilled 6,500 orders roughly that month. This was our bad month, mind you. So we're thinking, well, our average order, we figured out to be 1248. And we had no reason to believe that our orders were any different than anybody else's that was using the sales channel. And we determined this to be pretty typical. It's very competitive. So this was pretty typical. So we took our average sale, and by the way, that should not be July, that should be October. We multiply it by the number of orders that we estimated for October. And we figured out that the gross sales on this channel in the month of October, our bad month, was just over $81,000. All right, that's pretty cool. So now let's look at the other months. So we took the last order number from the last order that we fulfilled coming from the sales channel from April, May, June, July, all the way through October and we were able to figure out the number of orders that were placed, right? And from this, we were able to figure out the gross sales for that channel, right? Fun with metadata, cool stuff, right? We were happy because we looked at our October sales and it's like, well, it's not consistent with the channel. The market is fine. We screwed up someplace. We did something that caused our sales to be off or maybe it was just a fluke, but we did not enter the market at the beginning of a bubble. So we were able to sleep much better at night knowing this, okay? So better living through metadata. So this is what the channel sales actually looked like. Even though ours was really down for October, theirs was actually up a little bit. So we're like, well, what else can we figure out? Well, if you wanna figure out what you can figure out, write down what you know. Well, we know that the channel that we were selling on, their commission's around 20%. So we're able to figure out how much commission they were making every month. And they're making $15,000, $16,000 a month commission. It's like, well, that's interesting. Well, we also know that when somebody buys something on this channel, they're charged for postage and as resellers, we get some of that back, but the channel keeps $1.25 on every sale. So we're able to look at their postage that they get and we're able to figure out that their average monthly profit for this sales channel, this website, is about $24,000 a month and that they have annual profits of about $300,000 a year. Well, this really got us thinking, we're thinking this is probably not information they wanted to share, probably not with somebody who's got 20 years of web development experience. And this channel, I mean, honestly, it's probably something, their biggest task every month would be making PayPal payments to their, you know, doing commission checks off to people, basically. I can't imagine that it takes anybody more than a couple hours a month to run this, seriously. They don't advertise, they have no expenses, they carry no inventory. Their business model's clearly better than ours. So for now, but again, all because of metadata and sequential numbers. So I'm gonna look a little bit at how we buy inventory. One of the things that I've learned in retail is that there's a market price for everything. And if you're lucky, you will get the market price if it's inventory, as it gets older, you tend to kind of lower the price over time. So you really wanna get that market price, but you're not gonna make it in retail by trying to charge for more than market. You know, that's a good way to not sell stuff. So the way you make money in retail is not by selling stuff. You make money when you buy stuff, right? Because if something has a market value, you know you can probably sell it for that. But if you can buy it for well below market, that's where you make your money. So procurement is really important for us. And there's a number of websites where we buy stuff too. And we sell on multiple websites, but there's one channel that's got like 80% of our business. It's the only true market that we really deal with. And I refer to it as a true market because it's large enough. I mean, there's, I don't know how many people sell on this market, but there's a lot of them. They're not eBay, but they're big, okay? You know, and there's literally millions of products out there. So it's mature enough. There are enough professional sellers out there. There's enough market demand supply and demand curve going on. So that it is a true market. And if you wanna know what the market value is for something, you go out and look in here. So what we do is we have bots, we have software that goes off and autonomously looks for things that we might wanna buy. And it compares them to our main channel where we sell. And it looks for market prices. And it comes up with a little report like this for us. This would be really time consuming to develop on our own. Having software that does this is really nice. Basically what it does, it identifies the item where it found it, what the price is, what the market price is. Anything with a margin of less than 300%, we really don't deal with. And then it tells us if we should buy it or ignore it, which is really cool. It's a real time saver for us. So the little buy thing, there's actually a link. If you click on it, it'll actually go off and show you where you can buy it. And it's, like I said, it's a real time saver. The next step would be to automate this so that we automatically buy our inventory. We don't have to think about it. Kind of like, was it two years ago? The bots that I wrote that bought cars. Over a nine month period, we bought like $20 million worth of cars. Something like that. So one of the reasons that we're able to do this is because we're selling unique items. And there are major, major privacy issues for retailers that sell unique items. So unique items, truly unique items, would be things where there's only one of. So real estate. You've got an address. There's one piece of real estate there. Vehicles. You've got a stock number. You've got a VIN. It directly identifies a specific vehicle. Original art is another one. You only have one copy of an original art unless it's like a signed photo or something. Even then, you've got a numbering system to identify it. Then there's also likely unique items. That would be things like first edition books, autographed items, most used things. And these are things where it's possible that you've got two copies of something but you probably don't. It's rare enough where they only probably have one copy of a first edition of Catcher in the Rye, for example, because it's rare enough. So when you've got a situation like this, here's what we do. Remember that we were very new to this. We did not know this industry at all, but we wanted to learn. And what we did is we wrote software that automatically collected the entire inventory of what we considered to be our main competitors. And this channel actually made it really easy for us to do this, because they actually list the top resellers on the channel. So we would look at all of their inventory and we would do it again. And basically the delta, the difference between these two over maybe like a day or a week's time, would give us metadata that described what they sold and for how much and also what didn't sell during that period. So this is how we learned about our competition. And we did this a lot. So the more of this we did, the more metadata we got that described what sold, how much, how long their inventory stayed online before it was sold. And again, what didn't sell. And the worry we're able to do with this is because you look at the inventory one day, you look at it another day, if something's missing here, you assume that it sold for the last known price, right? And this is, I mean, you can tell so much about businesses that sell unique or probably are likely unique items. And if you think of all the businesses out there that put all their inventory out online, you can study this to the point where you can almost do their books for them. You know what their money is coming in. So okay, one more. How do we protect our investments? Well, we have software that looks for situations like this. So if we're seller C and our price is 19.50 and we're competitive with sellers D and E that are selling for a little bit more of this, we have software that'll go out and it'll flag situations like this where people for whatever reason, they're dumping stuff for below market value. So we have software that goes off and we immediately buy these things, okay? This is how we kind of pseudo manipulate our markets. So if we find things that are underpriced like that, we just buy them, they become our inventory and we're protecting our investment. But again, we couldn't do this without intelligence. And like I said, I have to believe if we were just typical sellers, we would just kind of not be doing that well. You gotta really know the business. And in our case, our proprietary intelligence tools and our collection of metadata really make the difference for us. So if you find this kind of thing interesting, you can follow me on Twitter and I'll leave my Twitter handle up for the rest of the slides here. I tend to almost live tweet while I'm developing, you know, like I'll talk about bugs I'm finding in my own code and stuff like that. The other thing you can do is watch the DEF CON website. This set of slides will be made available on that website. These are much more complete than what you've got on your conference DVD. There was also an article written about me on the Christian Science Monitor last week. That's something you might wanna check out. Six of my other DEF CON talks are online. They're all on YouTube. And with the exception of, I think, one, they all involve intelligence and competitive intelligence. And then finally, the last thing you might wanna do is this is the book that I wrote. This is actually the second edition of the book I wrote. And right after this, I am heading over to the No Starks booth to do a book signing. If you just have a question, that'd be a great time to sit and talk. And I'd really love to see you there. So thank you very much.