 How's everyone doing good just curious if this is yeah, who here is this is your first DEF CON? Oh Oh Look around But those of you this is your first DEF CON Welcome welcome Each and every year well except for last year when we run talks they ask this question and The numbers are usually anyway between like 70 to 80 percent So the numbers are pretty consistent So for those of you if you're you know, just your first DEF CON welcome It can be a little overwhelming let's be honest about it, but my biggest advice is Take every opportunity that you can This is an opportunity for you to learn as much as you as much as you can and also do yourself a favor Learn from other people including strangers here as well, too You may never never know that person that's going to be speed you speak to that could be a lifelong connection actually speaking of Speaking of the next speaker and start a lifelong connection how I got to meet you and You'll be on in two minutes So make sure you enjoy DEF CON and Learn as much as you can Let's get started. Hello everyone good now. Yes. Good afternoon everyone. Good afternoon Welcome to DEF CON 31 and Welcome to talks presented by the packet hacking village since 2013 been a long time the packet hacking village has presented a track of talks so and the whole premise is people who Go to talk And you walk away You should be able to take away something immediately to use after the talk So that's the whole premise of packet hacking village talk and each and every year they're well attended and First want to say thank you. Thank you for for being here for Actually, our final packet hacking village talk for today And it is my honor and privilege to introduce someone who has given a few talks in the past and quality world-class talk to the packet hacking village for years Mike Rago and Chet Hosmer So I'll let Mike give a little background about about him and his talk is on open-source intelligence open open-source intelligence and physical physical threat intelligence, so My pleasure to introduce an old friend Mike Rago Thanks Ming really appreciate it and super excited to be back Thanks for having me and huge shout outs to the packet hacking village for everything they do over the years My name is Mike Rago My good friend Chet Hosmer couldn't be here today some family-related stuff We wish them well send our thoughts and prayers Chet and I Co-present quite a bit at a variety of conferences around the world. So we share a lot of the same information in terms of our research and Look forward to sharing that with you This particular research and title of this presentation is oscent for physical threat intelligence We know that a lot of people In the security community today are leveraging oscent intelligence in a plethora of ways Whether it be for a person of interest or an organization of interest Whether it be for pen testing and going out there and gathering any public information they can find our particular focus of research for the last few years is Physical threat intelligence and we'll talk a little bit more about what that means and what it applies to in the real world And how organizations and people use this today So Chet and I as I mentioned have collaborated for many years We've written books together and we both have been professors with the University of Arizona Aside from our full-time jobs and it's been a great research ground for a lot of the things We do over the years further more we've presented out a variety of conferences including def con hat con in Norway and many other conferences in addition a lot of this research we collaborate on Allows us to work with a lot of organizations and understand a lot of their needs So as we go through this there'll be a lot of applied aspects and real-world scenarios and how we use this physical threat intelligence for situational awareness Chet has authored even more books than I have and again just a variety of research over the years in In addition, he does a lot of coding and runs Python forensics.org So Chet does a lot of research and a lot around Python So I think most of us here are familiar with open-source intelligence or OSINT Again, what we're trying to do here is apply that wealth of information To meaningful intelligence that we can apply in a physical way whether that be Concerns around business risk. Are we building a store in you know a foreign country? And are we concerned about someone targeting that store either while it's being built? Or the first day it opens with some kind of flash mob that destroys the store or things like that or maybe it's applies to in organization that provides Help for people around the world Our thoughts and prayers go out to Maui and everything that's occurred there But there's a wealth of data and pictures and information people are posting that can be leveraged to determine Where to mobilize help next and really provide that help to people that are in need So this data can be applied in a variety of a lot of ways Furthermore, it can be very cutting edge in terms of collecting it in a streaming format or live So it can be very actionable We'll talk about the overwhelming wealth of data out there and how to better curate that So you don't spend 99 percent of your time trying to find those needles in the haystack That are most helpful for situational awareness, but actually flip that model upside down to Pre-curate that data and make it far more actionable in a very timely manner as we mentioned here in the slide We also want that to be a key decision-making piece of information that we can use but it doesn't displace Information you may receive via private channels or other channels you have and how you collect that intelligence and use it for situational awareness Bottom line. It's meant to complement that data The applied aspects of it could be related to threat assessments and business risk Could also be related as I mentioned to disaster response In this particular talk we expose a lot of the research we've done around the Ukrainian war to actually identify different attack patterns Paths that troops are moving what things are being destroyed how to profile tanks and other Equipment out in you know the Ukraine and other areas and it also can be used in more Geopolitical type situations as well So this data is becoming more and more helpful But again one of the big concerns may be around misinformation and disinformation and we'll talk about that too Lastly before we get into how we actually perform this and some of the Python scripts We've created for the collections and weeding through that data is as I mentioned can we trust this data? one of the things that we do is by collecting it in a live and streaming format But furthermore Pinpointing a geo and collecting what we call ground truth for a bounding box or a radius from which those posts are emanating from We're not collecting data from around the world. We don't want data from around the world It's littered with misinformation and disinformation So you can leverage a technique that we'll talk about more here shortly where you can create a radius Or a bounding box around a city such as Kiev within the Ukraine You could do it around an area that you know has been impacted by an unfortunate disaster This will give you much more legitimate posts that are again Originating from that area giving you that ground truth and we have found bottom line that that data is Increasingly helpful and weeds out more than 99 percent of the misinformation So in getting to what we mentioned earlier about being proactive and pre curating that data So you don't have a huge data warehouse of useless data. This is one of the preliminary things you can do So you don't end up in that scenario So let's talk about the tradecraft and then we'll talk about our research and a lot of the interesting findings So conducting this Involves a lot of data sources. It may be social media It may be other sources as well and understanding the chemistry of these different social networks and other mediums that provide this information For example, if you're gonna take a look at discord on discord There are a variety of servers and channels that are focused on the Ukrainian war that are focused specifically on nothing But tanks and other things that people are passionate about that they're posting a Lot of those posts are also non-bias So there are pro-Ukrainian and pro-Russian posts going on But again, we want to collect ground truth So we want stuff originating from that area not across lines not from other portions of the world in addition That helps with countering all that counterintelligence misinformation and disinformation So when we perform these analysis techniques We also are profiling and understanding which of these accounts over time are actually spreading misinformation We'll talk about how we profile those so that we understand the top Posters or people that may be posting legitimate information from that ground truth and people who are Basically muddying the waters with that misinformation So in this case we want less data not more most people will get on social media Leverage a hashtag or a keyword and just start collecting a whole ton of data Oh, if you talk to anyone that basically takes that and tries to Automate that there's just so much data collected you end up spending too much time Analyzing all that data you can actually get ahead of the curve in a variety of ways Also, you know as a long time ago when I started out as a Unix system administrator You know my boss told me listen There's a lot of things to manage here on the network at the NASDAQ stock market, you know laziness breeds efficiency Focus on the most important things, you know, try to focus on you know Less data not more in how we pre-curate that so we'll talk about that too Also, we want to respect the terms of service these vary based on the website based on the social network Make sure you're familiar with what those are some of them have very important privacy policies We're not exposed to expose the accounts from which this data may be coming from also scraping and other things Sometimes they're not allowed But as you interact for example with these social networks, there are exposed API's for example with Twitter Those have changed quite a bit recently and now also there are tiers You have to pay for to get some of that data that otherwise with a free account you wouldn't get so understanding that too is going to help enrich your data and also ensure that you're within the terms of service so as We collect based on a different methodology rather than keyword or hashtag. We actually collect by geo again We get that ground truth data that's emanating from within that particular radius or bounding box in addition We increases the authenticity of it because we're collecting not only posts, but we want those images videos and even emojis and We pump those emojis actually through some ML that actually helps us in an automated way Understand the sentiment to which also helps us determine whether or not this information can be trusted So we created a Python script that allows us to perform this collection this allows us to either pick from a particular city or location and Select the bounding box or radius for that location Or you can actually put in the latitude and longitude and then define the radius or bounding box for what you want to collect That fundamentally performs the collection by basically eliminating all the rest of the data from around the world Are we gonna miss a few important things? Yeah, but what I will say that's a lot easier than collecting data from all around the world And trying to find those needles in the haystack Instead we're weeding out all those needles and we're getting roughly 90 percent of them and that is far more actionable and Timely in terms of leveraging this data for all of those applied aspects. We mentioned earlier We can additionally Put in keywords and hashtags even in other languages and this will help further curate a lot of that data There may be stuff within a radius around Kiev that have nothing to do with the war So if we're looking to collect stuff related to the Ukrainian Russian war there may be stuff That's irrelevant as we perform the collection then we're collecting each post it could be a tweet It could be a post it depends on the social network and collecting those images and Collecting those videos We'll talk next how that stuff is really important for cross correlating that data We also take a look at the accounts that are posting this information So as we categorize them and determine who are the top people that are posting legitimate and very helpful Information we can actually focus on those and eliminate a lot of the other accounts What we have found is of the data that normally emanates from a bounding box that there are about 10 to 20 people That dominate those posts with a wealth of positive helpful information But they're also about 10 to 20 that are littering it with misinformation But again knowing these from the get-go As you're collecting this data and weeding that out again helps curate the data in a very automated way So we've gone we've collected the data. We've done this in a very automated fashion again leveraging geolocation information And leveraging this in a chronological way So that you know we have a timeline as well of the events and we can use this for predictive analysis So let's talk about how we leverage this data Now as we look through the data We can see all the posts all the tweets and furthermore any of the information and referring links if it may point to Twitter if it may point to discord if it may point to telegram And as we look through the data that we've collected we start to find some really interesting information If you're using some form of computer vision in your favorite cloud vendor You can automate the analysis of these images and we've done that So if you're using optical character recognition you may start to notice here on the screen that there are serial numbers on these military vehicles and by knowing we collected that information for that vehicle at that location at that point in time We know that that vehicle was located there But if we're monitoring multiple locations around the you the Ukraine for example Kikarev that they targeted first Then moved on to Kiev if we saw that military vehicle in Kikarev and then later Kiev We're starting to see a motion on the map right of where these Where this equipment is going where the troops are going and potentially where they may go next Really interesting sliver of information. I'll show you later that was something that we found that we completely did not expect So there's a lot of Related data here across these devices if you're using optical character recognition whether it be an image or a video It'll go ahead and scan all those images and actually gave us Chronologically in time and location wise where those where that equipment was and You may have heard you may or may not have heard the terminology then Chrono Locationally, so we're taking a look at it chronologically, but also locationally as well So kind of a new term that's been floating around chronologically So to recap before we get we delve deeper into the data is we're performing live Streaming collection Secondly by geo to help pre curate that data We're not running a vacuum and collecting data from all around the world based on a keyword a hashtag We're not Doing that from the get-go. We're saying we want ground truth from this radius from this location In addition since we're getting the location information we can plop this on a map One of the frequently asked questions we get is how are you getting the location information? Then if that privacy is shut down or people don't have that enabled on their mobile devices when it's posted It comes from the apis and how these social networks are collecting the data They know that it's been posted from when it's in that bounding box They won't say that that person is specifically on this street on this corner in Kiev or San Francisco Or Chicago or wherever, but it's definitely within that region and in this case We don't care exactly where it's pinpointed on the map We can build a one-mile radius around a region and get all that ground truth from there And that's good enough for making these very important decisions in terms of mobilizing troops or mobilizing help in the event of a disaster So again, we're not collecting this from the device This comes directly from the social network telling us that this person with this device Posted from within that region or that bounding box This takes us to another piece of really interesting information a lot of activity going on in the war Initially the knee-jerk reaction is let's collect everything we can around tanks and troops and what's being bombed and targeted And that's great, and that's absolutely important But when we started to kind of take a look at the data in a different way Leveraging transforms we actually discovered a completely different narrative going on and that narrative involved places of worship So we found that churches synagogues mosques across the Ukraine were being destroyed as troops move through those areas So if we set aside all the data around what we knew about the troop movements and military equipment and things like that We also started to see Some very important trends of all of these places of worship being destroyed We would not have seen that had it not been for some of the automation We'll talk about next which is why now with you know the invent of AI chat GPT Co-pilot and those things that you can now have a right hand virtual person if you will to Have you or have them look at it in a completely different way, and this is how we found this very important artifact Why is that important? Well as we started to map this out You could use something as simple as Google Maps if you wanted to you could go in here and plot these on a map and In addition with the automation You can also include the initial post or tweet or whatever and in addition that image So now as you start to bring this intelligence together if you want to look at it on a map You can start to see in this case all the places of worship that unfortunately have been destroyed during the war and If you start to now look at the look at that chrono locationally or a chronological timeline by location There's some very important predictions. You can make here about where they might target next We did that and we found out we were a hundred percent spot on not by tracking the tanks Not by tracking the troops, but by tracking the churches Synagogues and mosques all these places of worship that were actually being destroyed So we take all that data and then plot that on a map The captured image the tweet or the post if it's from a sort another social network and what we found Is as we start to go through this we then said okay, we have another whole You know lump of data here around troop movement and tanks We'll plot all that on a map too and you can do layers You can do layers within Google Maps to help layer that or look at that data individually by type So here we mapped all the tanks These were some of the tanks with the same serial numbers that we saw at different locations at different points in time So what's going on with this particular one? The first one you'll see the tank here just out just outside Kiev But if you then look at the next tank, it's heading towards the area of Donetsk What we learn later is that attacks from the Russians then occurred, you know from the Donetsk region from the water from the west side of the country and Basically what we're able to predict is that a lot of this military once it had targeted Kiev Started to move in the direction of Donetsk and we also started to see movement from the water hitting land targeting Donetsk Bottom line all of this military equipment and troops were converging on Donetsk This tank that we show here is that that piece of military equipment We showed earlier with that serial number B 23 and we showed it first next to Kiev Then moving towards Donetsk and then further, you know closer to Donetsk So you could leverage Twitter for this and some other social networks if you go for example to discord There are some amazing servers out there with a ton of information around the war as I mentioned some people are just passionate about Plains or tanks or you name it So there is a wealth of data out there that we leverage with this to you know cross correlate the data We're getting from multiple social networks again, whether it be leveraging the API's we're doing this within the terms of service What was really cool about that then is as we start to merge that data on a Google map We can basically enable all of those layers to now look at this in the context of everything going on whether it be military equipment troops churches and places of worship being destroyed and many of the other activities by different categorizations This definitely starts to build out a storyline Chronologically to help understand what might happen next as we showed with that piece of military equipment There's a lot of different ways you can dashboard this and this can be very helpful for an analyst There are lots of either free tools or free trial tools you could use in this particular case I used power bi we took all the data and with a Azure logic app Fed this automatically into power bi Initially, I created all these dashboards for all this data. Why is this important? Well over on the far right, you'll see a filter this filter allows you to search against the data You can either automate that or if an analyst needs to investigate something for example the serial number of that military Equipment or Mention or other things that are going on it gives you full sorting capability built right into the tool in In addition to this you can also filter the tweets filter things by emojis. There's a wealth of things you can do with this Now with things like co-pilot or other things is the ability to say hey I want to incorporate some prompt engineering I want to ask it some questions and when I do I want to say hey build me a dashboard based on the intelligence You're seeing and it'll build another tab with that additional intelligence from a completely different viewpoint Giving you other analytics and data points. You didn't even think to look for so when you start to map it out in this way and You know a lot of these tools even have a free trial You can go in there and leverage these to get some really cool actionable and very short timeline type data There's also a plethora of widgets, you know built into a lot of these tools What I had used within power BI allowed me to not only do a lot of the mapping with Arc GIS Which is fed by Ursa. I believe But also other forms of word widgets like a word cloud This became really informative too because based on the data. There was a whole bunch of different keywords that popped up So I can see a variety of things there the obviously popular You know used word and Ukrainian and Russian and they start to look at some of this other data When I started to take a look at the top 10 words one of the ones that popped up was oblast Which I believe is a region within Kiev I had no idea that things were going on at that point in time within Kiev but This actually told me that through the word cloud through the widget through all this active data So again another data point I would not have looked at by doing human analysis of all of this But leveraging the widget and the automation in your favorite dashboarding whether it be Grafana whether it be power BI Whether it be whatever name your tool So then we start to get into extrapolation and prediction We've already got some really interesting data points that we didn't even think to look for but with the automation With a lot of different perspectives with the AI with chat GPT with prompt engineering We got a lot of additional very helpful data and got that in a very timely manner The data is automatically fading into it and analysts can be right there doing searches creating reports and getting that data on demand So as I mentioned we're able to track some of the Russian troop movements based on the data. We collect it And in addition We also found you know certain areas as I mentioned places of worship that were being destroyed Location the time at which things were occurring and other interesting data points the question here then was if I look specifically at these places of worship being destroyed Can I predict where the troops might be next? So if I'm on the defense or counter, I and I need to figure out where to mobilize my teams We were able to actually use this data to do a variety of prediction up the top you can see a trick nerve and Before that we actually had key correct where the troops were moving based on the chronological timeline And we could see they were moving closer and closer to Kiev It was pretty much expected they were in a target key of but these data points proved that and it gave us some really Interesting data points to actually say yep, they're on the move. They're all converging on Kiev Then you'll see a splinter of you know Some troops in a completely different direction One of the main highways in the eastern lane direction and we showed you earlier Where some of the other troops splintered off and started going towards the nest where they also converged So a lot of this data definitely can be used for predictive Extrapolation type of analysis and we proved it But we would not have known to look at just places of worship to get that data So the automation the prompt engineering and a lot of these things are what actually got our laser focus on some of these Other areas we weren't focused on so a lot of us are starting to mess around with things like chat GPT and What's kind of nice about it is a lot of vendors have taken this and basically Isolated it in a tenant for you so that you're running it only against your data and not exposing that data or that analysis to the external world This prompt engineering as I mentioned can allow you to put in a variety of different prompts to look at just this data in a lot of unique ways Tell me what are the top trends? We're seeing at this point in time. What are they? Are they places of worship? Are they troop movements? Are they posts from? people that are suffering damage and need help You know on the ground within this geo location within this bounding box Through this we basically said let me go ahead and put in a variety of questions And then once I got this honed in on what I want I basically can say build me a dashboard So whereas the previous dashboard I had was built manually by myself I can now have the AI and the automation and build it for me and when it did It looked completely different and that was really cool So we have that as a second tab looking all all of this data in a different transform in a different way Bottom line it helped us identify a plethora of things we had missed So how can we use this methodology? It can be applied to not only war but disaster response. There are a lot of organizations that want to know Hey, we're having all of these California rains. There are dams building up with water People are starting to post that the water is going over the dam and destroying some homes Also help us, you know find people in need within that geo bounding box of that region that we weren't focused on someone just you know posted a Something about their home being destroyed and there's a bunch of flooding from a river that overflowed or the things that were going on in Sacramento so this will obviously give you a variety of really interesting data points to help with the Situational awareness and mobilization of help for these individuals Another thing that we frequently do is the account profiling as I mentioned We basically whittle this down so that 50% of the data we're getting are from the top 10 to 20 People that are posting on that social network versus the whole gamut of that entire region that provides us about 50% of the data and also helps us further read out misinformation and disinformation Because we also profile those accounts that are spreading misinformation or disinformation and profile those top 10 accounts as well So we can say exclude that data. I don't even want that included with it So hope you found this talk informative I'll hang out for a little bit if you have any questions about this happy to chat about it Hope you enjoyed it. We spent a few years if you're interested in a copy of the research report It's available at our site listed there silent signals calm You can download it for free and see a deeper analysis around the research report. It's free And we also are looking for people at any time that may want to get involved with the research So if you are interested, we have an email there And then we also have our Twitter account where we post some of the latest information around some of our research So thanks so much for your time today. I hope you enjoyed it Mike we go everyone If you have a question from Mike, yeah, we can hang out here. We got time I don't think you need to use a microphone, but yeah, thank you so much for being here Tomorrow we will have two more talks sponsored by the presented by the packet hacking village in the afternoon Thank you again I've got a question What level of Twitter API were you using to gather your data? Can you clarify I'm sorry I caught the part around whether it's like a free version. It's limited There's like a five thousand dollar a month fire hose version Yeah, um the last That we were you know doing our you know updates related to it. There were multiple tiers So the free versus the other three tiers There's like don't quote me on this and it may have changed again, but there was basic I forget the middle one and then like enterprise the basic I believe the last I checked was a hundred dollars a month Yeah, and and that's what we've used and we've gotten like a wealth of data. Yeah Yeah, but we did have to move from free to a paid version But the basic did give us a wealth of data if you go on the Tweepy API website it'll list out the different tiers for you in different tabs so you can see exactly what API's are available Good question Well, I'll hang out for a few more minutes if you have any questions feel free to to use the mic Or if you want to come up and chat privately, that's fine, too, and thanks again. Hope you enjoy the show