 Okay if everyone's ready we'll get into the next session don't worry I will be disappearing from this microphone very very soon because I've got a pizza that's under that table that I need to eat but I will just introduce Michael Hoffman I introduced him yesterday into the wrong name so if I've done it wrong again he's just going to ignore that and we're swiftly going to move into his talk and this talk is called Oz in keeping track and reporting of all things and over to you. Thank you. Thanks everybody we're gonna have a good time there when I first submitted this talk to the recon village I was thinking hey this could actually be kind of a workshop and then I thought well you know maybe I'll do a little bit more of just a presentation so this is actually one of those neat mash-up talks you're welcome to play along as I do certain things here all this stuff first off my slides are on the internet already I'll give you a link at the end so you don't have to do the take in the pictures thing you'll get the actual slides and also there's a lot of things that I'll be releasing during the talk so you can again you can grab that off the interwebs so I am Michael Hoffman I go by web breacher on the interwebs I have a nice website web breacher.com where I blog about a lot of these open-source intelligence things that I find helpful fun entertaining as well as web application stuff I do pen testing run a maker space open-source projects teach for sands wrote a course on open-source intelligence for sands that's a lot but let's not talk about me I have a question for you all all right this is actually a question where you're gonna need to raise your hand yes exercise involved here we go ready raise your hand if you love to document okay wow you guys are weird I wrote this line like it's gonna be crickets but no all right how many of you really believe no so yeah I absolutely don't like documenting but I recognize that it's a necessity in penetration testing as well as defense work as well as open-source intelligence we have to do it because if we don't things go awry now when I was younger what I found about open-source intelligence is I love to do the data acquisition I love finding shit about people or things or eyepiece I love analyzing the data and putting it together and finding those connections and finding how things actually appeared whether it was using like Maltigo or something else and find those relationships that was awesome and then and then which of those things do I need to dive deeper in where do I need to pivot and go deep I love doing all that and then of course my my senior OSINT person or senior pentastro be like yeah Micah you know that the reports due tomorrow I'm like just a little bit more dad or whatever his name was just a little bit more you know I've almost got more things I can find and always squeeze that little open source the documentation to the end nowadays adulting Micah we I still like doing the same things but what I'm finding is that I'm doing a lot more analysis and the reason why I'm doing more analysis is I have more data more good data that I can go through it's that data that you collect along the way is going to help determine where your OSINT investigation takes you or doesn't take you and that's very important because I've written a lot of open source intelligence reports I've written a lot of pentast reports I can't tell you the number of times I've gotten to the pentast report I'm like yeah we pop that box and I got sequel injection and I stole a copy of the database where who has the database I know I popped in they're like yeah you had it remember you put it into our chat oh and the chats you know now in Slack you know that that that part of the conversation is gone so you can't actually grab it and I don't have that stuff that would be awesome sauce to my customer don't let that happen to you one of the things that I like talking about is documenting deeply as you go when I do web pentast I document if I'm doing sequel injection boom I'm tagging it I'm looking at it I'm recording it as I go when I do something in open source intelligence I have to record it as I go because that data might change in a minute you ever see somebody tweet something out then it's gone wait wait you're like oh that was cool and you click on it's like tweet is unavailable yeah stuff disappears and if you're doing a multi-day multi-week assessment that stuff is not going to be there potentially when you get back to it so document deeply and you know what I found is that when you actually do document deeply your reports get better because you have more data you don't have to start scrounging going well you know I don't really have a screenshot of that but if you imagine a website with a picture of the person on it that's what now you have the pictures as they showed up and that's really the impactful part of it when we do open source intelligence one of the things that we have to do is convey to our customers what we see and what it means and that what it means is so very important for the impact over the years of teaching for sands and teaching internally for my companies I found that people are still using some older technologies to document we do capture the flags in the sands class have you ever taken one usually on the last day we do a capture the flag and I see people pulling up these oh solid but old tiny types of applications to record their process and every time I see that I'm like we can do better and that's really what this course is about of course that's really what this talk is about because if you're using these to capture your notes during an assessment I'm guessing it's suboptimal and I'll show you how in just a little bit but before we get to the tools which I know everybody's like all right show me more tools you know release something before we get to that let's talk about why you document because that's simply as important as what you document because what you are collecting while you're actually doing the assessment while you're doing the scanning while you're doing the analysis is extremely important and we've got a lot of different data types we have a lot of different pieces of data also that we need to collect to keep that timeline of evidence if you're doing something for law enforcement or for forensics you got to keep dates and times you got to keep where you got something sometimes you got to keep how you got there too I've seen pentas reports and oscent reports where it's like hey I found that picture of the of the person that was you know it doing some kind of insurance fraud and and they are playing basketball in their driving and my customers like that is terrific how did you get it where did you get it what date was that and if you don't have that data you're ultimately failing your customer now when we actually do collect stuff we have to think about it we we may have a tendency to collect all the things and that's great before we actually do an assessment we have to think about what are the types of data where might things go awry and what are my special considerations for instance if you're doing some some gathering of sources gathering of information maybe on some dating sites or on some other more sensitive types of sites maybe you find some classified data or proprietary data maybe you find some some data that is against the law to have on a computer or to be sharing how do you record that safely enough so you can include it in your report but not violate any laws yourself by propagating that type of stuff and there are rules for this you have to think about it before you just collect some of the other things that I found is that when you are collecting information about people sometimes people share an address right share a phone number okay if you're doing that text pad notepad plus plus type of documentation which is serial right it's like I got this person I got that well then if you are putting that phone number of multiple people now you have duplicate data in your report right it's that phone number here and that phone number up here to which is a little suboptimal instead of relating a single piece of information another thing is some people are going to do an assessment in hours or minutes and that's going to be really quick some people will do campaigns and look at their targets over months those are going to have different constraints for how where and what you're storing that content on and your documentation system and platform has to account for that and another thing is some of you probably work in teams or maybe you share your data with the defenders or dent defenders or maybe you're working a cyber threat intelligence your cti people and you're actually going to be writing this up to give to somebody else well when you actually collecting data in teams it's a whole nother type of mess right if you've done this solo you know you can rely on yourself you like oh yeah I'm doing this great but when you're combining that data getting somebody else access to it it can be challenging you got to work that out first you got to work that out before you dive into the assessment especially if your team is around the world I work with teams that are geographically just first across the United States and hey that time zones and change it matters because when I record something on my system and say hey I found this at this date and time well is that UTC or is that East Coast time it makes a difference if you're going to court it makes a difference in some cases you have to think about these things the other thing is where are you going to store your stuff right cloud systems there are some amazing cloud applications that are out there that make storing information so so simple and yet there they can be a security risk right you're sharing your stuff with Google now Google would never read your documents that you store in their service right there you're using HTTPS it's encrypted right yeah you have to think about this you have to consider it and then also is your application that you're using made for multiple people to use it or just one also we have to think about are we documenting this for our notes so I can remember my process so I can create a report or are we documenting to hand off to somebody else so that they continue it when I do some of my assessments I do up until a certain point I'm like hey I'm done you keep going defenders you you go ahead and research the rest of those IPs or do it and I'm doing that handoff well there's a different level of documentation that you need there I'd recently we were doing some some work in a one-note document if you've know about that Microsoft product good tool multi-user it works within an enterprise it's really good or sharing outside of an enterprise and I was writing this my notes for myself you know because I know notebook wasn't shared out well somebody else is like hey do you have notes on I'm like do I have notes on it here let me just add you to this and he looked at my notes and he was like well what the heck does this mean this is bad person and what does this mean because I was documenting it for me so we have to think about who our audience is and could be because that and could be is that when you deliver your report or your notes you never know where they're gonna go I've been in organizations where I deliver my notes to a customer and that customer takes that report and then six months later I come back to somebody totally else in that organization and that other person has said hey do you know Mike is quality of work do you have anything and this person's like yeah take this really sensitive document and and gives it to somebody else in their organization that has no need to know that information so my report when I walk into that person's office is sitting right there on their desk I'm like well I'm glad I didn't include sensitive stuff in there but we have to think about that we also have to think about that how we're gonna document I always like to document more than I need because I can always scale it back I can summarize I can redact or whatever it is that I need to in order to do my output whether it's a presentation whether it's to do something like a report you can always summarize and redact but if you don't have that data it's hard to get it back I've had people that that do an assessment and they're like oh it was there and they take a picture of the thing that says tweet unavailable and put that in the reports like that's that's not the way to do it and then your end goal you know where is this document going where are your notes going are you going to be continuing to work on this project is are you gonna hand it off to somebody report it etc thinking about that is going to help you in understanding what to do with the data also you don't know where the data is going to take you I'm sure if I if I asked for a raise of hands which I'm not going to and asked you if you ever had a simple project that you were working on maybe a no sin project and you were just doing Facebook stuff and then the the assessment maybe took a left turn or something or maybe you're looking at some dating profiles or something like that and something weird popped up I've done assessments where I'm just you know doing kind of a little background on a CEO or somebody that's a sweet C sweet person and then I find that there's a tinder name with that same person that they that person has been using for their Gmail it's like huh that's weird and you look over there and it's a whole nother like lifestyle that this person is leading you never know where your work is going to be taking you now within open source intelligence there's a lot of different versions of this diagram of this this cycle essentially we have requirements gathering in the upper right then that leads to retrieving data of some type analyzing data and pivoting and reporting and what I like to do is tag what types of things I'm looking for what is my documentation what is my reporting look like at each of these different phases of requirements gathering is is I'm going to be asking the questions of what am I doing getting those requirements for my customers so that might when I start gathering the data I know what the hell I'm looking for what are we finding that's where you're collecting the data of course if we also have you know the what is what are we missing parts in the analysis section and then when it gets to reporting of whatever kind we're going to go ahead and make that now this might hit a little bit close to home for some of you but stick with me here let's say that we have an example let's say we have a scenario maybe your son or daughter's teacher comes to you says listen you know I know you do that cyber stuff and I'm wondering if you could help me find something there is this dude that has been cyber bullying a bunch of kids in the class and it goes by the the name dread pirate Robert's written like it is on the slide there most of the kids that are playing on play stations are actually getting hit by this so so I need you to go ahead and take a look at this you're like cool alright start me off with the username I got that so you start your documentation does anybody see anything wrong right off the bat we're in notepad so notepad what happens on notepad when you close notepad does it save it it asks you if you want to save it but then if you don't say yes it's gone what happens if your computer crashes it's gone so let's say that you went ahead and did this alright here we're going to do a dread pirate Robert's the goal is this fine so you go into duck duck go and you do a search now the search here pulls up a Twitter account with the dread pirate Robert's that actual username we've actually got a bunch of information here that we can pull from we've got a Twitter account at number one we've got an avatar we can do some reverse image searching on we've got an actual spelled out name dread pirate Robert's that's that number three and actually a location number four is County Claire cool so we go back to our notepad document we type it in I duck duck go reports dread pirate Robert's found on Twitter here's a URL here's this there is a screenshot of the avatar and maybe you save it as TW avatar one so now you have a separate file on your system alright cool alright so you look for some of the other things that are in that on that duck duck go search oh we've got Clash Royale ooh now we're getting into the gaming stuff that was kind of the primary thing that we were looking for there is a username dread pirate Robert's cool so we've got on that we're going to document that and of course now we've got the Twitter plus we've got this Clash Royale thing so we got a look up both of those profiles so we're just going to put it to do on the page cool alright let's go back to that Twitter thing and so here on the Twitter page we've got even more information coming across because again we've got that avatar we've got some tweets we need to look into we've got some geo located tweets maybe we can get some information about where that person is or has been and of course we're going to go back to notepad we've got to document this stuff right we got some more to do we've got to do the reverse image search we've got to do this and that and then you do the reverse image search and now you've got 213 sites to look at and maybe you do recon ng there's this cool profiler module some dude wrote that goes through and takes a username and and goes and looks at it across over 150 websites well you've got that and so now this is a blow up of one of those now you've got four other sites including adult friend finder that you need to go look at you've got an Xbox but you have to go look at it you never know where your assessment's going to take you right sorry yeah you've got Xbox gaming and stuff alright so we've got some URLs to go to and in 10 minutes we've got a hell of a lot of to-do items we've got images we've got user names we've got URLs we've got a lot of stuff that we have to collect plus we have that recon ng tool output which we somehow have to include in our documentation and like I said this was within 10 minutes if you're doing this type of an assessment in hours can you imagine how many of those branches you're gonna have to actually deal with how many to-dos you're gonna have if we're gonna do that we need to have a documenting system that allows us to say I need to go do that and when I do I can collapse it or I can mark that that branch as checked off we also have to have a documentation platform that allows us to collect note annotate different types of data whether it's the pictures that we're coming across on our site or pictures of the websites or the data and man wouldn't it be nice if while we're just doing the work of OSINT something's collecting all of that stuff for us and some of you already know some of you already know that Hunchly does that really really well if you don't know about Hunchly stay tuned now when I made this talk I thought about well there is a huge number of documentation products and many of you that do like OSINT cti or even pen testing you might have special systems or enterprise wide systems that you have access to I wanted to keep this talk geared towards that solo practitioner that person that's doing that OSINT or that pen testing a recon or whatever it is alone or in small groups of teams so that's really my sweet spot here and that's who I'm I'm talking about now one of the things that we have to think about as the type of data that we're going to be documenting I was here today for some of the other talks and earlier yesterday for a couple of talks and I saw some great information on Maltigo if you saw Andrew present on Maltigo and the how it puts together data we've got some data visual is a visualizer apps out there that are amazing for showing us connections between IPs domain names user accounts I had an assignment one time where I had to do I had to see if I could find information about a certain target and so I went to her husband on Facebook and I found through his network a whole bunch of people with their same last names and then through all of that right into Maltigo and what Maltigo did was I said show this show this data it showed me really good tight packs of people that were interconnected connected to another group of people that were that were connected very tightly and it allowed me to see this is a family group and that's a family group and it's connected via these people so we have visualizers Maltigo Gaffee cytoscape cytoscope sorry we also have word processing apps word Libre Office these types of things as well and they're good for general documentation but we also had apps that are made to make our jobs as O centers a lot easier let's talk about that now I wanted to focus on one flexible apps that are going to make sense to most of you it's not going to work for all of you some of you are mandated to do one type of thing or another I also there are a lot of apps out there so I don't want to actually say hey go ahead and use this and it just does one piece of the puzzle I'm looking for those biggest bang for my bucks because I don't want to focus on documentation I want to focus on doing the OSINT and that's something document for me also easy to use always a requirement for me and ultimately decreases the work I have to do now if you've looked at my blog webreader.com at all you know I'm a very big fan of of mind maps mind maps are amazing visual note taking free applications that are out there that allow you to organize data graphically so you have one piece of data like a username like dread pirate robert's that dread pirate robert's then breaks out into a Twitter account and a gaming account and the dull friend finder and stuff and you have nodes that are that then branch out when you're done working on those nodes you can collapse it and say this note is done I'm done checking out a dull friend finder and now I need to go and look at other stuff and that's really helpful for keeping track of where you are in your assessment and what you have left to do if you're interested in it again this slide deck is online so you don't have to take pictures but on github.com slash web reach or OSINT tools I have a mind map that you can have for free that's going to hopefully jump start your documentation now I created this mind map over a year ago and when I was teaching my sec 487 sans class I found that one of my students was doing a lot of note taking during class I was like what do you know and he's like well I took all of the things that you said to do in class and I put it into that mind map that's awesome so when you get this file it has a lot of the notes a lot of the sites a lot of the other things that are in the at some OSINT classes like mine now we've written it in the X mind application X mind is a mind map software application and it's free for Windows Mac Linux again easy to use on whatever platform you're using and what it has is it has the centralized process it has about five different tabs or sheets if you will and each one's meant for different things for instance one of them is meant on product meant to discuss things about process like hey I have email address what do I do with it if you were here for my yoga talk yesterday yeah it's kind of the same thing but in an easy to use mind map format so here we have an email address if you look there while you do email verification you might look for that email on breach sites like we saw it will do earlier with have I been poised it shows you that process of what you can do but we also have a tab in there on data collection that says hey if you're doing research on a person you might want to grab their name and address and phone number and date of birth and aliases you actually type that into this document and it will organize it for you we can also do the same thing for IP addresses hashtag sentiment analysis whatever it is you're doing you can put it in here and when you take that that content that we had earlier you remember our document that that notepadding you start to fill it in into a mind map the data comes to life and you can see how things are put together I love it because instead of duplicating the information like hey that dread pirate Robert's name is here and here and here I can just use those double dashed arrows to say oh that that that username was found on Twitter and then I can paste in there the username the the picture of what the site looks like on his profile page and put other data points in there and when I'm done with investigating the Twitter I can click on these things collapse the entire branch you could store other types of data other types of files just by putting it in there other things that we have to do document the URLs document the dates and times mind maps are manual so you have to copy that URL and paste in there it's not perfect however organizing the data this way can save you a lot of time a lot of effort and can be quite appealing to your customers now it's not all fun and games not all great I will tell you this that to be honest there are some drawbacks with mind maps first off getting the data out is sometimes a pain in the ass because think about this if for a simple investigation like the dread pirate robbers thing I was telling you earlier we've got now 213 sites we have to visit yeah we might use a tool like eyewitness by Chris Truncher or peeping Tom by Tim Tomes or someone like that to scan all those sites cool and we can shove that in there but what happens is this mind map keeps branching out and branching out and branching out and when we're using it on the computer we're just dragging over to this area dragging to that area when it comes time to report how do you take something this big and fit it down to a four size or eight and a half by eleven size sometimes very challenging so sometimes we'll cut it up or do other stuff with it and with the pro version of X mind there's some better methods of exporting the content sometimes you can do it to a PDF multi user not so much so if you're a solar practitioner or you're handing documents off a mind map might be something that you're that you could use and then everything is manual which is something I don't like so what I do is I'll use a mind map for the overall investigation where am I going what do I need to do but I'll also use hunchley for that automated easy button approach to Maya's investigation and hunchley is amazing written by Justin Sites and his team it is a great tool it's a Google Chrome extension and it makes our lives so easy because it does a bunch of things everything from going ahead and cataloging and keeping track of any files we download while we're browsing to recording screen captures of every web page you visit so when you're googling or duck duck going or when you're tin eyeing something or when you're doing whatever on adult friend finder or whatever that's like keeps it sticking in my head for some reason I'm sorry if when you're going to the ps4 gaming thing site you know those pictures are going to be captured automatically along with the date and time that you visited them and the URL so you don't have to do it manually anymore you see how this is like I'm hearing angels singing right now now this costs about $130 Canadian per year if then you're in the United States it's like $3 US or something like that so if you've never used hunchley this is hunchley to a lot of people might be using hunchley version one Justin his team upgraded to version two and it's significantly awesome or it's which is a word here what we have is a dashboard when you launch hunchley you can bring up the dashboard which does overall case management and summary it tells you how what case you're currently in how many pages you visited how many files photos tagged etc and then one of the neat things about hunchley is that sometimes you have these things that words you need to look for in pages if I'm looking for red pirate Roberts any page I visit I want red pirate Roberts to pop out at me so I can set it up as a selector and then I tell hunchley hey wherever you see this on the page I want you to highlight it in yellow so when I'm doing my duck duck going or when I do a forums thing and I'm looking at some forum gaming forum and there's dread pirate Roberts it pops right down on the page for me makes it easy for me to do my oceans also it keeps track of and I know this is a little bit small and all the slides are out there on the Internet but this keeps track automatically of all the websites you visit in Google Chrome so if you're doing if you have this like great like stream of consciousness like oh this duck duck go search showed me this and now I'm going to take this here and go there and go there it's going to keep track of all that for you and the date and time you did it it's amazing and one of the neat things is that if you're visiting social media sites that allow the metadata that's inside of some pictures you know the geolocation or what camera took that picture if that data still with the pictures hunchley will pull it out and highlight it for you automatically for you whoa that's awesome now one of the problems we have is that sometimes we still need to run like ill bills ill wills I'm sorry ill wills tool you might need to run some other tool like recon ng or spider foot or something like that and you have that output what I like doing is doing some side channel loading of that data into hunchley because I use hunchley is like my repository of all my data that way when I'm done with my my work I take all that hunchley data and I export it and that's what I say so what you can do is take that recon ng data remember that that profiler module I ran earlier we can export that to a CSV or a text file and then visit it in a web page take a look at this so I have exported to recon ng results dot txt it's a CSV and then I visit I open that document in my chrome and now hunchley has tagged at that date at that time that it has that content and it will highlight all of the dread pirate robert's names in there and that way when I'm looking through my data when I'm searching through my hunchley I'm like well hey where else do I need to go where where else was this found I can pull that data up and it's all in one place I mentioned to you the exit data yep if you're visiting web pages that have images it'll pull it out such as this beach shot that has actually the GPS latitude and longitude in there again we need to go to another site to take it out and take a look at it to see where that is does it corroborate with the data but it's one less step that you have to take you don't have to run the exit tool or visit a secondary or tertiary site to pull that out now again I like to present the positives and the negatives hunchley's got a couple of drawbacks in my opinion one it's only Google Chrome I like doing stuff in Firefox a lot of my my best plugins and stuff are on Firefox I just like doing that this is only hunchley also it's single user if I have two people that are doing the assessment each of them is going to have their own hunchley data and as of right now I don't know a way to easily combine that data or even if I'd want to but combining that data and creating a centralized report will be a little bit more challenging because you'll have to combine that data and the reporting it does have a hey dump all of the web pages I visited dump all of the pictures that I tagged it has that but it's not like Microsoft Word or it's not something of that caliber so again this is something that we can use to do that automated documentation and then we put it together I know it's like that's the one picture you chose like it guess it was free so if you know why you're documenting and where you're going to be putting that stuff and who you're documenting with on your team and who you're documenting for then what you need to do along with your team is find those tools that work for you and work for your customers and do things the way you want to I have actually mandated on my team that they use mind map to do their assessments and I had one person and just didn't work with the way that his brain works that's fine that's absolutely fine I realized that mind maps work for people that learn visually it's just beautiful but if you're not a visual learner that's not going to work whatever make sure that you understand what the requirements are that you need to keep track of and some people are not going to necessarily be able to use these tools but you can check out these tools and there's other ones out there too many of them have free trials and I'll take it and try it out and see what you're doing because if you find yourself or a colleague doing this instead of doing something like this then they have a little bit of learning left to do and with that I will put my contact information again my presentations are right up there on the oscendinger prezos prezos and include that includes yesterday's presentation some other ones I've done as well you're welcome to take them and the link to our my sands course as well so I don't know if I have other I think I might have a time a second do you have any questions for me yes sir so there are some some distributions like bus buskador buskador and other things that are specifically made for open source intelligence gathering and an analysis do I have any recommendations my recommendation is going to be a cop out because I'm going to say do what works for you I was having a conversation with a gentleman next door and he mentioned something that something similar I love buskador buskador I think it's a great tool for what it is it simplifies a lot of those command line tools I understand you know I come up as a hacker in this community and I know that command line tools are not hard for me Python Ruby go I get it but for a lot of people they focus more on law enforcement they focus more on the open source intelligence or the CTI aspects and that stuff's hard and where buskador excels in my opinion is it put it abstracts that hole you need to run this command on the this command on the command line and they give you a graphical just box type in the username and it does all that stuff behind the scene so that is a great distribution and it's a hundred percent free it's on inteltechniques.com's website my Michael Bazelle's website but some of the things that you could also do is create your own virtual machine depending again on what your threat what your work is asking you to do as far as who you're working on obviously the more sensitive to data the more nefarious the things that your places you're gonna be going you're gonna want more of a barrier between your host computer and the other other questions sir okay so the question is are there any laws that would prevent a company from doing this type of reporting on their employees and the answer is I am not a lawyer next question no no the answer is yes there are laws that companies need to be aware of there's HR stuff but mostly what I find is that many companies have privacy policies for their employees or some kind of policy that governs what type of information they will collect about their employees and about their employees social and how the social activities and how they will use it and and that is usually what you see guiding this except if you're in the EU in which case you now have GDPR laws and stuff like that so I hope that that's helpful yes man you answer the question in your head well well time are there any other questions yes ma'am so the question is is that in my assessment here the example I used the dread pirate Roberts user a single person and and then branch out is the mind map format good for multiple targets let's say you're investigating a gang or or something like that a group of people the answer is yes I like it for documenting that stuff I don't like it for visualizing that I what I would do is I would document all the relationships in here on no taking wise and then probably make something pretty in Maltego Maltego there's a free version called case file which will take a CSV and import it and it will be make a beautiful diagram of this picture so for report and for that that finalized version I'd probably use a real visualization tool instead of this okay all right well I'm going to go ahead and say thank you very much for your time if you have any questions hit me up or come talk to me thank you for your time everybody