 Please help me welcome to the stage Truman Kane. Thank you and I just got my uh hacker tracker notification so thank you to the hacker tracker people as well. Had to subscribe to my own talk I mean come on. Uh my name is Truman Kane I'm a uh a pentester at uh Tvora we're located in Southern California or based in Southern California and I conduct uh social engineering pentest there fishing, fishing, physical uh so we're gonna get into the uh TLDR which is that autopilot cameras are good for a lot more than driving so our Tesla's that's the end of my fanboy brand. No it's not. Uh next slide's gonna be my referral code. Um so the huge disclaimer on this is use your own risk everything I'm gonna talk about is for educational purposes only of course check your local and state laws to make sure you're not breaking any laws if you decide to check out uh this framework. Last year uh Psy and Agent X had a talk on uh surveillance uh counter surveillance surveillance detection routes and that is somewhat of what inspired me to submit this talk. I spoke with Psy after um uh around the closing ceremonies last year about the potential plausibility of using phones or something like that um to basically in real time uh do some type of license plate detection or to aid in SDRs and it just didn't really seem plausible but uh with Tesla and Century Mode it's kind of a natural fit and I'll get into uh why Tesla uh for this type of framework. So why Tesla there are three cameras built in and really there are eight in the newer vehicles uh potentially all of them um but Tesla gives you access to three feeds the front camera the left basically what the left and right mirrors see as well. So if you plug in a flash drive you're able to get essentially all that video uh so that's that's built in you don't have to mount any dash cams. Uh second is the detailed vehicle and owner API. Basically we can ping roughly every second we could do it more but you probably get get rate limited or potentially blocked so I'm not really taking chances with that but every second I can get uh vehicle location data, heading data, how fast I'm going and so that's really well you could even you could even program it to honk at people if you want to um but I don't want to make anyone angry. So using that data we're able to correlate with that video it makes for some um interesting use cases. Um the other part is that it's electric vehicle. Century Mode is uh if you have it enabled it means that the car's always on. It's not really gonna be using too much extra battery um to keep that on um so that's awesome because with a gas vehicle you have to either keep the car running if that's how you're powering your devices or you have to have a huge battery pack that you have to worry about switching out and charging that type of thing. Uh the next part is the in car uh web browser which makes it extremely easy to see these uh real time notifications if um a follow vehicle is detected. Of course that's not um completely necessary uh but it definitely helps. So to get into the basics of what surveillance detection scout is the real time license plate recognition and face detection those are pretty self explanatory so I'm not gonna go into those but on the next slide I'll get into what real time uh means in this context. Um recon and pattern of life querying is essentially just regexing or just using some operators on that data that's found and uh of course this is um open source and locally deployed. I was very tempted to use S3 or an AWS service for this type of thing but I definitely decided that was a bad idea um because I just feel like it's going to end with people getting their um footage dumped basically. So highly recommend running this locally and that's why it's not you know natively integrated with S3. So let's get into what real time is. Um every minute Tesla is writing three MP4 files to this flash drive and so you have this and and they're they're roughly 30 megs each so every minute you have roughly 90 90 megs of video to search through as soon as those MP4s hit the flash drive which it's not a flash drive uh we're we're using like a computer type device to act as the flash drive in this case and we'll talk more about that in a second but as soon as the device hits disk and I know it's not disk it's a solid state drive. Inference is going to start um right before that we're pulling just the iframes to get us down from what is it um 36 frames per second on a normal uh video as soon as it's written to four frames per second and so we're getting rid of a lot of those blurred frames that we're not going to most likely not going to be able to get anything out of anyway. Um in your settings you can adjust that so it's not going to you know it's going to take long to run the inference but you're going to have more accurate detections and more detection so that's all adjustable but that's what we mean by real time defined as soon as the you know files hit the disk we start running the inference and it ends up being pretty fast if you opt for the higher quality hardware so then it's just going to come down to um uh cost um savings perspective. So let's talk about some of the counter surveillance scenarios. If you're parked you want to know you know which cars are people uh which cars are people are loitering near my house or my car um you're not really going to be able to see any faces while you're driving the cameras just aren't at that level yet I'm sure you could if you used third party dash cams and this saw the software component of this could be modified to work with third party dash cams absolutely but it's just not going to be as natural of a fit. Um during your drive things like how long is the car been behind me or have I seen that car before and the last one is one that I really want to highlight is the fact that this device isn't going to take the place of an actual standard surveillance detection route that someone is running and someone is still going to have to remain highly vigilant if they're in a situation where they're seriously running surveillance detection routes daily um but the computer isn't going to forget over months, years, decades and not to say that a car's going to be following you for years maybe I'm the only one I don't know but the computer's not going to forget and that's what makes it uh really helpful um to look at it from like a big data perspective later on and we'll look into that uh in the demo. So some recon query scenarios this basically means so from a physical pentesting or red teaming perspective you can essentially park the car outside of an office that you're going to run a physical pentest on leave it there for a day and later on get the and you could put a mobile hotspot there if you want to be able to you like a you know Wi-Fi hub some type of Wi-Fi thing to in real time remotely access the data that's there or you could just pick it up later and run these recon queries to see when the first person arrives in the morning when the last person leaves at night you could scrape LinkedIn for the company that you're going to be running the physical pentest on get these photos load them into your database and then run facial recognition against those photos so somewhat scary but uh definitely an extremely um powerful tool from a physical pentesting perspective as well. So the web stack is not super important uh because you could you could modify this however you want uh view is just the JavaScript framework that I prefer um and then MongoDB again because I wasn't about to put this on Firebase uh I just am not I just wanted to keep everything local especially when I was testing this with all my drive data. The computer vision stack I expect that this might change a little bit but I'll give you a little bit of a backstory I tried to initially just build this out myself I was like okay AI computer vision I'm gonna train this model I'm gonna so I just start labeling images left and right I probably spent close to 20 hours just going through gigs of um video footage and just cutting out frames dragging the little box over the license plates I even roped in some of the interns which I appreciate you guys didn't end up using it. I will um the off the shelf frameworks uh so there's a AL uh there's a GitHub repo um with the name ALPR dash unconstrained that's one where I looked through a few and just right right out of the box it was able to classify um correctly half of the license plate images that I gave to it so that was huge and you can you can um modify or fine tune this setup with those labeled images so I will be making use of any manually labeled um um you know training uh images but the short story the short version is that I'm not some expert in AI I took off the shelf tools that are freely available online and it's just knowing you know nest or ring cameras have facial recognition there's facial facial recognition software and open source software out there it's becoming a lot easier to use and um I know that there's license plate recognition open source software out there as well and just putting the two and two together I'm going to drink water so I I'm using um YOLO v3 which is based off of darknet to run the um license plate um detection and then I'm using face net for the um facial recognition um features with the devices that I'm going to get to next um I'm making use of tensor rt which is a way to use tensor flow on these invidia devices so that they run super fast and again this potentially will change I'm just going to be fine tuning testing if you submit a pull request and it's just better than we would implement it. Here are the hardware options that this um one of the things that originally inspired this was uh github repo that was uh that's uh tesla usb basically uh a guy took a pi 0w and modified it or scripted it so that the tesla would treat it as a flash drive and so when you pulled into your garage at home connected to wifi you were able to move that video over extremely slowly. I was very happy to get off of the pi 0w. Uh with the pi uh four coming out you could potentially run inference on this but really the top two options are going to be for if you just want to be able to capture all of the video um by default tesla is only going to be able to uh is only going to give you saved events where either you manually saved an event by clicking a button on screen or uh when a century mode event um was detected basically when the enhanced car alarm would go off that's when you get an event. Quite a few false positives in there as well um but you would use either of these two top options if you wanted to save all of that video. So the bottom two options are the jets and nano which is it's only a hundred dollars is extremely affordable that's this one here and the uh jets and Xavier which is this one here are going to be the ones that you're going to want to run uh use if you're running inference if you want to be notified in real time with the other two you're just going to upload that uh video uh data to your computer and run inference on it later. So really for all my testing I used the jets and Xavier. Uh as far as uh benchmarks the Xavier is going to be almost double as fast now it's definitely more than double the price but seven hundred dollars I think it's still affordable especially if I just think it's I just think it's worth it um but the you're going to get things it's not it's not really going to be um real time if you go with the jets and nano so if you want real time you you need to go with the Xavier and again I expect these um FPS benchmarks to uh change as well. Alright let's get to the demo. Okay so here is basically the recently detected dashboard this is when you what you what you see first and as you can see a lot of plates are detected this is a very small sample but a lot of plates are detected in a very short amount of time and you can search by car maker model which is pretty cool uh but this isn't really going to be able to help tell you if you're being followed you can't really make use of the data in this way so we can click on all detected plates those were fake license plates by the way. Click on all detected plates and now we can start to make better use of this data we can really study the data here um at the top you'll see this link to friend basically we are able to tag something as a benign finding um if we notice like maybe we're looking at the video later we notice like it's it's a co-worker neighbor friend something like that. Now why is this high risk with only four uh detections down here? Well if you want to see a um a video of the real time live in car um notification there's a wired article and or just YouTube video that I'll post that will um have a link to that video. So basically you're able to set your settings on notification you're able to set your threshold do I want to be notified when a car is detected x number of times in x number of minutes do I want to be notified when a vehicle the same vehicle is detected over a certain number of days uh maybe within a geographical area and then outside of that geographical area that's what you can do from a threshold perspective and that's why this high risk uh rating is here when there are only uh four detections is because I had set something up to where if a car is seen the same car is seen over four uh over four days like you know if it's seen every day for four days let me know essentially and it was. So we can click into this range over to figure out what the deal is we we know uh an alert was triggered we want to know why we can turn on satellite mode if it makes it easier to see but basically here we see in the search bar that we search this license plate the place markers are going to show the photos of the actual uh detection that took place we can click into one and we can see at the time that this detection took place the speed that we were going the power usage of the car what gear we're in and we can click into uh play video if we want to um be brought back in time to when this detection occurred so on the bottom left we see the car driving by that's the front camera there in the middle and we can fast forward a few seconds here and now we can see in real time how fast we were going along with this video and along with um you know the place marker map data so uh we were able to put all this data together in a really cool way uh that just makes it useful um to to examine. Now let's see what's next here so we're going to get into the um face detection. Essentially we know that this is a male a 20s it's the same male uh we're able to click or unclick if we're able to train the model basically we see a video of the detection this is very similar to like a nest camera or ring camera if you're familiar. So we're able to name this person save it and now we can search uh for this person's name and we see that Truman has been detected snooping around my car and that is essentially it for the demo. So what's next? Remote live view again this is going to come down to that whole real time thing. I'm trying I'm working on a way to start running inference on those frames as they're being written to them before if anyone wants to help out with that that would be awesome I'd be greatly appreciated but that would take you down take you from zero to 60 seconds that you're waiting to start the inference to you know rest soon as the the frame is written essentially. Third party dash cams is another one it's just not as much of a natural fit so it's uh it's not going to be a uh super high priority but I could definitely see why people would want to make use of the software side of this. Gay recognition is something that I think is really cool and would just at just from because it's interesting I would like to implement that and uh more object detection you might have seen that little button or filter there that was for pets. I took uh the I took uh Google open images uh data set and I made it only see license plates so it'd be faster. So I removed thousands of different objects that already could have been detected there and I just I just thought it would be interesting when I was you know starting with this project like what just to see okay cat black cat how many black cats is my car seen like you can do that with any object in any of these data sets that's just really interesting to me that you can make use of the data in that way and then your request here submit it for request or issue if you think on a on github if you think something else should be added. I wish I could spend more time on the privacy implications of this but basically Teslas are out there there are a lot of cars that are starting to my self driving features and uh almost all of them are making use of cameras I mean essentially the amount of cameras out on the road is just going to exponentially increase very very quickly um private corporations and governments are going to want to make use of this data insurance companies everyone is going to want this data um market research firms so if you think about breaches in the next few years I I feel like a lot of them are going to include this type of data because it I I think it's going to be extremely valuable and that's it thank you very much you can check out the link there if you want to see the code uh the bill of materials is on there now the code um for all of this is going to be um added later today thank you very much thanks