 Good afternoon everyone, and thank you all for coming. This is abusing peer-to-peer to hack 3 million cameras. This talk is the culmination of about two years of research, so I'm very excited to finally be able to share this with you all. And with that, let's get started. I'm going to be talking about an obscure feature lurking in IOT devices around the world. Peer-to-peer is intended to make life easier for people, but it's had the nasty side effect of exposing millions of devices to the internet. Even those behind firewalls are not immune. Hundreds of different brands are affected by this, including security cameras, alarm systems, and even baby monitors. But exposure is just the tip of the iceberg. When hordes of insecure things get put on the internet, you can bet the end result is not going to be pretty. So I'm here to show you just how bad this can get. By the end of this talk, you'll see how peer-to-peer devices work, the numerous ways they can be exploited, and finally, how a $40 purchase from Amazon is all you need to start hacking into other devices. So who am I? My name is Paul Maripisi. I am a security researcher based out of San Jose, California, and I probably spend too much time getting angry at computers. I largely focus on offensive security, and I work on the red team at a certain enterprise cloud company. When I'm not breaking things, I make music, I dabble in photography, and I have two cats, who, funny enough, are the catalyst for this entire story. In 2018, I adopted a pair of rescue kittens, and I wanted to keep an eye on them as they slowly took over my apartment. I had an old IP camera from around 2002, but this was now 2018, so I figured I could get something a little better. Amazon has a wide selection of IP cameras, and some are marked as Amazon's choice, and there's quite a few with decent enough reviews. This SV3C model, for example, is $40, and it has four stars with nearly 2,000 reviews, so it sounds like a pretty good deal. I figured this would be like any other network device, where if I wanted to view it externally, I would need to forward a port. But once I hooked it up, I realized I could immediately view it from my phone, no setup required. This shook me a little bit, because it implies it's either doing something to get around my router, or it's sending my video through some third-party servers. Either way, I'm freaked out, so I load up Wireshark. And as we can see, this camera is very chatty. It's pinging three different servers every 40 seconds, two of which are in China. And as I continue watching, I start noticing that sometimes my video is being funneled through mysterious servers all over the world. At one point, I even see this camera talking to a Comcast IP in a different state. It turns out a lot of people have noticed a sketchy behavior. There are a lot of concerned reviews talking about this suspicious traffic on UDP port 32100, not just on my brand of camera, but on many, many different models. So now I'm extremely intrigued. What is this exactly? Well, this is peer-to-peer, and I'm sure the name is conjuring memories of file-sharing networks like Napster and Kazaa, but the context here is a little bit different. In the context of IoT, peer-to-peer is a feature that lets people connect to their device from anywhere in the world without any special setup. You have to remember, some folks don't even know how to log into their routers, never mind how to forward a port. So, vendors started coming up with ways to make connecting easier. To achieve this, devices with peer-to-peer will automatically work around both NAT and firewalls, so anyone from the outside can connect in. But thankfully, your device doesn't have any gaping security holes, right? So where does this feature come from? Peer-to-peer is not typically something that's developed in-house by device manufacturers. There are actually several different players in the industry who develop peer-to-peer solutions, and device makers will get it from them. So this originates from higher up in the supply chain, meaning any faults in it are going to propagate downward very quickly. The largest is probably ThruTech, whose Kali platform is in use by over 66 million devices. If you've got a wise camera at home, chances are it's using ThruTech's platform. This talk will focus on two products in particular, CS2 network peer-to-peer and Shenzhen Uni iLink peer-to-peer. According to CS2, their product is in use by over 50 million devices, and iLink peer-to-peer is actually a functionally identical clone of CS2. Its market share is relatively smaller, but it's still present in over 3.6 million devices. So with peer-to-peer existing in over 100 million devices, it's obviously a popular feature, but what are the risks here? Well primarily, peer-to-peer by its very nature is meant to expose devices, and this should be pretty scary given IoT has a pretty notorious history when it comes to security. But convenience always wins, so devices are now punching through your firewall by design. In many cases, there's no way to turn this feature off, and anyone that has your device's unique identifier or UID can easily connect to it. I'll be demonstrating how these UIDs aren't exactly all that hard to come by. These devices are typically running Linux under the hood with everything as root, of course. So when you pair that with how they're also essentially online without a firewall, the insanity of this whole concept really starts to set in. While there are obvious risks like spying on someone's camera or disabling their alarm system, those pale in comparison to remote code execution. Malware leveraging the ability to punch through firewalls could really spread like wildfire. So now we know what peer-to-peer does, and we've come up with a few ways that we can potentially have fun with it. Let's get a bit deeper here and see what one of these peer-to-peer networks actually looks like. A peer-to-peer network is governed by peer-to-peer servers, and these are our gateways to millions of devices. These servers are imperative to this whole system working properly, and the reason for this is these servers actually orchestrate everything. They manage devices, and they also are what handle connections between users and their devices. You can basically think of these as command and control servers. There are hundreds of these on the internet, because manufacturers will have dedicated servers for their own devices. It's very common to see these hosted on cloud providers like Alibaba and AWS, and they typically listen on UDP port 32 100. Then of course you have devices, and these could be anything, but the common theme here is these will be 100% internet accessible. A very important concept here is that all of these devices will have a UID or a unique identifier, and if someone who knows your UID they can connect to your device, so this isn't something that you really want falling into the wrong hands. These UIDs are generated by the peer-to-peer providers and burned into the device during manufacturing, so if someone learns your UID this isn't something you can change. Let's have a closer look at these. A UID has three separate parts, a prefix, a serial number, and a check code. The prefix is basically used for product grouping, and a single vendor may have several of these, and their server will only respond to their own specific prefixes. The serial is simply the device identifier, and these are effectively just sequential numbers. And finally, the check code is a security feature. It's used to protect UIDs and also try to prevent device spoofing. A peer-to-peer server will reject any request where the check code is not correct. And finally, there are clients. A client could be any desktop or mobile app for connecting to a device. It could be for viewing video, retrieving files, or even configuring an alarm system. But before a client can do any of that, the user needs to put in their UID so the client can make the connection. And under the hood, this whole thing runs on a fairly basic UDP protocol. There are control messages for setting up connections, and DRW or device read-write messages that actually wrap the application data. Messages are really just C structures with a four-byte header, and a signature of this protocol is its magic number. The first byte of any peer-to-peer message will begin with hexadecimal f1. And thanks to debug information being left in, it was fairly easy to reverse this protocol, and I was actually able to create a Wireshark disector for it. This was done using a plugin called Wireshark Generic Disector, which lets you quickly write disectors using a declarative language. I'll actually be releasing this today, so if you have any devices that use P2P, you can use this to get a little more insight into what's going on. So, with all that, we're finally at the crux of this. How does this connection actually happen? Or more importantly, how does peer-to-peer punch a hole through your firewall? As I mentioned, I was able to access this camera without forwarding any ports. The magic trick here really isn't magic at all. Peer-to-peer simply takes advantage of a particular behavior present in NAT and firewalls. UDP hole punching is a way to establish direct connections in the presence of NAT. For those unfamiliar with the terminology, NAT is just network address translation, which is used by routers to forward traffic from the internet to the proper host on the LAN. The router does this by watching outbound packets and automatically creating inbound rules to correctly bring the response back in. The router needs to do this because if it didn't, we wouldn't be able to get responses back from our requests. But consider if two networks that are using NAT want to talk to each other. Any packet sent to the other is going to be dropped because no forwarding rules exist. So how can we get around this? Well, this is actually one of the main functions of the peer-to-peer servers. Both sides can use a peer-to-peer server to swap addresses with each other. And at that point, they can go ahead and send packets at each other. This will punch holes in their respective NATs and let traffic from the other side come through. In summary, just by sending UDP packets to each other, both sides can create a direct channel of communication. So this demonstrates how this works. When a connection request happens, the peer-to-peer servers will tell both sides the other's information. The device will send a packet to the client and vice versa. And even though these may not actually be delivered, that doesn't matter. The point here is just to open ports. After both sides have done this, then they should be able to talk to each other. But the problem with UDP hole-punching is it doesn't always work. It usually works with most home networks, but it might not work in more complicated enterprise or mobile networks. For this situation, a client in a device can use what's called a relay server. Which exists on the network specifically to help clients and devices make connections. As long as both sides can connect to the same relay, then it can proxy data between them. What's really fascinating is that some networks will actually use people's devices as extra relays for the network. And that means someone else's camera may very well be proxying your own video feed. These are called super devices, and this behavior is kept secret from users. They are never asked if they are okay with having their bandwidth donated. And if they have quotas, this could certainly cause a problem. And as sketchy as this sounds, this concept isn't actually unheard of. The more standard terminology for these is super nodes. And there are relatively a common occurrence in peer-to-peer software. Skype actually used to do this too, but at least people had the ability to opt out of that. Spoiler alert, we're going to have fun with these. Okay, so with the background out of the way, now we can start getting to the really good stuff. Since these networks are supporting millions of devices, is there anything we can do to hunt them down? And specifically, is there any way we can guess or generate UIDs? If we want to do this, the first step is going to be to find some peer-to-peer servers. Again, these are essentially gateways to devices, so finding them has a lot of value. I started doing this by looking at desktop and phone apps, as these are obviously going to be using a vendor's servers. However, this is a bit tedious, and there is a much more efficient way to do it. We can use NMAP's host detection to do this. The protocol has a simple hello message, and any valid peer-to-peer server will come back with an act message. So we can configure this probe, and then run it against the entire IP space of a cloud provider like AWS or Alibaba. AWS in particular is very kind and provides an up-to-date listing of their current IP ranges. I've come across 618 peer-to-peer servers since 2018, and some basic fingerprinting has shown that 86% of these are CS2, and 14 are iLink P2P. So next, if we want to use these servers to find devices, we're going to need to find out what prefixes they support. Like before, I started looking for these in apps, and even Amazon reviews, because sometimes people will include their UIDs in their photos without realizing what they're doing. These two reviews have 11 different UIDs, and they're using three different prefixes, but sifting through these is also kind of tedious and quickly provides diminishing returns. So what is a better way? Well, when we try to connect to a device, the server will respond with an error code if the UID isn't valid. But that error code will change depending on if the prefix you asked for is valid. So I wrote a script to brute force prefixes. It continuously sends requests and saves any prefixes that appear to be valid. There's no rate limiting or checks to prevent this sort of thing. And depending on the latency between you and the server, you can find all three letter prefixes in about an hour, and all four letter prefixes in about 36 hours. In total, I have found about 488 distinct prefixes. And while the average server hosts about four prefixes, there are a couple that have 130. So we're at a point where we've got servers, and we know which prefixes they support. We can also predict serial numbers because they're just sequential numbers. If we want to find UIDs, the problem is now the check code, which exists precisely to stop this sort of thing. If the check code is off by even one letter, the UID will be rejected. These are five letters long, and while that's not an impossible key space to brute force, it's still enough to thwart any efforts to find a large number of devices quickly. So how can we get around this? Well, with iLink P2P, luck would be in our favor. I found a variant of the iLink P2P library that contained the algorithm they used to generate check codes. It basically hashes a few components using a modified MD5, and then puts together the letters from the hash to create the check code. This was apparently included in the library to do client-side UID validation, which is odd given the peer-to-peer server does this anyway, but someone thought it was a good idea to ship this. This is a big deal, because we can now generate any valid iLink UID, meaning we can now connect to any online iLink device. And with this ability, we can start to do some very interesting things. For one, we can now find out how many devices are online, but what would be even more interesting is finding out where these devices are. We can use a UID to get the IP address of a device, and using that, we can approximate its location using IP geolocation. So what you're seeing here is an enumeration script I wrote. It generates a UID, grabs its IP from the servers, and then looks it up using Maxmine's Geolite database. This is going at 10,000 requests per minute. Taking this another step further, I was able to visualize this data with Leaflet Marker Cluster and OpenStreetMap. Of the 3.6 million devices I've seen, Europe makes up about 21% of this, with France, Italy, Germany, and the United Kingdom all having over 100,000 devices. North America makes up about 8% of this total, with the US seeing over 205,000 devices, and Canada and Mexico both having over 40,000 each. Finally, Asia is the most heavily impacted with over 1.8 million devices. China and Thailand alone make up more than half of the entire dataset. Horrifyingly, lots of these devices use default passwords, meaning the UID is enough to gain full access to the camera. I disclose this to Shenzhen Uni in early 2019, but despite multiple attempts, I never received a response back from them. And despite this vulnerability, I've continued to see an increase in the use of iLink P2P. In fact, thousands of new UIDs come online every single month. CS2 doesn't appear to be affected by this vulnerability, but that's not the end of the story. So now I have a way to directly connect to millions of cameras, but I'm not really interested in spying on people or talking through their cameras. That's creepy and frankly, we're not in middle school here. What we really want is shells. So let's find some camera vulnerabilities. When planning for this, I really wanted to focus on a big vendor for maximum impact. Shenzhen High Chip Vision Technology is a major manufacturer in the IP camera industry. They caught my attention because as I was looking into different brands, I started to realize that a huge number of them were ultimately manufactured by High Chip. In fact, close to 3 million of the iLink devices I have found are using High Chip prefixes. This sort of practice is very common, where manufacturers will sell white labeled equipment to vendors who then resell it with their own branding. This is a list of 50 different brands that are actually using High Chip under the hood. So given their ubiquity, any vulnerabilities in High Chip's firmware would be both widespread and very accessible. So to get started, I began analyzing their firmware. One function of interest was this read request method, which is used to handle any commands coming in over peer to peer. It even handles login, and we've got a 1024 byte buffer to read data off the wire. The message header has a field that says how big the payload is, so of course that's validated, to allow 10 times the size of the buffer. Okay, so we've clearly got a buffer overflow here, but how easy is it to exploit this? Because after all, there have been defenses to stop exploits like this for years, and the library doesn't seem to use any of them. So one very reliable RopGadget later, and we've got ourselves a system call and then a reverse shell with root privileges. Obviously any vulnerability is bad, but the most devastating really tend to be those that are pre-auth, because any anonymous person can exploit them. In this case, as long as you have a UID, you can get a shell, and we now have all the UIDs. Once you get a shell, you can run binaries, you can browse the file system, and you can even access the camera's password. You can even directly invoke the camera's Wi-Fi utilities, and here's why this is a really bad thing. You can do a lot of really crazy things with base station MAC addresses. Google has an entire API that uses MAC addresses to perform disturbingly accurate geolocation, and the more information you give it, the more accurate the results are going to be. So once you get a shell, you can use it to scan for nearby Wi-Fi networks and potentially pinpoint the exact location of a camera. If you want to try this yourself, check out Google's Geolocation API, feed it some MAC addresses of nearby wireless networks, and see just how accurate it can get. It costs literally pennies to do this. It's hard to say exactly how many devices are affected by this vulnerability, but given I found this bug in almost two years worth of firmware, I think it's a pretty safe bet to say that this could be weaponized, but thankfully High Chip did release a patch for this in June 2020. I did find another vulnerability worth mentioning. It's not exploitable over P2P, but it's still pretty amusing. Up until recently, you could anonymously reset the password of any High Chip camera just by being on the same LAN. The reason for this is apparently some devices don't have physical reset switches, so instead you can just send a magic packet to the camera and it'll reset the password to admin. I'll be releasing a script for this, but I actually found out High Chip provided a tool to do this already, so I'm not sure if this qualifies as an exploit per se, but have fun with this. Okay, so we'll get a few more tricks up my sleeve. Let's talk about man-in-the-middle attacks. When people hear man-in-the-middle attack, they might associate it with something that's restricted to the local network. Sniffing packets at a coffee shop, for example. Peer-to-peer should be pretty scary by now, but we haven't even talked about how it lets you intercept traffic over the internet. This comes from the fact that, as we now know, the peer-to-peer servers are responsible for coordinating all connections. Clients and devices obey the orders of the peer-to-peer servers, so if we can influence that process in any way, we may be able to insert ourselves into a session. This is particularly scary, because despite being a transport layer for the application, peer-to-peer doesn't actually perform any encryption or identity checks. These measures are left entirely up to the application, and as you can guess, most don't do a very good job of this. In fact, some even outright lie about it. This is a screenshot from VSTARCAM's app, which assures you that it is building an encrypted channel, but as you can see from the packet capture on the right, there is absolutely no encryption happening here. So, how do we get in the middle? Well, once per minute, a device is going to log into its peer-to-peer servers. This action tells the servers where the client needs to go when they request a connection. The problem is, this login message just contains the device UID. Absolutely no device-specific secret is used, which means if we have a UID, it's very easy for us to forge this login message and confuse the servers. And if our timing is right, when the client requests a session, the peer-to-peer server will tell them that we're the device. And at that point, the client will happily try to authenticate to us. And yes, that is base 64. It's always base 64. So, we have the password and now we can turn around and connect back to the real device. This works with both CS2 and iLink P2P devices. CS2 did try to patch this problem, but it's pretty easy to circumvent. The login message is encrypted with what's called a CRC key, but a vendor's devices all use the same CRC key, so you can just grab it right from the firmware. And with some servers, you actually don't even need to go this far because it will still let you log in with the old login message. So there's not much of a point. All this attack is certainly doable. There are a couple of drawbacks. You need to have a specific UID to target. You need to know knowledge of the protocol and you actually need to wait for the person to try to connect. However, there is another attack that has none of these requirements. In fact, there is a way that someone with practically zero hacking experience can start owning handfuls of devices. Let's talk about super devices again. These are devices in people's homes that are proxying sessions for other users. A lot of vendors use these, so the chances of you buying a camera that will act as a super device is pretty high. And as I said, P2P doesn't actually perform any encryption. So what happens when a vendor's crypto sucks too? That means that anyone can buy a device and access other people's traffic with it. It means literally anyone on the planet can sniff your entire session and capture your password or your video without you ever knowing. There's no need to expend a ton of effort setting up a man in the middle attack when this exists in the product by design. There are plenty of devices that don't even use encryption for sessions. A little PCAP parsing can give you a raw video stream that you can immediately watch with something like FF Play. And all the while, users will have no way of knowing whether their connection is being intercepted like this. In fact, it's a risk that they're taking every single time that they connect to their device. As an added bonus, this has the side effect of leaking UIDs during the peer-to-peer handshake. So alternatively, you can just take the credentials which you also probably captured and connect back to the device yourself. I actually developed a proof of concept of this just to collect UIDs by posing as a super device. And by doing this, I was able to collect over 236,000 unique CS2 UIDs over the course of 10 months. So even though CS2's check code algorithm has yet to be cracked, this is a very reliable way for anyone to collect UIDs. The funny thing is, CS2 actually mentioned this exact concern in a marketing deck. They said a super device cannot spy on any data it relays because there is no API for this. But of course, that's insane. You don't need an API to pull this off. It's absolutely trivial to set up a camera and then leave Wireshark running to capture every single packet flowing through that device. So that's exactly what we're going to do. This bullet cam is running iLink P2P. So against my better judgment, I'm going to put this on the internet and we'll run Wireshark for a while and see what we come back to. Okay, so I left Wireshark running for a few hours and we can see that this camera has stayed pretty busy. I have the P2P disector active so we can clearly see what it's been doing. Each of these hello messages is the peer-to-peer servers telling the camera to reach out and try to help someone with a session. So this will generate some handshaking with both the client and the device. Of particular interest is these relay packet messages which will actually contain a UID of the device that we're trying to relay for. But we want to see some actual session traffic and we can locate this by filtering for message DRW messages. So I'll go up here and apply that filter and sure enough looks like we've got some traffic here. So if we actually pull up the resulting conversation here, well that sure looks like base 64. So if we copy that, yep, nice secure password there. So there's some other information in plain text over here. We can see some information about the camera, the firmware version, and it looks like some other sort of name or identity information. But what about video? One of the ways that this DRW concept works is there are message indexes and message channels. And each channel is going to be used for a particular purpose. So in this case here, it looks like there's a lot of sort of authentication or negotiation going on on this channel zero. But on this channel two, we start to see binary data. So what I did here is I actually wrote a quick script that is going to take the data from this packet capture and just concatenate it together so we can pipe it into a video player. So what we're going to do is just go and export packet dissections as JSON, call this export.json and make sure we hit packet bytes. Save this. And then we will go here and do node json to vid export.json. Pipe that right into ffplay. And we have some chickens. How about that? So final thoughts. I had a feeling things weren't going to be pretty when I started down this rabbit hole, but I never could have imagined some of these things. I've reported all of these problems to the vendors. Shenzhen Uni never did get back to me. And the number of devices using iLink P2P has steadily increased since I first reached out to them. I've tried to contact a lot of vendors in China over the past few years and it's honestly pretty rare that I get a response at all. So I'd also really like to give credit where it's due here because both CS2 network and Hychip kept in close contact with me earlier this year, even as the COVID situation was first emerging in China. It was refreshing to see my concerns taken seriously and especially during such a difficult time. The issues with CS2 are slated to be fixed in the next release and Hychip just released patches for their problems in late June. With the warm and fuzzies over, boy, things are not looking good. Even with patches on the way, I think it's safe to say that everything you just saw is going to be a problem for a long time. And this is because a lot of these peer-to-peer problems are outright architectural flaws and they can't really be fixed retroactively. I think even if they could be fixed, it doesn't matter because most people don't update firmware anyway. I've seen some devices with firmware versions going back half a decade. In June of this year, I worked with the UK consumer watchdog group, Witch, to try to get some of the affected devices pulled off of shelves. Amazon, who sells more than 20 vulnerable brands, didn't respond at all. And eBay said to use IP cameras without an internet connection, which isn't even possible with a good number of these. So, you know, thanks for that advice. I do hope that this was inspiring to some of you because I think there's a lot of opportunity here for further research. I think it's pretty clear that peer-to-peer is a pretty nasty vector. And I'm sure there's a lot more vulnerabilities in devices beyond what I've discovered on my own. I also haven't really had a chance to dig into other peer-to-peer platforms. For example, Kale, which is used by over 66 million devices, including Wwise cameras. And I know they use a similar paradigm where a UID will get you device access. So, I think there's certainly opportunity there. And while I focus my efforts on high chip, there are definitely other massive device manufacturers out there. In general, I encourage folks to climb up the supply chain a bit if you're looking for vulnerabilities that could be both widespread and potentially high impact. I'd like to leave you with a few reverse engineering tips that have led me to some major discoveries. First, if you're doing reversing, be sure to collect as many different samples and variants as you possibly can. Run diffs against them, look for abnormal changes in file sizes. Some may leave in debug symbols, extra logging, and alternate functionality. So, those are all new avenues of exploration. Also, APKs are a great way to check out things with minimal effort. Java can decompile into beautiful readable code that can very quickly reveal interesting things. And finally, and most importantly, throw every single interesting file name or magic string you find into GitHub. Seriously, it is incredible how many people push sensitive source code to public repositories. And this has helped me find everything from SDKs to documentation to full-on source repos for firmware. Finally, I'd like to give a shout out to the folks who published some of the initial research on peer-to-peer. Their work was really inspiring and key to the success of my own research. So, cheers to you guys. And that's a wrap, my friends. Thank you all so much for coming. Happy hacking, and be safe out there.