 Next, and last today we have Daniel, we have Morrow, and we have Jen who are going to talk to us about Smart City stuff, and taking over Smart City stuff and generally causing all kinds of chaos and havoc. Why don't we give these guys a big round of applause. Hi everybody. So, some quick introductions. My name is Daniel Crowley. I am the Research Baron for X-Force Red. You might be wondering why I have such a silly title. Well, I wanted to be director of research, because I do direct the research program, but directors apparently reserved word at IBM, so I had to kind of work around that. I pitched them a couple of things including tyrannical research dictator, they didn't like that. I pitched research sultan, but there was some suggestion that maybe there was some cultural insensitivity there, so we passed on that one. But I do hold, and I'm getting ahead of myself here, but I do hold the noble title of Baron in the Micronation of Sealand, so as long as you respect its sovereignty. Well, you marry Libertas to you, buddy. But I'm a Baron as long as you don't mind me having paid $40 to get that title. That's $40 I ever spent, by the way. I've been doing pen testing since 2004, and I've been a hobbyist before that, and I also happen to have an interest in physical security and a bit of a lock sport enthusiast. Oh, I didn't advance the slides quick enough, did I? Hi, I'm Jennifer Savage. I'm a security researcher at ThreatCare. I'm also a member of the Black Hat Review Board. I've had a couple decades experience in tech, including software development, management, vulnerability management, vulnerability assessment, penetration testing, security research, etc. I'm Mauro. I've been doing pen testing for many years. I've been passing through different areas, like secure architecture, developing, system administrator, and I love to find flags and correct them. So you might be kind of curious about the term smart city. What exactly makes smart city technology? Well, it's a pretty broad blanket term, kind of like internet of things. It's more specific than that, but it's still in the Venn diagram. It's a pretty large bubble, and there's lots of little circles within that. So for instance, they're the industrial internet of things. Cities have to have utilities. You know, you have to have water infrastructure and power and all that sort of thing. So when you have technology running that, that's part of smart city tech. Something that fits more squarely into that is urban automation. So an example being automated waste trucks that drive around and pick up people's trash cans and read RFID tags in the trash cans so that they have an exact log of when each trash can was picked up and which trash can belong to whom and how heavy it was and all that sort of thing. And then you have public safety things like police body cams. You have things like emergency management systems. So you have systems that detect impending disasters and allow people to respond quicker. You have intelligent transportation systems, devices and software that try to reduce traffic congestion. Things that will detect how much traffic is on a stretch of road and then communicate with the traffic light down the road to say, okay, you're going to want to open it up a little bit. And then you have metropolitan area networks which are just sort of city sized. They're like lands but city sized. So you might have public internet kiosks or you might have city wide Wi-Fi provided just for all the citizens to use. And there's more smart city tech than just this. There's lots and lots and lots of different tech but these are different just example areas. So when it comes to privacy there are a lot of concerns with smart city technologies. It's a very different thing when you can choose what you have in your home. You can choose not to have IOT devices. You can choose not to have a smart TV in your home. But you can't really get much control over the fact that outside your home, right outside your door, every street lamp on your street might have a camera in it. And that's what we're talking about when we talk about smart cities. Everything's monitored. There are billion sensors everywhere. It could become the case that there are legitimate purposes that are subverted by malicious actors. And so if a legitimate person could use a connected vehicle infrastructure, like a vehicle's infrastructure hub, to monitor the location of a car or use cameras to monitor the location of a person walking down a street, then a malicious actor could use it for the same purposes as well. So speaking of intelligent transportation systems, this is one of the biggest pushes in smart city tech. There is a lot of advancement, a lot of adoption of smart city technologies. I was lucky enough to speak with a gentleman from the Federal Highway Administration who corrected me a little bit on this slide. So there was, as far as we can find, a proposed OBD3 standard at one point, which was basically OBD2 plus a little transceiver. But the more we looked into it, we weren't sure if it was a thing that was pitched a long time ago, around circa 2000, and then died because it was obviously a terrible idea. Or it might have actually been a hoax because we chased it down and it was, some of these things looked pretty odd. So thank you to Ed from the FHWA for steering us in the right direction on that. So something that exists in Hangzhou, China, is what's called the city brain or traffic brain, which is a gigantic intelligent transportation systems project that aims to reduce the traffic problems in Hangzhou. And as a Western, this particular quote kind of horrified me that in China people have less concern with privacy, which allows us to move faster. And that, for context, is being spoken by the manager of AI at Alibaba, who created, Alibaba created the traffic brain, and he's speaking about it at the World Summit AI in a talk about the traffic brain. But it's not just in China. There's also street lights with cameras built into them. And it took me a while of staring at this picture before I could actually see the cameras in these street lights. But sure enough, they are there. Now, in addition to that, lots of cities are either talking about or have already deployed facial recognition software to their surveillance cameras. So in 2017, a former government, red leather, yellow leather, unique New York, a former government official for Singapore said that they want to deploy cameras to every single one of their lampposts, all 110,000, and put facial recognition software to work on those cameras. And if you think that's crazy, Dubai won up them. They want to make the first police station manned entirely by robot police by 2030. There was a movie about this. It didn't end well. So let's talk about reconnaissance. How do you discover what's in a city? So you just start with search engines. That's the most obvious place. In fact, everything that you need in order to discover what's in a city can be done entirely through passive reconnaissance methods. So we started with case studies made by manufacturers who talked about what their devices were being used for around the world. And you can get some really interesting information about the deployments of those devices just by looking at the case studies. There's also news reports. So local news will quite often cover smart city developments. It's all new. It's all fascinating. And it's all recorded by the news. And then, oh, the open data initiatives. So some of you may have seen a lot of open data initiatives offered by various government agencies and cities will quite often have their own open data initiatives where they publish data quite often taken from those smart city sensors. And then some city contracts are public. I'm looking at this upside down. It's kind of hard. So some city contracts are public. So in the US, everything's foyable. So you can look up a purchase order online if you just Google for it properly. You can check bid net, et cetera. And then you can see what your city has. So also, public systems are already mapped. So there's some really great search engines out there that are used for mapping out internet infrastructure. So if you first identify the IANA ranges for the city that you're doing recon on, then you can just check showdan and census by searching literally for that IANA range. There'll be an ID for it. And then lastly, physical recon. So just going outside basically and looking with your eyes. You can do traditional methods like war driving, looking for Wi-Fi. There's all kinds of different war driving methods out there. There's even war driving for Laura Wann. You can find, I think Travis Goodspeed has some really great stuff on other types of war driving out there. But all of this requires that you actually log off and walk outside your home. So a bit of a challenge for some of us. And then source code repository. So a lot of this stuff is open source. You can check github, bit bucket, GitLab. And then lastly we found this thing called OS ADP run by the Federal Highway Administration. And I was recently informed that they actually are requiring that a lot of these manufacturers open source their software so that independent security researchers can do this kind of work. And it really enables us to try to find these kinds of flaws. So I'm really happy with that. So let's apply these methods real quick to a city. So Austin, Texas, which is the city I live in. Here's a roundup of some news reports that were done about Smart City Tech that was being deployed. So autonomous transit shuttles. A smart street, so 6th Street, which is a real big party street there. They were going to turn into a smart street. This is city up. It's basically a website all about Austin's Smart City initiatives. And you can find lots of details there. Here's the census results for the city's IANA ranges. This actually covers a lot more than just the smart city tech that they have. It's a list of like all of the systems that are running on their range. And this is kind of neat. At all of the low water crossings in Austin, they have flood sensors and these boxes are on the side of the road and you can just walk right by and see them. And here's how they transmit here. Interestingly, after we started doing our research and I became concerned about whether or not flood sensors might be messed with and nobody would know to go check to make sure it's legitimate reading, somebody went ahead and installed cameras without us even reaching out or talking to them. They installed cameras at every low water crossing. So now when you check the ATX Floods website that reports the results of the flood sensor, you can verify it visually to see whether or not the low water crossing actually is flooded. And ATX Floods, by the way, is a website you can use to plan your route around the city during times of flooding because it floods quite frequently in Austin. And then this is actually just a purchase order we found by Googling for purchase orders like we said before. And this one's for police body hands, which falls under the safety subset of things. Right. So I imagine some of you are here just for the bugs. So here's the bugs. So the first device or rather devices that we looked at were in a device family called the Ilon devices from Echelon Corporation. We looked at the smart server, which was previously called previously as the Ilon 100, and its successor, the Ilon 600. Now both of these things have the same function, but different feature sets. So basically, you might know something about ICS security, but if you know anything about ICS security, the general recommendation is never ever attach these things to the internet, just like put them in an air gap network and never let anybody touch them unless they are already authorized to touch them. So we found that this was a pretty interesting device because what it does is it hooks up ICS devices to IP networks like the internet. And actually we found about 450 smart server devices exposed to the internet via census. So that's great. So these things talk to a variety of different devices over various protocols like it speaks to the very popular Modbus, including the Modbus over TCP IP variant. It speaks backnet over IP. And it can also speak to any sort of web services that take SOAP communications. Hooking this up was kind of a harrowing experience for me because it doesn't take, it has these screw terminals to receive power and I couldn't just cannibalize a power, like an ATX power cable and plug in. I had to get like a little power adapter and I, my did a terrible wiring job. I actually, when I hooked this up, I plugged it in on one of my like outside ports on my concrete patio and I was wearing like safety goggles and oven mitts because I was like, is this going to blow up? Is there going to be fire? I'm not an electrician. If there's anyone from OSHA in the room, I'm sorry. I probably did bad things there, although it wasn't at my workplace, anyway. So, we found a bunch of things here. We found, first of all, that there were default credentials. One interesting thing is that there is a web server and an FTP server and there are separate credentials for both. So, you might have one of these things and change the web server password, but not the FTP password. In fact, we sourced one of these devices from eBay and found that while the web application password had been changed, the FTP server password had not. So, we were able to, because of the fact that the credentials are in a configuration file in plain text on the FTP root, we were actually able to get the original credentials for this device, which is scary in and of itself, but that's, you know, that's neither here nor there. One interesting thing about this is even if the default passwords have been changed, the default configuration for what to authenticate on the web application does not include the API, which does most of the heavy lifting. The user interface which calls the API is authenticated, but if you know the right way to make the calls, you can just invoke all sorts of fun API functions like, hey, change the FTP credentials to blah, blah, blah. So that's good. And this is, of course, over plain text HTTP, and it's not FTPS or SFTP, it's just FTP. And on top of all that, there's another authentication bypass bug. So even if you change the default configuration in both passwords, you still have an authentication bypass bug. So I talked about retrieving the clear text password via FTP, but you can also replace the binaries on the device over FTP. You can fiddle with the ICS gear that's connected to it in the way that the legitimate administrator would or could. And if you want, you can also just change the IP address and prevent anybody from being able to connect to it. So here's how the authentication bypass works. What the, the iLon devices do or did before patches were made available. They looked at the path to see does it match any, does it match any of the items that I have in the configuration file for the authentication section. So in this case, we're hitting an endpoint that is authenticated by default. So forms slash forms slash echelon slash star is a default item. So this falls under that, that pattern, but it's just string based matching. It doesn't do any sort of canonicalization on the name. So if we instead request slash forms slash slash echelon slash anything, it says, okay, this isn't slash forms slash echelon, there's another slash here. I don't need to authenticate this. And then it hits the operating system, the extra slash is thrown away, and well, you know the story from there. An interesting note, the iLon 600 units have this weird thing called security access mode, which basically means you have to stick a paper clip into this thing and hold it in there as you reset it. So, you know, like either two paper clips or just pull the power and put it back in. So you have to go through this process in order to put it into a mode where you can change credentials. So you can't really get the plain text if you're just using the authentication bypass. And by the way, the default configuration is secure, or at least we didn't find any problems with the default configuration on the iLon 600, but this authentication bypass works on it. So you can still use all of the ICS stuff that they've configured into their iLon 600 when you use this authentication bypass bug, but you can't really change the FTP credentials and backdoor the device or anything like that. But what you can do if you really just want to be a jerk is change the IP address since that's outside of the purview of the security access mode. Now, something interesting that we stumbled across that we weren't looking for as we were doing this research is that there was an exploit for the default configuration bug that affects the API. And this was interesting to discover. This was posted to a GitHub gist back in August of 2015. The comments and the code shown here suggest that this is older than three years, so that's interesting. We contacted iLon, when we contacted iLon to disclose the vulnerabilities that we discovered, we also let them know about this. They were unaware of this exploit and they were unaware of the bug that it exploits until we spoke to them. So that's interesting and it tells us something that we normally don't get to know, which is that, yes, there are people looking for these things and finding them and not reporting them. So. Okay, this is another one. Can you hear me? This batel between iHUB. What it does? Batel between iHUB manages communication between connected vehicles and interpretation infrastructure. It translates data from multiple sources and protocols that are using transportation. It has a modular infrastructure. The system can help deliver messages that are useful for transportation applications like red light, violation, speed warnings, over high warnings. With iHUB, it was possible to gain access because it has a hard-coded password. It has various different API keys that you can access with authentication or you can bypass. You can perform cross-scripting attacks. Executing SQL injections in the API is also possible and you can gain access with authentication. With all these flaws, an attacker or adversary can do many things. He can track vehicles. He can send, say, failed messages or can change the messages. He can create traffic or modify the database to change something inside the database that may create some different behaviors. Or just shut down the iHUB so nobody can send or receive messages from the iHUB. This shows why it is possible to shut down a device that is running before iHUB because as you can see, it doesn't require any authentication. It doesn't need any API key. Before iHUB has an API and this API requires a key. Even if the key has been changed, it is possible to access the key through the web server without authentication. As you can see, using a string compare function compared to a string that we will see pretty soon what it is. Even if the key file was restricted, the input key and the compare key or the write key are compared using the string compare function which has an odd set of return values and different conditions. As you can see, there is a list of return values that we can use after. They mostly make sense, but something interesting happens when comparing strings and arrays. The string compare function returns null with a warning. If this warning is ignored, something can happen here. When zero the value returned by the function when two strings are identical is compared to null. Remember that we saw the compare function trying to compare the correct key with the input key using that function. The comparison returns true as long as you are not checking types too carefully. This means if you add left square bracket and right square bracket adding a key and the URL, any key would be the right key. That means you have access to the API always. In the case you can call all the features that the video I have is using. Lastly, resize call injection and version 3 as well and the login page so you can strike all the user names and passwords without any authentication. The final device that we looked at was called the libellium meshlium. The meshlium is a part of an ecosystem that works on sensors. These sensors can detect all sorts of things and the meshlium is actually designed to be able to communicate with even sensors that are not produced by libellium themselves. They have their own sensor ecosystem called wasp mode and their own set of sensors that they sell that plug into wasp mode pretty easily. Some examples of these are radiation level sensors and water level or distance sensors which are used for example in flood prevention by detecting water levels. We have sensors that detect rainfall and wind speed so depending on what this is used for and we do have some limited information provided through customer case studies about what this is being used for. For instance, we know that the Spanish government is using these devices to detect radiation levels around nuclear power plants. We know that there is a dam in somewhere in Europe where the meshlium or the wasp mode ecosystem is being used to detect water quality. If you are using a meshlium, there are some interesting problems there. The meshlium essentially just acts as a hub and a centralized location for storage or to be sort of collected and then passed on. It takes in data from all these sensors and then passes them to either a database or pushes them up to some cloud platform. So what we found was that there were a number of end points on the application that were just missing authentication entirely. So they could be invoked directly and didn't require any username or password and some of them could actually function as a whole, like correctly, some of them couldn't. A number of them actually took user input directly and fed it into a shell command without any sanitation. So if you take a look at the last line here, you can see, and this is pretty much, there's this, I don't remember if this is the start of the file or not, but this is one of the exploitable cases of this. So if you just put something like, let's say semicolon rmrf slash in as your link variable and you have your type variable set to download update while interesting things happen. Now you might be saying well, Daniel, first of all, no preserve root isn't in that example, okay, okay, pedantic, sure, let's add that, but you're still the web application user, so you're not going to be able to do much. Well, I have a solution for you which involves the fact that the web application has the ability to sudo without any password. So if you just do semicolon sudo rmrf slash no preserve root, well, funny or terrible things happen depending on, you know, what side you're on. So we want to do a little demonstration. We have a whole dam simulator built into an aquarium based on a meshleum system. We were not able to use that, so we instead have a backup video which there we are. I guess there were AV issues with bringing the dam here, so. Cars, rock wall, even a scenic view and a road over the dam. It's all inside an aquarium. While this is a simulated dam, the vulnerabilities we're about to use are very real. What we've got set up here is a dam with a gate which is controlled by a raspberry pie. And that raspberry pie is controlling the water level based on data sent by this ultrasound sensor attached to a light belly in place. Now this is telling us the water level in centimeters distance from the sensor. Now because of the vulnerabilities, and the fact that this data is being read from the meshleum, we're going to launch a proof of concept exploit and run a little bit of code on the meshleum and corrupt that data making it think that the water level is in fact very low. So the dam is going to open all the way and it's going to stay open no matter how high the water level gets. You can see it reaching the top of the river banks and now starting to spill over onto the edge of the river bank on the other side starting to flood the road. Please don't dismay. We worked with the vendors, reported all vulnerabilities. They were all patched. Cities have had weeks to roll out these patches. Everybody has been notified. So that's the positive side of all of this. Additionally, I think when it comes to the implications of things, you know, as hackers we have to ask ourselves to what lengths do we, independent of the companies who are selling these devices, independent of the cities or the governments that run them, want to go to try to find these vulnerabilities. With the v2i hub, it's fairly simple because the code is open source. With the meshleum, we had to pay 3,000 euros for a meshleum setup in order to test it. With the I-Lons, we got some off eBay used so it was a bit less expensive. But the point is these devices are very, very, very expensive and it can be very difficult for us to get the ability to do the independent security testing that's really required. But as far as the vulnerabilities that we found, here are the implications. Surveillance of connected vehicles. So following a governor around the world or a celebrity or God forbid, even the president. Traffic manipulation. Causing traffic to slow down industry in the city that you live in. And sabotage of disaster warning systems, similar to the dam demo that we showed you, but for something like radiation monitoring where you cause a false panic because you set off the sensors and everybody thinks there's radiation and that's right. But after you've finished setting up your city, it's a fully, you know, a smart city place, I hope that you are also going to set up your IOT paperclip so that you can reset the device when something goes horribly wrong. I hope also that cities will take into consideration whether or not the devices they purchase have been tested by independent third parties on a regular basis, that cities will have their implementations of these devices tested and that the information about the remediation plan for any vulnerabilities found will be made available to the public so the public can feel safe about what's in their city. Thank you so much for coming to our talk.