 So, a very warm welcome to Thomas Rote. He is a security researcher, and his specialty is exploiting techniques and reverse engineering and industrial security. And the talk today will be about Skyder, the gateway to Shell. Yeah. And just one little notice. This talk will be in English and will be translated in German as well. Thank you. Thank you. Awesome. Thank you. Okay. Yeah. Welcome to my talk, Gateway to Shell. Who am I? He already introduced me, but still my name is Thomas Roth. I'm a security researcher. I do a lot of low-level security, so a lot of arm-reverse engineering, cold fire, and so on. And yeah, you can find me on Twitter, or if you want to write me an email, feel free to send me one to thomas at stacksmachine.net. Before we start, a short introduction to the background of this talk. So this year I did some SCADA penetration tests, and I found that while the PLCs and so on are pretty well covered in the security research area, I found that all the small devices that surround SCADA environments are not really well covered. So basically, we have the big Siemens PLCs and so on, and there's a lot of research going on about them. But there are also a ton of other small Ethernet devices involved in industrial networks that are not really researched very well yet. And all devices that we're going to talk about are running their latest respective firmware. Unfortunately, there will be zero days. And these are not theoretical attacks. If you go to Shoden or a similar search engine, you can find tenth of thousands of these devices vulnerable and open in the Internet. So let me give you a quick introduction into the terminology in SCADA, because I know in the title I say SCADA, but actually it should be ICS, which stands for industrial control systems, because basically ICS describes the whole system from your supervision, the big room with all the big screens, up to your PLCs, the sensors, the actors, and so on that you will find in your installation. And the term SCADA just describes the supervision and control centers, so the big screens that you might know from movies and so on, where when the bad guy comes suddenly all the lights turn red. Then there's something called a PLC, which is a programmable logic controller. It's basically like an Arduino just for industrial applications. And they are really easy to program, and you can get them from Siemens or Schneider and so on and so forth. Then there's something called an RTU, a remote terminal unit, which is a small device that generally or well back in the day was only used for monitoring. But today you can actually program a lot of RTUs, so it's kind of a mix between a PLC and an RTU, so it's basically a PLC in a remote location. Alrighty, to the actual topic, industrial control gateways. So when you look at an industrial control network, you'll find that there are a lot of different sensors and actors, and a lot of them speak different protocols. So for example, some might be serial, some might be IP, some might be Modbus, and so on. And so you can buy these small gateways that connect all these different protocols to an IP network. So for example via Ethernet, or even via GPRS, or Wi-Fi, and so on. And I've seen them in almost any industrial installation that I've seen. So for example, they're used in power plants, they're used in water dam control systems, they're used to control the power grid and so on. And the security concept is, hey, but these devices are air-gapped, so it doesn't matter really if they are vulnerable or not fully up to date and so on. But that's not really true, because a lot of these devices, while they might be air-gapped, they also have antennas and they are interconnected by a ton of different wireless protocols, such as Wi-Fi, Lora, or GSM, or even proprietary radio links. So yeah, and even the case studies show that basically in this case, you would have a monitoring network that's connected via the cellular network to control the water mains and so on and check the pressure. Or even worse, they even recommend that you connect actors like valves and water level gauges and so on over GPS, which we know is not a secure protocol to do anything that could be critical. Or you have stuff like water storage tanks that are controlled via Wi-Fi and so on, or even public in the internet. So yeah, these devices are air-gapped, nope. So attacking in the field, I already mentioned, if you go to Shodan, you will find a ton of different devices reachable via the internet and even via GPS. So if you live close to, for example, a dam or something, it's kind of interesting to look at an SDR or similar radio equipment to see what's going over the airwaves, because you will find a ton of interesting stuff. And sometimes you can even very trivially get physical access to the infield devices because they might just be in a white box somewhere hidden and if you break into it, you can pull out the SIM card and it will put you directly into the SCADA network if you're lucky. Don't do that, by the way. So yeah, let's hack some gateways. So the equipment you will need to, and everything in this talk was done on this desk just using these devices. So you really, you just need a laptop. You need an oscilloscope or similar measurement equipment just to ensure that you don't burn out your logic analyzer. You need a logic analyzer, a soldering iron, a multimeter and a power supply. And that's really basically it, because you can hack almost any embedded device that's using these devices. And to find potential targets, I have this kind of map where I first tried to understand, can I get the firmware of the device or do I have to somehow, for example, use JTAG to get it out of the device? Can I actually buy the devices at a sensible price? Because some of these devices cost like 600 euro or so. And if you buy 10 of them, that gets expensive very quickly. And so I need to check eBay and see what devices can I actually buy. And they should be a half what current, because if you look at all the devices like 10 years old or so, they're completely broken. You don't even have to look to start to look at their security. So yeah, the first device that I that I choose to really look at was the Moxa W21508, which is this small device, which is also mounted on the board right here. Mainly because I found the firmware was available and it looked like an interesting device because it has Wi-Fi and so if I manage to break into it, I can jump an air gap potentially. And the W21508 is just a simple device server so you can connect any serial device, any RS485 device simply to it and it will be exposed via ethernet or even via Wi-Fi and you can download the firmware publicly and it's available on eBay relatively cheap, so like 150 bucks or something. So I downloaded the firmware and I looked at the entropy of the firmware and I immediately saw that the entropy is very high, which means either it's very compressed or it's encrypted. Unfortunately, using a tool called BinWalk, which is really useful for looking into firmwares, I saw that there's no compression detected and so it was very likely that this firmware image is encrypted. But I noticed on the web page that before you upgrade to version 2.0 or 2.1 of the firmware, you must upgrade to the firmware version 1.11. And I thought, hmm, that's interesting. Let's look at the release notes for version 1.11. And it turns out that 1.11 adds the support for the encrypted firmware. So I downloaded the 1.11 firmware and sure enough it's unencrypted. And if you've ever done anything with ARM before, if you just look into a firmware hack stump, you can immediately recognize whether it's ARM or not, because the first four bits of each instructions are the conditional bits and those are almost always E. So if you see a hack stump and roughly every fourth byte is an E, you know this is an ARM firmware and it's not encrypted or anything else. And so yeah, sure enough, I ran BinWalk on this image. This time we see there's a huge drop in entropy, which is the boot loader and so on, and then a high entropy, which is basically all the compressed file systems and so on. And BinWalk was able to detect the SquashFS file system and extract it for me very, very easy. And so my goal was to extract the firmware, find the firmware upgrade code, and somehow try to decipher the new firmware. And yeah, so I was browsing through the files and sure enough found a file that was helpfully called libupgradefirmware.sharedobject. And if we look into the symbols, which they luckily didn't remove for anything, there's a beautiful symbol called firmware decrypt. So we load the whole thing into this assembler and we see that there's some fancy, oops, that there's some fancy XORing going on in the bottom left corner and I'm going to walk you through what exactly is happening in this code. So basically first there's a variable called password loaded into the register R2. And then a second counter variable is basically set. And it starts looping and increasing always by four and goes through this whole, oops, sorry, my screen's so here. And it goes through this whole XOR shebang and it turns out that this is the obfuscation method for the AS key. So in password, in memory we have an obfuscated key and we can be obfuscated by just implementing the code we see here in C or in the emulator. And sure enough, eventually this will be used as the key into the ECB 128 AS decryption. And so I implemented the whole thing in C. It was almost a copy paste from the decompiler. So you can, in Ida Pro, you just hit F5, copy the C code, add a bit, fix the memory offsets and so on. And you have the whole key obfuscation method, basically reverse engineered almost automatically. And so I compile it and sure enough, moxa key extraction, it turns out that the key is 2887 con 7564. I build a short script to decrypt the 2.1 firmware and this time binwalk finds all the files and we can start reverse engineering the actual firmware. These scripts for this are available on my GitHub. I'll push the actual decrypt stuff after the talk because this is the first time I think this has been released. And so after I was at this point, I knew that the firmware is, I can decrypt the firmware, I can look into it. By the way, it's not signed or anything. The only verification method is CRC32. And so at this point I knew, okay, I can buy this device and start playing with it. And so I went to eBay. I bought one. I got it. I screwed it open and sure enough, there's an ARM processor in there. It's an free scale, which is just a regular ARM processor. It's like 400 megahertz or something. I don't know. And I started probing all the small pins inside of the device to try to find JTAG or serial or anything. And so I actually hooked up my power supply to foot pedals so that I can probe and just press with my foot to reset the device. And sure enough, I found that there's a full serial console available inside of the device on these pins. And if you boot the device, it even tells you please press enter to activate this console. And so you do that and you are root on the device. So that's kind of cool, but that means that you won't require physical access. So it's not really a vulnerability, but it's very nice to have when doing security research because it means you can suddenly debug all the code on there. And so if you write an exploit, you can just attach GDB to the binary and start very, very simply writing the exploit. So at this point, it was time to look at the available services on the device. So for example, there's a web interface. There's a proprietary configuration protocol. There's Telnet. There's SNMP. There's a serial driver protocol and so on. I started looking at the web interface and there was cross-site scripting. There was cross-site request forgery. There was insecure authentication where they basically hash on the client. So they have some JavaScript that hashes your password and then locks you in. Then there's a command injection which lets you execute code as root. There are stack overflows. And just a week ago, there was a zero-day released for the web server. So yeah, demo time. So just let me open up the Moxa page right here. And so this one is authenticated. So I'll just enter the default password, which by the way, in the field, 90% of the time, these devices will be configured with default credentials. But still, so if we just start browsing through this thing and go to the basic settings, we can start with a simple cross-site scripting just in the device name. So for example, we just paste in some JavaScript, submit the whole thing, and hello 34 C3. I know what you're thinking, cross-site scripting, come on, that's not a vulnerability, that's just nothing. So let's look at the ping test that's integrated into this device. And finally, a different device from Moxa that runs an entirely different firmware had the same vulnerability in the past. But if I just paste in my ping, so my IP address, a semi-colon, and then for example, I cut ETC past WT and activate N, sure enough, here we go. Kind of handy, but yeah, for sure not intended. Alrighty, but I know what you're thinking, right? Like these are authenticated bugs in the web interface. So we need something unauthenticated, we want something that's like cool and real exploit, right? And so I decided to look at the this custom TCP protocol which runs on port 4900. And my goal was to reverse engineer the whole protocol and build a fuzzer for it to find vulnerabilities. That turned out not to be necessary. So during some testing, I just sent a lot of bytes onto this thing and enabled crash debugging via the serial console. And sure enough, it crashed and put my program counter right to 4141414141. Wonderful. Thank you, Moxa. So demo time. So let's increase the size of this a bit. So I built a small script which is called Moxa Pong and I'll just supply the IP address to it. Let's see. Opening a second shell to connect to the VNETCAD. Here we go. We have a root shell on the device. So yeah, that was the Moxa W2150A, basically rolls of the tongue. And so the next device I decided to look at was the advantage EKI 1522, which you can find right here. And it's again just a simple serial device server this time without Wi-Fi, even though they are available with Wi-Fi. It comes with two Ethernet ports, two serial ports and so on. I basically followed the same steps again. So I looked at the, I downloaded the firmware, I looked at the edit using binwalk and this time we see almost no entropy. So this guy is basically completely unencrypted. And again we saw it's ARM32 bit. It runs a Linux kernel 2.631 and a Boa web server where the last update was in 2005. And the firmware I think is from 2017. So these are kind of outdated. And I found during the initial analysis just of the firmware that the main binary to look at will be this edge server binary. And so I loaded it into Ida Pro and looked at the different things it calls. And there are a lot of calls to functions like string copy, to system, to sprint, and so on. They're generally kind of considered unsecure. And sure enough, during static analysis I found that there's some code for sending an email as an alert, for example, when the system reboots. And the full command invocation is mail X dash S blah, blah, blah, blah. And we have control over some parts in the string because we can configure the two address in the UI. And if we look at what's happening here, it basically just sets up this format string, then it goes to include the subject, and then it gets some arguments from the stack. And basically calls into system. And so there's no filtering going on at all. So we have a non-filtered part of the system invocation code execution. And this was before I had the device in my hand. And this is kind of a funny story because I first bought because it was just 40 bucks, I bought this device, which in the firmware has the same buck, but the mail functionality is broken, so I couldn't test it. So I had to go to eBay again by another one, and yeah, by the bigger one. And so I ordered the bigger one on eBay. It looks like this. It comes with a Cavium CNS CPU. It has JTAG exposed on the bottom there. And serial console is available again without any authentication. So beautiful. You just connect your bus pirate or your UART adapter to it. And you have full serial console. So again, we had to look at finding vulnerabilities for this device. And there are again a ton of different services. There's like a web interface available. There's a proprietary configuration protocol that's based on UDP. There is Telnet. There's SNMP. There's a serial driver protocol and so on. And again, looked at the website and again, cross-site scripting, cross-site request forgery, command injection, broken authentication, which basically if you look in from one computer, it uses I think HTTP digest authentication. You can connect from a completely different computer and it doesn't ask for a password. I don't know why that is, but yeah. I was thinking I was doing something wrong, but it turned out it was just broken. And there's again a stack overflow in another protocol. So, I guess again, demo time. Let's first look at the device itself. So we have a nice device description here. This is just a basic web interface. And we can again just copy in some basic JavaScript. Hit the save button. Reload. And there we go. Cross-site scripting yet again. Okay. Again, not really interesting, right? So let's look at the stack overflow. Again, I have a small script advantage. Pawn. Forgot the IP there. And we have Netcat running on there. Sure enough. There we go. That's root on the advantage device. Again, the stack overflow. Yeah. So two of three devices basically broken already. Let's look at the next one. This one is the Lantronics EDS 2100. And this one is kind of interesting because it's not ARM. And I normally, I almost exclusively do ARM. So this one was kind of interesting. And this device, which is mounted somewhere right here. Yeah. This device comes with a serial, to ethernet, secure device server. It has two serial ports. It has ethernet. And you can buy it in two variants. One comes with Linux. And one is Evolution OS, which is, I guess, a proprietary operating system from Lantronics. And I'm using the Evolution OS variant in this talk. Looking at the firmware, it turns out it's unencrypted. And it's cold fire architecture, which I've never done really anything with before. And there are no obvious external software components. So if you go through the firmware, you'll find there's an SSH implementation. There's an SSL implementation. But it's not open SSL. And it's not anything very well known. And the same is true for the web server. And so there's not really anything that's well known. And this time, while probing the device, I did not really find anything interesting in terms of serial console or so. But I just found a potential debugger port. But I didn't have a fitting debugger, unfortunately. The CPU is from NXP. It runs at 160 megahertz or something. Yeah. This time, we actually have a web interface with Telnet, SSL, and it even has a file system. So you have like FTP and TFTP, which allows you to download the configuration, upload the configuration, and so on. And it's kind of hard to secure it correctly, because there are so many protocols. And it's not really clear what's set up by default. But yeah, you get the idea. And this time, the web interface was surprisingly secure. So there was no cross-site scripting. There was no command injection, because there's also not really a shell that you could execute commands into. But I still found some stuff. One is the configuration injection, which allows you basically to change the format of the configuration using a different field. And I found an authentication bypass. So I was able to write a small piece of code that takes a while and then completely removes the password from the device. Demo time. So if we connect to the Lantronics device, it will currently ask for a password, which, in theory, we don't have. Let's clean up here a bit. I don't know. I think it's just... And let's run Lantronics on. Oh, that was fast. I hope that worked. Yeah, sure enough, the password is gone. Awesome. To be honest, I didn't expect the demos to go so smoothly. So I put in an hour for the talk, but this went very well so far. So that's cool. So before we finish already, some other devices are even worse. So, for example, as I mentioned, I bought some other devices. For example, this advantage device, this moxa device, and this Lantronics device, which are basically the predecessors of the other devices. And those guys are really interesting to look at, one could say. So some of those are running ECOS, which is an embedded Linux platform, which was last released in 2009. And some devices run a Linux kernel with the 2.4 version, and you see Linux without any memory protection whatsoever. So even a small stack overflow in one of the user space applications gives you full root access to the device, because you can directly exploit the kernel. And there are unfixed public vulnerabilities. So in the first penetration test that I did, that included actually this device, a moxa and port, a small one, I found that using SNMP walk, it gives you back the administration password via SNMP. And so I went online, I tried to report it, and it turns out it's well known, there's a Metasploit module for this, and it's unfixed. And these devices are still in support, so I don't know why the vendor is not patching this. So in the summary, we have trivial vulnerabilities in most devices, or at least all that I've looked at. There are no security mitigations whatsoever, so they don't even enable the compiler flex that you just set, and then you have at least some kind of stack protection and stack cookies and whatnot. And some vendors are really bad at responding to vulnerability reports, so I'm not going to name the vendor, but on Twitter I asked them to please give me a security contact, and they responded, please use our contact form, and I said, I did three times, I send you emails, you're not responding to me, and so they stopped responding to me on Twitter too. So how to mitigate? Well, the only way that I would see to mitigate against this, and I'm more on the deconstructive side of the story, is defense and def. So never directly expose any of these devices to the internet. Even if they say they support VPN, even if they say they are a secure device server, whatever, just don't do it. Get a real VPN gateway, and make sure that you never rely on a single level of, for example, encryption. So for example, WPA2 was broken by the crack attack, and they actually released a patch for it after two months, and these are still two months where you are exposed to vulnerability on your potentially mission critical system. Also never use GPRS for these devices without VPN, because it just, it will go wrong. Aki, yeah, thank you. I guess now we have time for Q&A. Thank you all for coming. Thank you very much for the talk. So we have very much time for Q&A, so please line up to the microphones, and we have someone at microphone for already. Yes, hello. Thanks for your talk. This is, obviously this is a problem, this is a part of a bigger problem of security in IT, right, in anything related to any kind of technology, and this is only going to go worse with time, right, internet of shit, I mean things, and so on and so forth. So my question is, you gave some ideas how to mitigate this in this very specific area, like use VPNs, et cetera, et cetera, but my question is, so hacker community is not very, let's say, interested in regulation, right? When we see a government trying to do something with technology, that usually goes bad, we have this idea in our head that okay, this can only go, like this can only go bad, right? So my question is, do you think that perhaps there is some space for regulation here? There's definitely space for regulation, but I think regulation does not solve the underlying technical issues. So these devices, it's 2017, and these devices are using C code. That's just asking for trouble basically, and so we really need to see the shift even in the embedded world to switch to memory safe languages, for example Rust or something similar, and really to stop using C in this kind of context. I don't think there's anyone who can... Thank you. But there's definitely space for regulation. Thanks, there was a question from the internet. Hello, okay. Yeah, the internet wants to know why you are not naming the bad vendor, because it looks like it's the only option basically if they don't respond to you. Let's say I ask them on Twitter, and my Twitter is right there, and if you click on, Twitter replies. Yeah, somebody just posted the link on IRC. I did not name them, just for the record. So we have a question from microphone number two. So you've shown an exploit for the last device that disabled authentication. What did you use to achieve that? So this one is unpatched and not yet fixed, so I would rather not disclose the details yet. Microphone number one, please. I wonder if you've also been looking at building automation systems, control systems, or just industrial automation control systems. So you can use these devices basically wherever you want, and I think some of the moxa ones are used in home automation, but I've looked at, I guess, Crestron, it's called, but not in a lot of detail, so I'm more on the industrial side at the moment. Thanks. Microphone number three. Hi. Any field experience or even just opinions on using industrial strength Raspberry Pi hardware with community supported Linux distributions or something like OpenBC, whatever on them? Yeah, so I guess the big trouble there is support, right? There are some German companies and so on that provide support for industrial Raspberry Pis and even like nice casing and so on, but I'm not sure if really Raspberry Pi is the way to go here. I think there are boards that are, the problem is not the underlying stack, right? It's not the hardware really that's the issue, it's the software, and you will have the same issues on the Raspberry Pi. So yeah, I guess you could buy these devices which are like industrial grade shockproof and whatnot and put some Linux on it and do it better, but I don't think that the hardware or platform will change anything at the moment. So there's another question from microphone number four. Hi. More social question. Did you get in contact to any development team, software development team in any of these companies or might it be that there is no one behind the emails and everything? So I guess some of these companies are really so big that they don't reply to you if you don't have a support contract with them. But for example, the support of the ones that are not on my Twitter is kind of decent when it comes to security reports. And so my next steps will be to go via the ICS cert to report them. So yes, there are development teams that will get in contact with you just not from all vendors. Thank you. We have another question from the internet. Hello. Okay. The internet wants to know what to do about because there are a lot of old devices in the field. How do you propose a vendor should deal with legacy devices and updates? Yeah. So keeping legacy devices supported is very expensive because for example, if you buy a Qualcomm chip, they will eventually drop support for the Linux kind of weight and so on. But if you buy like a free scale automotive chip, they guarantee you a certain time of support. But then you actually have to invest the money to regularly provide the updates and ensure that your devices are secure. The problem is that the lifetime of industrial installations currently is much larger than the lifetime of these processor supports and so on. So I guess we'll have to get used to upgrading our hardware regularly or switch to or figure out a different way of deploying secure software onto them. But I really think the underlying problem is that we are still using memory unsaved languages. And I guess the fact that there's cross-site scripting just shows that there's no security awareness really at those vendors whatsoever. That's some of the vendors. So microphone number two, please. I was wondering, you mentioned that some of these facilities use GPRS. Do you know if they have mostly their own closed infrastructure or if they're using general consumer telecom stuff? So they will use commercial networks mostly, and then they have custom APNs which have an IP sector or something similar to their premises. But there's also a company that sells industrial control SIM cards which give you a public IP and you don't want to search on shoulder for that vendor. Yeah, thank you. So there's the question from microphone number three. Hi there. Isn't economics meant to solve some of these problems? We're not talking about dirt cheap devices. Surely at 300 bucks you should better hire someone who's read Security 101. How long before large organization gets the result of their security audit and goes to the aforementioned vendors and says provide us something that's not trivially hackable, otherwise we stop buying your rubbish? Well, I mean it's the same in all of IT, right? So everything has vulnerabilities and yes there should be market pressure but that's why I'm trying to raise awareness for the issues that these devices have. Thanks. So there's another question from the internet. Yep, the internet wants to know how and if it's a good idea to raise awareness in public because they think it's a good approach to make people, the public know that well infrastructure in the city is at risk. Sorry, could you repeat the first part of the question? Yeah, they want to know how to raise awareness for this in the public. Good question. I guess we need some news articles or something about this in regular paper but I personally think it's just an accident waiting to happen. So eventually someone will turn off the lights in a city or wherever will open a flat valve or something and that's when the awareness will start. There's another question from microphone number four. Okay. For what kind of industrial processes are these devices you just demoed used? So I've seen them in power utility. I know they're used in water, water dam control systems they're used in connecting a CNC machine to the network they're used in connecting all kinds of stuff because if you have a big plant you have a ton of different sensors so you might need a water level sensor and for whatever reason you only can get it with a mod bus and then you need to convert the mod bus to a TCP and then you need one of these gateways and so I've seen in one cabinet 20 of them so they really used a lot I guess. Okay, thank you. I just retweeted your tweet to Star Alliance. Thank you. So there's another question from the internet. Yeah, the internet wants to know how if you did any research on MQTT for example from like back of users? I actually talked to someone who recommended me to look at back of yesterday but I've not looked at them whatsoever yet. And there's another question from microphone three. Okay, could you show the Moxa web panel because I would like to double check which browser they would like you to see their web page and I think this browser isn't very secure. Okay, let's take a look. Yeah and undergo a head web server there is small print. Nice finding. That's probably the issue. Any more questions? Any questions from the internet? Hello, okay. The internet wants to know how a memory safe language would prevent the authentication bypass you showed? That one would not be protected against but protects against a ton of other stuff. It's just one example of where the industry needs to change. We need to stop using memory unsafe languages. We need to start really thinking about security design from the start and we must not in 2017 there's no excuse for having cross-site scripting or anything on the web page. That's also if we in the Lantronics website if you click logout it tells you logout is not supported in your browser. Probably because I'm not using Internet Explorer 5. So there's another question from microphone number three. Any remote part of the exploit where you did a buffer overflow I think? Yeah. What I'm wondering is aren't it like very standard to have ASLR on these devices? No, it should be but it isn't. Okay, thank you. That was pretty much my question. Okay, is there another question from the internet? No, it doesn't seem like it. One just came in. Okay. If you want to hurt. Okay, no. So all right, give a very warm applause to Thomas Road again.