 I just want to thank everybody for coming out. We're here today to talk about some of the, it helps if I get the right slides. We're here today to talk about IoT botnets and lack of basic security and how they contributed to the IoT botnets. Okay. So why are we here? What are we looking to do today? Am I missing something here? Okay. So we're going to just briefly dive into a couple of case studies of some recent botnets. I think probably most of the folks here are familiar with the three that are listed here. Then we'll dive into some of the common security problems that are common amongst all three of them. They actually were very similar. The issues that allowed them to propagate and continue to have as much success as they did. They're very similar amongst all three. So we'll dig into some of the common security problems, and then talk about some of the solution designs. For those that are in this audience, I'm sure most of the solution designs are pretty obvious. They're pretty straightforward problems. The scary part is that things will be getting worse over time, not better, and we obviously all have to be diligent moving forward. Now, some of the motivation for why we're here, obviously, we always want to review past issues to make sure that we don't repeat the same mistakes moving forward. Especially as things move into the connected devices market, it seems some of the embedded folks are repeating a lot of the same mistakes that have been made in the enterprise and the server world for a long time. So those of us working in the embedded space are really trying to get the word out and avoid these issues. And of course, just the sheer number of devices available in the connected device world makes the scope of these kind of issues much worse. So we really need to keep an eye on things. And as far as the guys that are here are concerned, we want you to start thinking about designs in your products. Actually, show of hands, who in here actually works in the embedded or connected device space? OK, more than I thought. So about a third or so, so that's pretty good. And obviously, in the Linux world, we're definitely getting moving into the connected device space a lot more in the last five years or so. So it's definitely picking up. All right, so just a little bit about me. My name's Drew Mosley. All my contact info is on there. I'm part of an open source project called Mender.io, which is, it deploys over-the-air updates to connected Linux devices. So that's kind of the bias that I bring to this whole presentation. These are all things that we've come up with talking with our potential users and those that are going to be using a product like we have to offer. So if there's anything in here that seems biased, this will at least give you an idea of why I see the world the way I do. So this is a real-time update that just came in, came across my feeds this morning. I know at least one person has asked me about it. This is a new botnet that's out there. It's called Reaper. Don't have a whole lot of details on it yet. It's allegedly relying on more sophisticated techniques and relying on actual software vulnerabilities in the installed stacks on some of these connected devices, as opposed to the ones that we're going to dig into here in a bit, which aren't quite as sophisticated. So far, they're reporting up to 1 million connected devices. I think the early, I think I saw an early date of sometime in September was when they started looking at this. So this thing's spreading pretty rapidly. Fortunately, at this point, it's kind of dormant. It's not actively attacking anything or doing anything. So as far as we know, somebody's collecting a botnet out there. But for what purpose? I don't think anybody has any idea yet. So there's lots of stuff on the web about that. If you want to chat about that one later, I'd be happy to. I don't know much about it since, like I say, it just came up this morning. So the way we look at things, just to discuss these attacks, the first step is always reconnaissance, trying to figure out what devices are out there and what services are available and that kind of thing, with the idea of being to discover vulnerabilities. From then, how do you get into the system? What's the hook? How do you get into the system? Then how do you get back in when you get discovered? And finally, how do you cover yourself up? How do you get out? How do you clean up? That kind of thing. So those are the basic roadmap we're going to use for looking at the existing botnets. So Mariah, I think, was the first of the botnets that came out. It was back in August of 2016. The name, I'm not even sure who gave it the name, but it does mean future in Japanese, which has some pretty bad connotations, if the expectations that this thing's out there. Obviously, it was used for, I think this is the most obvious one, was the DynDNS distributed denial of service attack, which took down a lot of the services. You see the logos on the right half of the screen there. The DynDNS provider is a very large provider that many of the large service providers on the internet use. And the 24 or 48 hours they were down was pretty painful for a lot of people. And that was all thanks to connected devices issuing just enormous amounts of traffic. Additionally, Brian Krebs, a well-known security researcher, he, his blog, was taken offline with over 600 gigabytes per second of traffic going to his blog. So it's pretty hard for anybody to be able to sync that much traffic and still keep things running. So the scary thing of Morai and the other devices like this are that it can be extended for other uses. Distributed denial of service attacks are, they're bad enough as it is, but these botnets are still out there communicating with a command and control servers and could potentially be used for much more nefarious purposes in the future. So there's a good reason to learn about these things and figure out how to mitigate them moving forward. And surprisingly, the source code is actually available. So if you wanted to go download it and try it on your home system, I wouldn't recommend it. But you certainly could do that. I would recommend spinning up a VM or three if you wanted to do that just to cover yourself. So how does this system work? So initially, it connects just to a straight up telnet port. It's looking for port 23, or I guess some of the devices use port 2323. So the initial connection is just a TCP send scan of those two ports looking for potentially vulnerable devices. A later iteration also used what's called CWMP. It's a device management protocol that's used in the telecom space. I don't myself know anything about it, but that was one of the later additions to this botnet. And basically, once they find an open telnet port, there's a set of 62 default user names and passwords. And they attempt to log in. And these are things like admin, admin, those things that you can look up when you get a new device and you get home and you say, how the heck do I access this so you Google it. And it's amazing how many devices on the internet facing devices to the wider internet are still sitting there with an unencrypted telnet connection with default user names and passwords. And so once they identify a vulnerable device that they're able to log into, that device information comes back to the report server. And it's basically just a database of all these vulnerable devices that are out there on the network. And at that point, that's effectively the botnet. So that's just the initial scan, connect, and enumerate the devices that are available. And then we move into what happens next. So once the devices are identified, then the actual Morai binaries get installed onto the device. And then once they're installed and running, they make some attempts to obfuscate themselves. They rename themselves as telnet D. They randomize a process name. They delete their executable. Which the nice thing about that is that means Morai does not survive a reboot. But ultimately, that doesn't matter in these cases. Because the simple fact is, the device reboots, it's still on the internet. How long do you think it takes before it's going to get infected again? We've all heard the horror stories of people putting unprotected Windows 2000 systems on the network. And what's the average time? 30 seconds or something till it gets attacked by something. So it's nice that these things don't survive reboots, but that's only a minor mitigation. In addition to obfuscating themselves, then the Morai binary attempts to remove other services that could potentially allow you to get in and clean up. So if it's got SSH or telnet, it tries to remove your access to that. And it even does some clean up of other malware, presumably so that there's less chance of adverse interactions with other things on the device that it's unaware of. And then finally, it just sits there kind of quietly waiting for the command and control server. And while it has nothing else to do, it's out scanning for more victim devices that might be close by that it could pass the infection off to. So the amount of code needed to actually, for these devices to scan is pretty low, because all they're doing is getting the IP address and information and passing it back to the server. So there's plenty of bandwidth for these devices while they're waiting for the command and control server to come in to be able to spread that network even further. So in summary on this, there's at least 30 vendors that are known about of these devices that are out there. They're fairly commonly used devices. They're all embedded Linux in this case. And obviously, the biggest issue is default credentials that anybody can find. And it's pretty astounding the number of infections that can happen just because it's as simple a problem as it was. This slide says 600,000 plus. I saw a number this morning said, I think, over 2 and 1 half million devices are now on the wider internet and infected with this. And cleaning it up is tough, because how many people are going to go into that closet and find that webcam that's been sitting there for five years and get it updated? So the next one we want to talk about, Hajime is very similar in scope or in the way it works. It was discovered right around the same time. And it was actually named by the researchers. And interestingly, they published a report and mentioned some bugs that were part that they discovered in their analysis. And in the next release of the Hajime worm, those bugs were fixed. So the unknown author is actually keeping an eye on things and improving it. Now, the name was adopted by the authors. It wasn't actually from the authors. So whoever is in control of this definitely knows what's going on. The first thing that happens when you get this infection is on your serial console or whatever kind of text console, you get a message from the Hajime worm that seems to indicate this is a white hat hacker trying to do good stuff. But again, it uses the same techniques as Marai. So ultimately, it could be hijacked for bad things. So far, it hasn't been used for any attacks. It's been used really just to go in and clean up the devices if it finds telnet ports that are open, presumably it disables those and that kind of thing. So the idea being we assume it's a white worm with good intentions, but nobody really knows. So the fact that this has to be done to target Marai, that's a problem. So device manufacturers really need to step up. So similar to Marai, we used just a standard IP SIN check to port 23. We've got 64 usernames in passwords. So this one actually has two more than the Marai. So this one's actually more functional than Marai. So it's a good thing. The author doesn't seem to be wanting to do bad things. And then there's a file transfer binary. In this case, it's only 484 bytes written in raw assembly that's written as part of the connection to the device initially. And then that binary goes out and downloads the actual Hujime binary at that point. And at that point, the device is infected. And it just sits there and waits for commands. Now, it's a little bit more sophisticated in the way that devices talk to each other. In the case of Hujime, they use a distributed hash tree with an overlay network on the wider internet. So all the devices actually know about each other and can communicate directly if they need to. Then installs the Hujime scanner and continues propagation onto any other devices that may be nearby. And again, it does the similar kinds of things. It renames itself to Telnet D to make it a little harder to figure out if it's actually there or not. And again, it doesn't survive reboots, but it's the same issue that we had with the Maribot net. You reboot and you get reinfected pretty quickly. And in this case, the reason that we expect that it's intense or positive is that it's actually closing the ports you see here, which are fairly commonly used for access for nefarious purposes. So far, this one appears to be doing good things for the wider internet, but it's unclear what the actual motivations are. Yes, probably just because of the time the research was actually done. I suspect if somebody were to do an analysis and comparison now of the number of vulnerable machines of both of these, I can't imagine they would be all that different because they're kind of fighting it out on the internet. So that's the other interesting thing, right? If a GMA gets in there first, Marai theoretically cannot connect. So it definitely would be interesting to see how they compete over time, but I don't think anybody's actually done that. So in summary, you see the details here. Arm 5, Arm 7, Intel X8664 and MIPS. So that covers a very wide range of the devices that are out there. And again, we're exploiting the default credentials, which is starting to be a common theme. Basically, it targets the same devices as Marai. So as I just mentioned, this does appear specifically designed to target Marai, which one's winning? I don't think I've not seen any indication that would tell us whether one or the other is winning the war. And the last one we're going to talk about is called Brickerbot. This one's a bit more nefarious, but also it appears to be an attempt to mitigate the negative impacts of the botnets. So the author claimed 2 million total infections based on some of the numbers I've seen for some of the other botnets. That doesn't seem unrealistic. But once it connects to the device, and I don't know if you can read the commands in this window here, but for those that know what those commands do, those are things you don't want to do to your Linux systems. The idea is it's trying to wipe the storage and effectively make the device completely unusable. People have coined the term permanent denial of service attack on these. And the intent, it seems to be, once these devices get infected with the Brickerbot, the intent is to make the devices worthless. Erase the flash, turn off all network capability, and literally turn it into a brick. I guess not literally, but you get the idea. And so again, we're going back to TCP port 23, brute forcing the telnet login attempts. And then as I mentioned, we attempt to completely brick the device. Of course, the interesting thing with Brickerbot, since it's destroying the device, it doesn't then continue the infection forward. So once the device is taken offline and no longer useful for anything, it's out of the network. So there is a static set of attacking devices. And it turns out they all appear to be, well, I've got the details on a later slide, but the attacking devices appear to all be embedded Linux devices running some variant of telnet and the DropBare SSH protocol. So they're fairly low powered devices that are doing this. Presumably they're just the first set of devices that were contacted and taken over by this bot. So and you see the details here from the person who claims to have written this tool obviously is not particularly thrilled with the industry's attempts to resolve this and figured that this was the nuclear option. So to speak, to address some of these challenges that don't appear to be taking being fixed by industry itself. So as I mentioned, DropBare with telnet, you see the map on the screen there. That's where the known devices are. So effectively with some tens of devices, this botnet has effectively captured, at least reportedly, up to 2 million devices. So with a very small number of devices in the fleet, he was able to capture quite a bit. And again, this is targeting the same devices as the previous two. So at this point, all three of these botnets are competing for the same space. And it's going to be interesting over the next, I don't know, three to six months to see how things play out. And as I mentioned, the industry doesn't appear to be fixing it. We've all had a cheap consumer electronic device that the firmware update mechanism was such a pain in the neck that it just never got done. And that's for those of us that are spending all our time doing technology. You can imagine how difficult it is for those where technology is supposed to just mesh into the background. So you can't rely on the end users. The buyers do ultimately need to demand better security, but most of your average consumers aren't aware enough to actually demand that. So there is at least one bill in Congress in the United States that's attempting to require that at least for US government purchases, there's more security involved. I don't know all the details, but obviously getting a heavy weight like the US government to start pushing for this, maybe we stand a chance of getting some improvement overall down the road. And of course, the alternative is more vigilante botnets like the bricker bot, where people take it on their own to get in there and try to figure things out. And just kind of a little bit broader view of things. Ultimately, your goal is to make it hard for somebody to attack your device or make the value of a successful attack low. If I have, say, a fast pass for a toll system, the value of that attack is pretty low. Yeah, you can charge me a couple euros at a time off my credit card, but you're not going to be issuing a distributed denial of service attack. So the value of that attack is fairly low, whereas with connected devices, obviously with the devices we've been talking about here, the value is potentially high depending on what your goals are. If your goal is denial of service, it's clear what the value is. And with those devices out there, it could just get worse. Generic solutions for these kind of things are just basic good security practices. I mean, default username and password is a bad idea. We don't want that. Close unused ports. Make sure your firewalls are sound. Remove any software you don't need. The most secure software is that, which isn't even installed. Don't provide unnecessary features. And obviously disable the default credentials. I'll say that again, because that has been the result of most of the issues that we're seeing here. So we'll come back to this slide here and just briefly talk about some of the mitigation effects. So in reconnaissance, obviously if you can slow the reconnaissance down, you slow the spread of these things. IntelNet is generally fairly easy to get to. It's pretty simple. Close all the ports. Don't leave anything open that you don't need. Proper network segmentation. If your devices are behind firewalls, they're not available to be hacked from the outside. So just keep those things in mind. To minimize the intrusion potential, obviously providing random passwords and not default credentials for everybody. And then some kind of security update mechanism. Obviously we've all seen CVEs and lots of critical vulnerability that needed to be fixed. And then in terms of what they can do once they get in, inserting backdoors and that kind of thing, don't provide privileges where they don't need to be. Again, all standard security things. I don't think there's anything new here. The sad thing is that over time it's going to get harder. Our jobs will get harder. But we shouldn't make it too easy on the attackers now. And one kind of self-serving thing, obviously the ability to update things over the air is critical in addressing these vulnerabilities moving forward. So if you don't have an update mechanism in your designs, you should get it. Just to show a hands, for those that do the connected embedded devices, how many of you have an update mechanism built in? And is it OK? And those that do, is it a home-built? Or is it a third-party project? What's that? Who has a home-built? OK, so that's the majority. And who uses a third-party project? So one in the entire room. So there's a lot, and I've got to talk later on some updated technology. So I'd definitely be interested for those that do a home-built. I'd definitely be interested in discussing that a bit further. So with that, I do have some backup slides. But I wanted to give a little bit of time for questions and five minutes for questions, 10 minutes for questions. So we got more time than I expected. So we've got a microphone coming around if anybody's got any questions. All right, in the back. So you recommend having some sort of OTA update mechanism. How do you protect that against attack? Well, it has to be well-designed. You know, I mean, again, it comes back to standard security practices. You've got to use TLS connections. Ideally, your images will be cryptographically signed to ensure that you're installing the right things. I don't think there's any real unknown problems that can't be solved by standard mechanisms. It's just a matter of being diligent and using the known good practices. How does your company handle it? Exactly that way. We use TLS for all the communications. The certificates are baked into the images at build time so that you know that you're talking to the right server. And then in addition to that, we have cryptographic signatures on all the images so that once the client downloads the image, there's an additional verification step just to make sure there wasn't some kind of man in the middle of attack or something. Do you handle certificate rotation as well? It's something we're working on. I don't know that we have a, I don't know all the details on it, but it's definitely come up. And if you want to ping me afterwards, I can get in touch with you and get you a more definitive answer. OK, thanks. All right, anybody else? The botnets you showed used very trivial attack methods, which is default using in passwords. Is there a reason that all the examples you showed have those? Or have you just not seen many more sophisticated attack methods? Well, those three are practically all using the exact same attack method. And so they're kind of fighting each other. The other one that I mentioned, the Reaper, is relying more on vulnerabilities in the installed software. And that's as much as I got from the article that came out this morning. So I fully expect that over time these attacks will become more sophisticated. But this is low-hanging fruit. So it's not surprising that somebody took advantage of it this quickly. Have you seen any two-step attacks? I mean attacks that take over a host computer and then use that computer to attack the IoT devices that are connected to the local network that usually wouldn't be available from the outside because of a router or a firewall. I can't say that I have, but it certainly sounds like something that could happen in the future. And obviously that would be a means to kind of bypass the whole network segmentation firewalling, because a lot of the best practice recommendations are don't put any of your IoT devices on your home network, right? Put them all on a separate network. So there are obviously ways to work around that. Basically, assume any IoT device you get is untrustworthy. I haven't seen anything like you suggest in any large numbers, but it's certainly something that we need to be on the lookout for. All right, anybody else? Very good. Thank you so much.