 I'm Kev and I'm an Embedded Penetration Tester, so I work for a large company, I can't name them because I'm not here officially in the capacity of speaking on their behalf or anything, but they, amongst other things they do, they make embedded devices that they sell, and I kind of work for the team that tests the security of those devices before they're sent out to market. The idea being that we get about a month to have a look at these devices and see if there's holes in them. Try and work out what attackers who buy these products might do to them before we sell them and try and fix those holes so that they have less opportunity. We are very much hunting for the low hanging fruit because we don't have time to do a complete job of finding every bug. I mean, you know, it's a bit of a guess as to whether or not it's ever possible to find every bug, but it's certainly the time scales we have, it's impossible. So, we look for the things that we think people will find and we get them fixed. Now, whether your interest is similar and you're looking to find bugs or whether you just want to take one of your own devices and be able to run your own code on it, make it do something it wasn't supposed to do, I'm hoping there's going to be something in this talk for you and I'm going to go for it quite quickly. If you don't understand some of the terms, then I'm sorry, just come and find me in the portcullis tent later and I'll explain. I don't work for them by the way, but find me in the portcullis tent who are hosting me later if you want to talk and ask some questions. So, embedded devices, here's some examples. You've got an internet router, there's an IP camera, there's a mobile phone, and there is a child tracker. They're all embedded devices. Three of those things run variations on Unix. Obviously, the phone is BSD, but the other two things probably run Linux. The child tracker itself is an embedded device that probably doesn't run Linux. It runs off a microprocessor with its own stack, which is incredibly difficult to change because there's a lot less structure to them. We're going to focus on the ones that deal with Linux because that is 99% of the embedded devices that I see. So, they run Linux. The trouble with embedded devices is they lack processing power. Because they lack processing power, they can't have all of the modern protections that you find on modern Linux servers or desktops. At the same time, though, because they're an embedded device, they don't have a keyboard or screen. They have minimal interfaces, minimal services running on the network for you to interact with. So, there's less to look for, or there's less places to look, I should say, but when you go looking, there's less protections in place to stop you. So, they are an interesting set of things to hack, and you can do quite a good job with these things. So, as I say, there is lots of low-hanging freight. That's what we're going to look for. The main thing that you find is there's no user or process separation on these devices. Out of all of the embedded devices I've looked at, one had a user that wasn't root and ran processors that weren't running as root, but everything else, they all run as root. Sometimes they've renamed the root account, but as long as it's UID 0, who cares, it's still root. So, if you can get one bug, you are root. You can do everything. Now, when people talk about embedded testing, they often think about the hardware because, you know, isn't that the cool bit? Well, maybe it is. I mean, this is my toolkit. There's only one specialist tool in that slide, and that's the Lego Brick Separator Tool. They're £1.99 from the Lego shop, either online or in person. And they're brilliant, and the reason they're brilliant is because they don't damage things as much. If you want to get a case apart, straight in with the screwdriver, make a gap in with the Lego tool, and then just get rid of the screwdriver. If you try and take a case apart with the screwdriver, you'll be knocking components off-boards, you'll be scratching things, you'll be smashing clips. With the Lego tool, and it's open. So, I do recommend them. If you're an embedded tester, you definitely need one. In reality, of course, embedded testing is 99% Linux host auditing and 1% hardware. Basically, we do a bit of hardware at the start of the job to get in, and then for the rest of it, we're just looking at the Linux and looking at the mistakes that are being made by the developers. If you want to be good at embedded testing, or if you want to find bugs in embedded devices, it helps if you've got good Linux skills, particularly Linux skills where you're running a busy box environment from a RAM disk, for example, and it would also help if you've been a sys admin for somebody else at some point in your life, because if you've ever had to set up and run services and keep them working for other people that shout at you, then you've probably got a good idea how those things work and how they could fail in lots of interesting career limiting kind of ways. So, if you've had that kind of career, it really does help. You've got a bit more of an eye on how these things work. The two things we're going to try and do at the start of every test is rip out the beating heart of the Linux system. I extract the firmware, and we're going to try and take remote control over it. In other words, get an interactive shell, or be able to execute commands and turn that into an interactive shell. Obviously, prompts the question which came first, the penguin or the egg shell. Obviously, both. It's the same in evolution, apparently. Well, you can argue with me later if that's the case. Basically, they've both come first. They're both as important as each other. If you can get an interactive shell, then you can almost certainly extract the firmware. Equally, if you can get hold of the firmware, you can almost certainly find a bug in it that will give you an interactive shell. So, we tend to start with one, probably the hardware, trying to get a shell, and when that gets tricky, we move to trying to get the firmware from somewhere, and when that gets tricky, we move back to trying to get some hardware attack. We flip between them. We basically take the path of least resistance, the path with the easiest bugs to find, the easiest way to get either an interactive or firmware downloaded. If we can, and then from there on whichever one we've achieved, we achieve the other one, and then we're ready to start actually looking for bugs. So, we'll start with firmware, because firmware is an easy thing to do. Where can you find the firmware? Obviously, the vendor that makes the thing has to host it somewhere, but equally, people who have taken these devices apart before often publish the firmware elsewhere. Sometimes it's leaked out of the factory, sometimes it's leaked out of the vendor, but they often end up on other sites. If you're looking for firmware, then obviously Google is clearly the answer, whether it's official or unofficial. If the device you've got does an automatic firmware update where you push a button or you use a phone app or something to tell the device to update itself, and it goes to the internet to go and find the firmware file, then you can probably man in the middle that connection and steal the firmware as it goes to get it, and obviously you need some knowledge and some skills to do that, but we'll talk about that in a little bit, but it's not beyond the window man, and that's not a bad attack to actually grab the firmware. Obviously, the firmware will be on the flash chips. The firmware will always be expanded as something that it can boot off and use, but often it will be laying around as the actual firmware file that you could have got from the server, partly because that's how they revert when you screw them up. So, if you wanted a factory reset, often they just reinstall the firmware. It's kind of makes sense. These flash chips in the picture here are T-socks. They're pretty standard kind of things that we see. They're lexel down two sides. They're dead easy. Surface mount, chips. If you want to remove those, flood solder, you need to pull the chip up. Well, flood solder with a scalpel underneath. Just keep pulling behind some pressure as you keep on heating that solder up, and the chip will just fall off the board. Can I eat it to clean it up and chuck it in a reader? Obviously, you're going to need a reader and a solder and iron, but you know, it's not too hard. If the chip's a BGA chip, that's a ballgate array where the connections are all on the bottom, they're a lot harder. You can do it, it is possible, and you can read it out, but trying to put the BGA chip back on and making the device work again is a bit tricky. So, my advice is work for a large company that makes these devices and demand more than one, and never give them back, because they will get used to expecting them to be broken by you in the course of your job, and they never demand it back, and when you get a good one, you get to keep it. Other chips you find for flash or serial e-prom, and the serial might be UR, I2C, SPI, 1Y2Y, et cetera. Lots of different types of serial buses that may well talk to it. You'll spot those because they've only got five or six legs, and they're dead easy. You can pretty much do those in-circuit. Just power down the device itself and then hook up onto those pin legs and wire those back into your flash reader or your serial reader, bus pirate, whatever. If you're extracting from dev, then you're going to need interactive control, which is kind of that penguin or egg problem again. So, I want to quickly talk about my network. I mentioned Man in the Middle in the update process earlier, so the kind of network I use, I have my internet for broadband routes that goes to obviously the internet, and then I put in a Linux box that acts like an internet gateway router. So, it provides DHCP, it provides DNS, all relayed through, obviously. But on that box, I can also run wire sharks, I can see the traffic, and I can run Man in the Middle proxies on there, so I can interact with the encrypted traffic that's going through and try and decrypt it as it's passing so I can see what kind of communications it's having with the internet. I also have an open wireless AP, the only reason it's open is because it's a lot easier to eavesdrop on. You should really encrypt these things when you're doing this testing because the wireless bleeds and other people join your network and ruin your day. But if you do encrypt that AP, then eavesdropping on it becomes a lot harder. Whether you need to eavesdrop on it is debatable, but hey, so I generally chuck in a couple of wireless cards, I want to connect to the network as a client and want to eavesdrop as needed, and then I drop in my iPhone, gel broken obviously, and my IoT device, they connect to that network and now I can monitor their communications and I can see what they're doing. The idea here is to obtain as much information as you can. But say we're going to try and do the Man in the Middle and we quickly need to talk about this, so typically firmware updates will be over TLS, Transport Layer Security. That's the HTTPS as opposed to HTTP, for example when you're browsing the web. Excuse me. You would not believe how warm it is up here. I'm literally dripping the sweat. Anyway, so TLS is usually broken. The reason it's broken, the reason TLS exists is because of Edward Snowden. Everyone realises that things need to be encrypted because large TLA intelligence agencies like to come in and snoop on your data. So everyone employed TLS, but unfortunately no one understands TLS in the defender world. If you've got three jobs and one of them is producing embedded devices and your employer doesn't pay you enough or give you enough time to do a good job, then you're only going to do a crap job at this. Expect it to be broken and it probably is. But we're going to have a quick look at how Man in the Middle works. We'll talk about it from, say we'll look at web browsing. Say you've got a monkey on the internet and wants to browse the web and there's some cloud-based server it wants to talk to. It wants to make a direct connection and have that encrypted obviously. So when it connects, the authority that has signed that identity to say that this identity is valid. Everyone trusts Chuck Norris, obviously. Now, what happens if an evil cat tries to get in the way and can do something with the networking so that the packets go via the cat rather than directly to the server? Can the cat do anything? Well, when the monkey makes the connection it actually goes to the cat instead of the server and the cat itself then makes the connection to the server. The cat presents its identity with it signed by the trusted authority and then the cat presents its own identity signed by its own authority to the monkey. Now, if the monkey is not checking the certificate, as is the case in a lot of tier-less implementations, then the monkey will accept this as being a valid connection and, of course, it isn't. So the monkey will communicate encrypted to the cat, the cat will decrypt it and see it all in clear and then the cat will encrypt it again on the onward connection to the servers. The monkey won't notice any difference if the cat is sitting there seeing all the traffic. If that's the firmware file, you've won. Great. But what if the monkey does check that certificate? Well, you know, the cat could always fake a copy of the original certificate. Obviously, it can't provide the exact same certificate because it won't have the key that set that certificate up. But what it could do is create one with the same settings but with a different key and then sign it with its own authority and pass it on to the monkey. Now, if the monkey just checks that the identity is correct, it's going to fall for this and then yet again, you're in the same situation. So, but what happens if the monkey bothers to check the authority as well? Well, so it sends to its faked and it sends a faked authority. Now, if the monkey only checks to see that the authority is the right authority and that the chain of trust works but doesn't actually check that the authority is a valid one and I've seen that before on embedded devices, why? Because they can't be bothered to put a list of certificate authorities on the device. So they just assume whatever CAU turn up with is going to be valid. But they do a quick check and make sure it's called my CA or Bob's CA or whatever. Well, again, the cat could fake all of that. So in these situations there's lots of ways that TLS can fail and I recommend you should always have a go at the man in the middle on the TLS because they've probably done it wrong. Now, where they've done it right because that does happen occasionally, shock horror, pick yourself up off the floor, if you've got interactive access to the device itself then you can build this situation with the certificate you clone the CA. I've got a tool called Jackal which will just pop those out from command line parameters. You can then insert your trusted authority onto the embedded device itself via your interactive access and then from the onwards the man in the middle will work. You can't just do it arbitrarily on somebody else's camera because you need interactive access to their camera but if you just want to see the traffic so you can analyse it then that's the way you can do this. Awesome. The next thing to do is take pictures. Smash the thing apart. Start with your hammer and your screwdriver and get it open. Take pictures. You need to take pictures of both sides of the boards in high-res. Better quality pictures than these because obviously this is me and my rubbish photo taking. I am no David Bailey as it were. On the chips there are codes. I'm sure everyone's aware. Google the codes to find out what they are. You'll find that the embedded device looks a bit like this. Obviously things will be laid out differently and the system on chip in the middle may actually assume the role of many of these things. The RAM might be part of the SOC in some cases the NV RAM the non-volatile RAM might be part of the SOC the network interfaces, USB interfaces etc all might be in the SOC but often they're not. If you just check the numbers and google them you'll be able to work out what chip type each one is and then that'll give you an idea of what you've got. The NV RAM you can grab it. If you're looking for the SOC because you want to know some of it you can just grab the SOC data sheet and see whether you can understand how it works. On top of that there'll be these configurations of what look like solder tabs. There'll be gold roughly this colour but they look like solder tabs on the board so sometimes they're labelled up beautifully like this so the one nearest the SOC is a UART serial device like standard serial the sort of serial you see in your computer and of course you don't anymore but the one you saw 20 years ago on your computer but not running it 20 volts or whatever it's running it 5 volts or 3.3 volts maybe 1.8 volts but if you're looking at that configuration sometimes you get them labelled which is really nice so you can just hook that thing up to your serial USB serial device and now you've got potentially a serial connection sometimes they're not labelled and you just look for that pattern of square round round round or you look for the SOC data sheet and you trace the wires out yourself there's a number of things you could do we're going to talk about those in a second the other configuration you see on here is JTAG sometimes it's labelled sometimes it isn't, sometimes it's this configuration sometimes it's different I'm not going to talk about JTAG again in this talk because it is literally our port of last resort it's rubbish we find it proprietary, it's slow it's difficult to use and we will only go there whenever other attack has pretty much failed if you want to play with JTAG do so, often I hear a lot of people when I say I'm an embedded tester they go oh JTAG and I go yeah it's shit isn't it and they go really what because that's all they've heard about embedded surely that's how you attack these things well not really if you're developing things yeah JTAG might be awesome but if you're hacking them not so much if your boards just got these gold solder tub looking things all over it but they aren't laid out in any configuration or they're not labelled and you want to work out which ones they are well the first thing to know is they're not gold solder tabs nothing is soldering to these these are better nails contact points for testing in the factory so in the factory they have a thing with these pins that stick up and the board is pressed on top of it and they connect with a number of these points and they use it to test the board to make sure the board works well the serial port is almost always used in that testing situation and so some of these contact points will have little dimples in the middle where they've been touched by a bed of nails and some won't so if you've got 20 or 30 to look at and only 10 have got dimples in the middle start with those 10 because they're probably the ones that are more useful to you what do you do with them? well we chuck it through a logic analyser logic analysers like a digital oscilloscope they're relatively cheap usb devices these are just pictures I don't even think that logic analyser and that bit of software go together they're certainly not the ones I have they're just ripped off google but the idea being you hook up all the wires coming off the logic analyser to the different points on the board stick the sampling on the software reboot the device a few times and you'll get some traces by looking at the traces and looking at the widths of the pulses you can work out the speed they're running at if they're running at roughly your art speed for example or the sort of thing you expect from i2C or SPI then you can pretty much spot them and then you can use the software itself to try and decode that data after the event you just capture it in realtime and then decode it in lots of different types of ways to find out if you can make any of those pins give you some data in this one it says the word found back to us then I think across what they found in it but essentially you'll see things like the linux kernel starting up once you've found that you know you've got the transmit pin the receive pin will probably be next to it unless they've gone out of their way to obscure it we have had them on the other side of the board and all sorts because sometimes they really don't want us to find the serial port but they're great if you can't find a serial port or you can't find the port you want get a logic analyser what do we do when we've got the serial connections or we've guessed them we use bus pirates they're about £30 maybe a bit less you can get them from call components spark fun places like that basically it can emulate SPI i2C1Y2Y and a whole host of other serial protocols and it provides a USB serial interface back to the computer so you connect to it and you talk to it over minicom and you configure it to do the thing you want and it sets up those pins to be the right kind of pins for what you've asked it for and then you can bridge together the two chips essentially so that your serial coming in is then bridged to the serial leaving and going into the device the other device on the slide which is blown up to about twice the size it should be but you need to do that simple fddi connector it only does your art essentially you buy them for about £3.50 for Amazon or eBay and they'll do as good a job as a bus pirate when you only need your art so sometimes the timing on a bus pirate struggles at 115kb is a warning sometimes we've managed to fix that by tweaking the timing manually but sometimes it doesn't work whereas a USB fddi 115k just works so have both take your pick as there's always a use for both of them just to quickly show you how easy it is to set up a bus pirate this is like the minicom session so we can sort of see the bus pirate login thing and is a prompt and we went M for mode we selected UART we then put in some answers to the questions to give it the parameters of what kind of UART we want and right to the end we run macro 1 which is like the bracket 1, bracket bit right towards the end of it if you want to do this your device is then trapped in bridge mode until you de-power it so you say yes and then if you're really lucky it will already be logged in as root and you have now one you've now got interactive access so you can now go and grab the firmware from it or whatever you want to do I'd say probably 25% of the cases we find it's already logged in as root no login prompt to get passed in another 50% plus cases there'll be a login prompt and you can guess it's root and no password or root and root as the password or admin with root as the password or whatever is written on the sticky label on the bottom of the device admin and whatever but you can guess there's about 20 guesses that you want to make at this point about what those creds might be and if you can't guess within about 20 goes make the assumption that it's not what you expect and it's hidden in the firmware or the password file crack the password for the account and then go back to this and then login on a couple of devices we don't find any serial port on some devices more commonly we find the serial port during the boot-up process we get the kernel messages we get a lot of log output but we don't get a prompt at the end of it which is really annoying and they do that because they've simply turned the thing off well we'll come back to how you might attack that but things like the boot loader so instead of running slash init or slash sbin slash init which might be some program or script to start up the system we might change that command line parameter to slash bin slash she so that it starts up a shell instead of running the system up at which point we can then run whatever command they would have run in the background with an ampersand on the end and that will bring all the system up and leave us with a login prompt a logged in root prompt there's other kind of attacks we could make at this point literally given the kernel parameter if you can edit the boot loader we can set one of the kernel parameters that actually turns on the serial port adds a getty on the TTY so there's stuff you can do if you don't have but in most cases we find it will start up there will be a login prompt and we can go root, return, return and we're in as root so what do we do once we're there, well we want to get stuff out obviously trying to find some useful bits and bobs this is just a snapshot of some of the things that we typically run I mean if you do use Linux this is pretty much all obvious bits and pieces but what we're trying to do is get an idea of how the system works how it hangs together, what's running how the system's laid out how the networking processes fit together that sort of thing so we tend to dump all of this stuff into files but get it off and refer to it continuously while we're testing the device because it saves us having to go into the device again and type the command again we like to do most of our work on a nice Linux box with all of our tools available not on the crappy little embedded device that's got no tools so the obvious thing to do at this point is to regress the firmware we want to get the firmware off the device we've got interactive access I did say once you get interactive you should be able to get the firmware off you need to find some space because you need to you need to copy the contents out of the actual dev nodes quite simply if you went into slash dev which is the dev node kind of directory that has these sort of special kernel hooks to all of the different devices that the system has on a normal Linux system there's loads of them there's probably 200 but on a embedded device there's kind of 20 30, it's tiny numbers so you can go through them all have a look at the dev nodes have a look what they're called and just find the ones that you want one of them will have a number of numbers so they'll be like MTD1, MTD2, MTD3, MTD4 et cetera or there might be MTD block one MTD block two it might be SDA, it might be SSD there's a lot of different ones but you'll be able to spot it because it will be the one with the incrementing numbers they are the disk partitions essentially except of course it's not a disk it's a solid state device in non volatile ramp so when you find them we can't copy those directly we need to use DD to extract the contents from them to copy the actual underlying block information rather than the link itself so you need to find some space so we use mount and DF to find the space DD it's somewhere where there's space and then pull them off how do you get them off well TFTP is my preference because every device seems to have it and it's trivial to set up a TFTP server for lots of good reasons but more fun is if it's got a web server chuck them in the web route you probably have a complete interactive access over this thing you can write anywhere you like if you chuck it in the web route you can just browse those files down quite happily with your web browser alternatively set up some kind of tunnel some devices have USB either as a client or a host somehow even have an SD card inside which is perfect I had a 4 gig SD card inside and had no dates on it I don't even know what date it was supposed to be written to it but it was easily enough for what I wanted to do wrote the whole lot to that pulled the SD card, checked it in my Linux box and had all the files the other thing we want to do is we want to get the running firmware what's the difference between firmware and running firmware the firmware is the one that the factory set that is there for when the system starts up and the running firmware is what it looks like after it's started up when it's created all of its temporary files it's quite interesting the temporary files that are created are literally the ones that describe what's going on in the system so if you can get the running firmware expanded as a file system and you can get the actual firmware expanded as a file system you can do a diff between the two and it will quickly tell you which files change in the start up which will give you some clues where to look if you're looking for settings for example but it's very useful to have both because the surprising differences between the two and they often are surprising for reasons that you can never quite fathom but those differences will be the keys to finding out how the system works so exactly the same trick as before except we tie up the file systems instead so what does a firmware image look like or what does a firmware look like well it's divided up into a number of different sections obviously the boot loader starts the system up it starts up the kernel it provides the kernel with an uncompressed RAM disk that's pulled out from an MTD block and then you have a number of compressed file systems possibly by that I mean squashFS or cramFS they're pretty much read-only read into memory as a RAM disk and then they're used read-write by the system but none of the changes ever get written back to the block they're kind of just RAM disks but then you have real file systems running on the SD which will probably be in some esoteric format because you can't run like EXT3 or EXT4 on an underlying SD device because you trash them so instead we have very odd file system formats like UBI or YAFS so you have to look but you'll find those and they'll be like read-write ones and they're useful because if you want persistence so you can get some code on there that's going to run every time it starts up they're the places where you might want to hide it and then there'll be raw settings partitions partitions which you can run strings on and find some interesting information but you can't find a way of mounting it well it's probably because it's just written too rawly you know as directly in a firmware image these are often packed together one after another sometimes they're packed in clever ways sometimes they're encrypted but not usually sometimes they're signed but hardly ever but hey ho so how do we extract if we have one of those files that's got all the say we've like man in the middle the update mechanism where we found the firmware online and we just want to be downloaded we've just got one big like 4 gig image or whatever it is and we'll just extract the bits out of it well you can just bin walk but bin walk is a bit poor a bit rubbish it'll guess based on signatures in the file very much like antivirus and we all know that antivirus sucks but hey it might just work in this case you know if they're not trying to hide stuff then give bin walk a try the minus e-switch will actually extract the sections for you or you can just take down the numbers and use dd and pull them out yourself but I should say if any point you sort of think wow this is all going over my head I don't understand that as long as you get the basic idea of what we're trying to achieve on the slide google it it's all off the shelf techniques I mean everyone's been talking about this for years so but then to actually extract the individual sections once you've got the sections out as individual files you want to extract them then firmware mod kit of which there's about 200 different versions because it got forked badly find those they can do all sorts of things the unsquash ffs all and uncram ffs all are brilliant if you've never tried to unsquash a squash file system you probably don't realise that there's about 19 different variants of squash ffs and you have to have the right one to extract they do horrible things like just change the sqsh from the beginning into capitals or reverse the order or change it to four different letters entirely and your unsquash tool suddenly dies doesn't work there's also lots of other tools I mentioned on there you can get them all from the internet 7zip on windows is always worth mentioning and the reason for that is because I believe it finds lots of stuff that we don't find any other way I don't know why we think it's because it really lacks on error checking so when things go wrong in the archive and it carries on whereas all the Linux tools go no you have to stop this isn't a valid archive so definitely get 7zip and stick it in your windows vm because you'll probably want it one day I'll let you digest that one while I have a drink I'm not drinking enough if anything be good for time yeah we're only halfway through this is about right yeah this will be fine we might even finish early at this rate you never know oh we have come ding dong thank you very much I hope I'm not talking too fast but I didn't say you probably can't stop me from where you're sat so you just have to accept it maybe watch the video later or something I don't know so this is a module of you of how an embedded device looks pretty much every one of them runs Linux or the ones that we care about do because they're the ones that we can hack easily they typically run DHCP and DNS if they're a client device that connects to your home network then they'll have DHCP clients and DNS clients if they're a router or gateway device then they'll probably have DNS servers and DHCP servers or DNS relays and DHCP relays whatever however they want to do it most of them have a web server on there why because it's easy to tell your users users know how to use web browsers so you say go to this URL and it pops a page and then they can log in and they can set the settings up and users feel fine happy with that so we see that that's the standard go to for an embedded device so you're probably well aware you've probably got some they've got these horrible web servers on well of course it's not a real web server like Apache it's going to be a web server that is tiny and it's got squeeze on to an embedded device and the way they do that is they leave out all the security checking which is just hilarious really there's different types you often find go ahead web server often with a web proc back end you get mini-htbd you get lighty and more or less they have bugs that you can dig out and find on top of that they typically run CGI scripts because they're still web 1.0 and they love them and the great thing with CGI is that there's command injection opportunities left right and center because again they're written by people who have no idea about security so on top of that these days we find web services because they're feeling a bit more web 2.0 and again that might be JSON or XML or whatever but their parsers are pretty poor and out of date so they're all attack services that we want to look at UPNP again if this is a gateway device it will probably be a UPNP server if it's a media device it might be a UPNP server but if it's a device that needs access to the internet like your xbox or something like that then it might use UPNP to tell your router to open a port up for it so the internet can get to the device via your router UPNP we know is really a lot of the time it has bugs in it that's all I'll say you need a tool to exercise UPNP properly we have one in house which is brilliant but it's never getting published I'm afraid but you do need a tool so if you want to interrogate UPNP it's kind of like an HTTP XML kind of thing sitting on top of a new DPSSDP listener yeah if that made no sense then that's how it feels trying to hack UPNP but they do make massive mistakes because no one's looking at it and then there'll be other services running on the device that are specific so if it's a camera it might have a video service where you can watch the video of the camera if it's a VoIP telephone it might have an audio service on top of that there'll be comms to the cloud because at some point you want to go to the cafe you might want a baby monitor that you can put on in your house and then go down and have lunch or dinner with all your friends and watch the baby through the phone really stupid idea, we called that the Madeleine McCann model when that one came out and told them what a stupid idea it was but there'll be comms with the cloud because you want to be able to roam you want to be able to leave the house and then the device will talk to the cloud and then you'll talk to the cloud via your phone and get the data that way on top of that there's the NVRAM for the settings that is an attack surface because it tends to trust whatever it finds and NVRAM is being gospel so if you put something in there it will use it without really checking to see if it's in the right format so we want to look at those things so what I'd say is you want to get used to how the device works use it as an actual device and feel around it and get an idea of what you'll have all these services running those PS and net stack commands from before but it'll be useful to actually understand how they're hanging together when you kind of have that understanding just to start attacking them so the sort of things we're looking at are command injection command injection is a great little bug it's where your input that you type in gets added to a buffer somewhere and then they call system with that buffer or they call exec or popen or any way of starting a shell process and if you happen to put in some bad characters in your input like these back ticks in this situation then by the time it gets to the point where it runs it in the system command the shell interprets that as I want to run this command like your input in another shell and take the output of that and use that as the placeholder replacement in the command back ticks are only one way to do it there's like semicode and there's dollars brackets, there's all sorts of things you can do depending on the actual shell in use which varies from device to device but yeah if you've got any way you can input data you want to be sticking some back ticks in and various other shell meta characters and seeing what happens that could be a form on a web page but it could be the response to a DHCP request for example which is the funky places where embedded developers don't look for bugs you know it's sort of like here's a DNS reply that happens to have back ticks in what do they do with that, they run it brilliant we have the traditional buffer overflow that's simply where the amount of input that you provide is more than what the developer expected and he conveniently or she conveniently allows you to copy all of that data over the top of their buffer and over the top whatever comes after in memory which might be another buffer or other variables so you might be able to affect the state of other things if you provide enough data it might start overflowing whatever comes after it, the structures of the program so you know if your variables on the stack then you might run over the return address for the function and you might be able to control the program counter from there I will say there will be no modern Linux protections on here so if you can overwrite the return address and get execution then nothing will get in your way, there's no stack in areas there's no address randomisation or anything along those lines if you're in heap then you'll probably overflow the heap structures at which point the next free is called against this again you've got complete code execution if this makes no sense, there's some papers on the internet you can read, one's called once upon a free and the other one I can't think of it, I can't remember it's on my head that's it, smashing the stack for fun in profit a la foen anyway you believe how much I'm melting up here it's like I'm dripping out of my eyes onto my laptop now drink more every mistake ok we're still good for time right so the next one last class of Berg obviously there's millions of different classes of Berg but another class of Berg that we always look for format string vulnerabilities format strings are the ones that people forget to look for sometimes because they don't completely understand how they work perhaps but it's as simple as somewhere in the program the developer uses one of the printf functions and uses the format string argument that is provided to it to have some control over some of your buffer ends up in there for magic reasons and the magic reasons are because they generally use a printf function like sprintf or snprintf when they should be using a copy function and they're simply trying to copy from a format from one buffer to another but because they're using printf the buffer they're copying from gets interpreted as a format string which they probably weren't expecting they might not be expecting you to put percent characters in there for example and they might not sanitise them out so if you put percent characters into that buffer and again I'm not saying this is just like forms on a web page for example this might be you know say DNS responses or it might be literally in the nvram settings when you can get something in there because it's not checking or sanitising then when it comes to use that printf function or from that family it interprets that as a variable on the stack that it needs to insert into this string and if you haven't if you're providing extra requirements for extra stack variables and they aren't being provided to the function then it will find them on the stack and with good use of format string characters percent n being very interesting you can take code execution from this point so stick a percent in or percent s or a whole load of percent x's and see if it crashes because it probably will so obviously looking for bugs might be quite tricky it might not be your normal thing of course we always take we haven't got a lot of time when we're doing this for a day job so what we do is we typically load to the internet and ask the internet what bugs exist and the best way to find them is on the actual page which lists all the bugs so look them up and find out if they're relevant often the modules you need won't be there but find out if they're relevant and then sometimes the defenders will change the code themselves and they'll accidentally fix bugs without realising it and that's really really annoying but on the web page where you can get the code you can usually get the release notes so you can find out what changed so if you the CVE details tells you which version fixed it you can get the release notes so that version will tell you what fixes they made sometimes they'll tell you quite a lot of detail on top of that you can download the source code so I recommend download the source code for the version where they fixed the bug and the version before and the version that you've got do a diff between the version where they fixed it and the previous version that'll give you the exact change required to fix the bug and then retrofit that fix back to the version you had before and you'll see where the bug is once you see where the bug is you now just have to use a little bit of science and maths to make you to be able to exploit it so you know the great thing with open source software is they're really open so what else do we look at well if it's not open source software I mean half the services will be open source because they're too lazy to write a web server but half the services will be bespoke because no one's writing the code that they need so they sit and they beave away and they produce some stuff that typically has a load of bugs in it so you generally only have the binary executables at this point to disassemble them so we use IDAPRO which is a screenshot of this tool here it's not that expensive for what it does there's a reason why it costs money it's because it's really really good the add-ons themselves are amazing but they cost even more money the things we're kind of auditing for is the use of system, exec, p open, things in that kind of families because they run commands in the shell and if we somehow control some of those buffers that get run then we might be able to get our command in there that's where we might look for command ejection for example we look for the str-copy, str-copy, mem copy there's a str-copy str-len kind of like configuration which is pretty awful but we're looking for buffer overflows if anywhere where it's copying data for one place for another and it might be ignoring how long it is as I said the classic is where you swap out your str-copy for a str-copy which has a limit on how many characters to copy they do a str-len on the source buffer to decide how many characters to copy and you're like no, no, no, surely it's the length of the destination buffer that matters not the length of the data you're going to copy because that's no different to just copying it regardless so mistakes get made left, right and centre and no one's really looking for them nobody in their QA teams actually know how to find these bugs and fix them so go look printf family obviously for format string vulnerabilities and buffer overflows and the likes receive, read anything that gets data and handles data from the user, from you, from the network have a look in there because they typically will expect the data to be a certain size and you can probably see that quite easily and the other thing is is look for the hard-coded values it's really difficult to use credentials or crypto keys on an embedded device and secure them it is potentially possible with a proper secured bootloader to have a chain of encryption where the first key lives in silicon that the users can't get hold of but no one ever implements that because it's expensive and it's difficult so instead what they do is they hide the creds in the file system and then someone spots them and goes do you know if you get to this file you can get all the crypto keys and they go oh no so they hide them in the binaries instead they literally compile them into the binaries and do you know what's brilliant about that? they don't change at least when there's a file on the file system you could go someone's discovered your crypto key and they go yes change it and it changed but when it's compiled into the binary you can't change that without updating the firmware so once you find one key or one credential or one piece of useful information in the binary on your device you can apply that to every other version of that device out in the wild so you go to your next door neighbor's house where they've got one you already know the keys in it because it has to be the same because the software is the same so yeah they're probably going to be in the binary they're going to be encoded or obfuscated so you sometimes have to do a bit of tracking to actually work out where they are but you know they're there going to have a look so on top of that again this is another post-node thing but lots of people add crypto to their devices partly because it's a bit of sales technique these days and partly because they feel like the NSA is watching them well if they implemented themselves it will be broken I can guarantee that now because world-class cryptographers implement crypto and make mistakes it's really really hard to get crypto right I don't want to go into too much detail I've broken AES 128 bit before not because there's a flaw in the AES algorithm that I exploited but because the way they used it was poor they only had a million different keys when there should have been a trillion trillion trillion different keys but all I'll say is everything has crypto now and if you are trying to find bugs and stuff and you don't understand crypto then you're missing out masses because typically you see something that looks random or encrypted and you think oh I don't know what to do with that that's encrypted take one of the free courses at Corsera.org such as the Stanford crypto course or the Mariland crypto course and next time you see some encrypted data you'll go ha ha I bet that is really badly broken I just need to look a little bit harder to find out how and they do in fact that previous slide if anyone did bother reading the code on it was an example of broken crypto but I won't go into that any further modifications you're going to want to change it if you're trying to repurpose repurpose your device so that you can make it do stuff it shouldn't do run an IRC server on it or make it make cups of coffee or whatever then you're going to need to modify it but equally if you're hacking it to find bugs again you're going to need to modify it because you want persistence you want your shell to come up every time you turn it on to make life easier for you you don't have to want to exploit it every time you turn it on you want to be lazy like a pentester so things to change with the bootloader like I was saying before we can change the kernel options so we start up with a serial port we can change the kernel options so we start up a different startup script instead of in it we might start a different kernel we might use a different RAM disk there's all sorts of things you can do there you can change the RAM disk literally extract it copy it off, egress it the way we talked about before extract it change the files in it, repack it and then get it back on the device and then overwrite the RAM disk that exists it's a bit of a cavalier, crazy approach but it will just work other things you can do is file systems these might be easier mounting file systems that are designed for SD cards unless you've got it on the actual SD itself so UBIFS probably won't mount but your UBIFS extract will work so the best way to modify these is from the interactive shell itself you want to be on the device to modify running file systems rather than trying to do it externally like we would with a RAM disk overwrite the MV RAM depending on what it is repacking the firmware file is a great idea because no one really checks the signatures on firmware or not if you can find a way of giving the system a firmware file it will probably believe you that that's a valid firmware file when they start checking signatures as they sometimes do then you might need to find a bit of code that checks the signature and extract find a way around that bit of code you probably won't have the key to create your own signatures but you might be able to turn off the signature checking for example so we've mentioned the update mechanism before I want to mention it again briefly because we've now come a long way from where we were 40 minutes ago so imagine we're attacking the update mechanism to steal the firmware for example one of the things that's interesting about the update mechanism is typically written by different people who write all the other TLS code so what you've got on the system is the vendor that supplies the chip set and the underlying reference design they often provide the update mechanism because that's part of their bit of house and it gets passed on to the next vendor down the line who then writes the software on top that provides the network services and the video services or whatever it is that your device does so if they haven't made a mistake for example in the actual network services don't assume they haven't made a mistake in the update process because they're probably written by different people who will make different mistakes so do go after it if you can get the firmware then there's lots of other ways of updating it there's often hidden web pages that let you update firmware there's usually a function or a command line tool on the device itself that will let you update firmware so yeah, go for nvram it's a massive gaping hole it should be encrypted it's so easy to extract the contents of why isn't it encrypted well the reason is because encryption is really hard like I keep saying there's a lot of settings excuse me there's a lot of settings data hidden in there so you might want to extract them in some cases they do clever things like encrypt some of the settings data when I had one device where every piece of data in the settings was in clear and I could just go they provided me with a command line tool that I could run like nvram get and it would just tell me I'll give it the name of a key and it would just give me the value it's brilliant but one of those values that gave me back the password was encrypted of course the device itself has to be able to extract that key and decrypt it so everything you need to decrypt it is in your hands already you just need to find out where that is and that's where we go back to the binary and we start looking for keys because when you find keys that's the sort of place where they might get used so I sort of say identify the complexity and then extract the data and the reason for that is if you just jump in with both feet start writing code trying to build a technique to try and get the encrypted data out you might spend a few weeks doing that before you're successful and in the meantime miss loads and loads and loads of other bugs like I did so identify the complexity first and then if it looks reasonable or when it becomes the most high priority thing to look at then look at it then extract the data so almost at the end the attack surface is massive and tiny at the same time there aren't many services running on these devices so it's quite a small attack surface but when you look at the individual with the actual bits and pieces there's so many things you can do with them and there's so little protection in place to stop you doing stuff with them that let's say it's big and small at the same time what I typically say is go after all of it audit the lot it's a very little excuse to leave anything running on that device that you don't actually understand because you might be able to count the processes that are running on two hands that's how little there will be actually running on the device and if you can't audit all of those processes then take some more time get some more skills, do some more training learn some more stuff because the bugs will be there I guarantee I think the rest of that slide kind of speaks for itself but yeah there's just lots of mistakes that people make it's a lot easier to hack these devices oh yeah that's what it seems we've got a couple of minutes to say actually so I went and saw a talk at 44con last year I think it's last year or the year before and someone was hacking into a kid's toy the toy was a rabbit a plastic rabbit and it had these ears that could move and light up and things like that and you had an iPhone app and you could kind of go like ears up, ears down like go red, go green I can't remember the details I can't remember what actual toy it was and the guy did a talk about how he hacked it fair play he hacked it but he treated it like it was a server on the internet at no point did he take the covers apart and have a look inside it's in your hands if you're hacking something on the internet yeah it's really hard to go to that data centre get your screwdriver out start taking it apart and steal bits from inside it to make your job easier but if it's in your hands like you've bought the device yourself get a hammer hit it hard and find out what's inside it because when we were 10 that's how we discovered how things worked and now all of a sudden we're adults oh no we've got to do it in a high brow way be clever and stuff no we don't have to be clever we can just take stuff apart and look at how it works it's mostly linux host auditing as I said at the beginning so get good at that when you fail it's because your linux skills aren't good enough typically so you know that's how it'll go we all go there we all have to learn off each other but as I say it's a lot easier to start with a firmware image and sometimes you get that from the web and sometimes you get that by taking the device apart but it's very satisfying to smash a device open and end up with a serial port that won't be logged in as root and through that extract the firmware like even taking the chip off the board and extracting the firmware from it that way it's a very satisfying thing to do even if it takes another five minutes because you feel like you've done something useful I guess so in summary get interactive root get the firmware and the running firmware and diff the two and then look for vulnerabilities in everything that is running if yeah it's not that hard honestly hey if I can do it and I'm drinking Guinness then I'm sure you can all do it I'll be in the port Cullis Cisco tent not that I work for either of those companies but they're so lovely with their hospitality that I'll be over there drinking their beer so if you want to ask any questions please do I'll quickly answer the obvious question before I stop because I've got another four minutes according to this the two people on the left that's me and my boss looking for bugs the two people in the middle that's our customer at the company we work for being gleefully happy that we found holes in their product before they released it to market because this means they don't have to get embarrassed and explain to their boss why we've got all these vulnerability notices coming in or things being talked about at conferences and screwing our brand positioning and the final person over on the right there that's how the vendor responds to us telling them that their product is shit pardon my French but it was kind of necessary to add the emphasis to that final point so my website has a few bits of tools on there if you like debuggers and P-trace then there's a couple of tools on my website that you can get hold of in the future there'll be more stuff on there but right now that's where we're at so thank you very much we've got time for a couple of questions of anyone's got any anyone at the moment cool I've got one very quickly so your job now is securing things given that if you do your job perfectly given that you should know what you're doing but new vulnerability is going to come out so how can you protect something so you can release products say I can't find anything in here then new vulnerabilities come out so do you think it's ultimately impossible well the new vulnerability depends if it's a new class of vulnerability that we haven't considered then fair enough because new classes of vulnerabilities do come out and you have to retro check those against every device that you've ever thought about but new classes of vulnerability only come out once in a blue moon usually like once in ten years or so so they will be there but don't worry about them too much in terms of actual bugs if I do my job well then I will have audited all of the code that is running on the device and found all the bugs that exist or contained it within a defence and depth environment so that you can't exploit them and then when new bugs supposedly come out they won't affect the product hi thanks for the very interesting talk I wonder how secure boot will affect your work or does already affect the stuff we try we're only just starting to see secure boot loaders coming through we're not hacking mobile phones for example we might hack mobile phone apps but we're not actually hacking a lot of mobile phones themselves which is where we're seeing SPL coming in early some routers are starting to employ SPL and yes it will make our job harder because one of our attack vectors as I said is via the boot loader and if you can't modify the boot loader but the other thing to remember is that the people implementing the SPL don't have any skills history or knowledge in how to implement that securely so they will make mistakes again the trick is to not just naturally assume that it's going to be secure because they've used secure boot loader type technology but to think how would I have messed this up oh yeah I'd have forgotten to do that I don't know on a Friday afternoon but I've just been in the pub and then you'll find the bugs because so yes it will make it harder but you know I'm still confident wonderful cool I don't think we've got any other questions now so you've demonstrated very well that Guinness is good for you because I've been drinking water and fell off stage and you've been alright thank you very much thank you thank you very much Kevin Sheldrake