 Okay, got some misbehaving equipment here just a second. As is always the case, right? I forgot to sacrifice enough to the demo gods this morning. Yeah, it could be a problem. I don't know. But let's see what we can do about getting something to run. Come on. What the hell is it doing? I hate when that happens. All right, when all else fails, reboot. And we'll see what we can do about that. And hopefully it'll reboot. Well, I'm happy to see you all here. My name is Mike Anderson. I am Chief Technology Officer and Chief Scientist for the PTR Group. Now, the PTR Group is a small embedded systems consulting shop. We're kind of a boutique engineering shop. It's kind of the best way to describe us. And some of the things that we do as kind of a general basis, one of them happens to be reverse engineering. So commercial reverse engineering and reverse engineering for various other organizations. I happen to be located in Washington, D.C. So you can get a guess as to who it is we do reverse engineering for. And what we're going to do in this particular session, assuming I can get my computer to behave itself, is we are going to talk a little bit about the process of reverse engineering. And we'll talk about reverse engineering on the hardware side and reverse engineering on the software side. See if I can get everything to behave itself. And I happen to like to live dangerously, so I actually have a board that I've hacked into. And we will give it a shot to see whether or not it will actually do what I want it to do. This one happens to be a router. It is a router from our good friends at, where are they? Verizon, that's it. So it's a Verizon router. Yeah, yeah, now it's an ActionTech router. And let's see what we got here. Oh, that looks better. Come on, give me, ah-ha-ha, yay. Yippy-skippy. No, it's an ActionTech router. So let's plug in some video. See what happens. I want to do the following. Let's do that and apply. Yes. Now a little Klingon, never can have enough Klingon, you know? Ah, you piece of crap. Let's try it again. And this time, let's just do that. You piece of shh. I hate windows. Well, this is just PowerPoint. Running under wine, and sometimes it does what it's supposed to do, and sometimes it doesn't. I could certainly do that. But we're going to try it one more time here and see if we can do a slideshow. There we go, all right. And they asked me why I drink. Okay. So we're going to do a quick introduction to reverse engineering, although this is not going to be quick since I've got a couple of hours here, but we are going to, let's see if I can turn that on, and yes, yay. We are going to talk about what reverse engineering is. We will also discuss a little bit about why you would do such a thing. Some of the hardware and software tools that we'll use in the reverse engineering process. Some of the impediments that various manufacturers will throw up there to try and slow you down, and the general process for being able to hack into hardware and then hacking into software, as the case may be, and we'll actually finish up with a kind of a brief discussion of some protocol hacking that we did. Definitely an interesting problem set, and we'll get into all of that as we go through the material here. One of the big issues, though, is knowing when to declare victory. It is always a tough thing in doing reverse engineering. You're not quite sure when you've done enough. There's always something else you could potentially do, and it's really a question of when you know to declare victory, fold up your tent and go home. And then, of course, we'll finish up with a little bit of where should you go from here? All right. So what exactly is the reverse engineering process? First of all, let's assume we're given a piece of hardware or software, as the case may be. We want to deconstruct it. We want to find out how it was put together. We want to find out how they built it, what kinds of things are inside of it, whether or not we can extend it, and a few other issues we'll get into as we go through the material. Now, this will typically entail removing the case, repopulating connectors, disassembling the software, and it may require the use of hardware debuggers. We'll talk a little bit about that as well, and some other test equipment to figure out exactly what kind of connector we're actually looking at here on the motherboard. But I will put this out there and make sure, especially for you at home, don't do this at home unless you know it's legal in your jurisdiction. Here in the United States, we have this nasty thing called the Digital Millennium Copyright Act, and the DCMA gets really testy about disassembling firmware. So don't do that unless you have somebody's written permission. Hopefully a law enforcement agency. Now, that being said, as we go through the material here, we will be talking about some techniques that in certain jurisdictions will get you arrested. Don't do them. I am not going to come bail you out of jail, and you can point to the YouTube video all you want. That's not going to do anything for you. This is the same sort of thing that I can explain how to make a silencer, but if I make a silencer, that's illegal. I will explain to you how to do reverse engineering, but actually doing the reverse engineering may be illegal in your jurisdiction. Please make sure you check that first. All right, so why do reverse engineering? Well, part of the goal may be to repair or repurpose or upgrade a particular piece of hardware that you may have, or maybe you're just curious what's inside the box. Those of us who back in the dark ages used to take televisions apart and radios and things of that sort just to see what was inside of them, much to our parents' dismay. It was definitely something fun to do, and only occasionally, actually after, it only took once. When you get across the anode of the television set and you get lit up by 35,000 volts, you only have to do that once before you respect that red wire. Don't touch the red wire. Now, often, and of course, part of this now is we're starting to see some ground being gained by the repair movement. Repair movement is a great thing. We should be able to repair those devices, especially phones. If we were going to spend $600, $700 for a phone and the battery goes dead, I'm sorry. I am not going to buy a new phone simply because the battery can't be replaced inside of the phone. We'll figure out a way to get in the phone and replace the battery, and we'll do all of that, again, within the limits of whatever our jurisdiction may be. There are going to be cases. Now, this is something interesting, and we found this happening more and more often. It's referred to as archaeology. Now, whether it is because somebody made a device and when they made the device, they lost the engineer that made the device. They left. They forgot to document it. The documentation was in a package that was installed on a server and the server went buns up and they can't get to the package anymore. The same thing happens with software. I've had cases where they need to figure out what the software that they wrote actually does. It's oftentimes many years after the development of the software and nobody remembers exactly what they did or how they did it and it's not documented. All those happy things, those are all reasons why you might have to go in and take a look at an existing device to figure out what it does and how it works. Now, the other possibility is you might be presented with a suspicious device. Let's assume something fell off a truck and showed up at your back door. Well, what are you going to do with that device? You're certainly not going to plug it into your own network, I hope. You never can tell. Now, in this particular case, actually that picture that you see right over there, that hotspot right there, that's an actual chip under another chip. That motherboard was specially modified by somebody that happened to come as a commercial device and when we popped open the box and started looking at it in an infrared, we said, wait a minute, this chip's a lot hotter than it's supposed to be. I wonder what's here. And you start taking a look at it and it turns out somebody had actually hogged out a small space underneath the existing part and put something special there. Now, obviously that kind of thing shouldn't happen all that often, but it does and we're actually starting to see quite a bit more in the way of counterfeit equipment coming out of certain locations in Asia. This counterfeit equipment, some of it is really interesting. You'll look at the real part and it has silver solder leads and the counterfeit part has copper solder leads and it's the only visible sign that you have that it's actually a counterfeit. These kinds of things are starting to happen more and more. General rule of thumb, never buy anything off of eBay that you think is going to go into your corporate infrastructure. Just a suggestion. Now, there may also be some special software on a device that you're concerned about, whether that's malware, spyware, or govware. If you happen to be, now what's govware? Govware happens to be something that we find in the German speaking countries. The Swiss have megapanzer and the German government has one called R2D2 and these are legitimate pieces of software that the government installs on your machine when you're asleep. And at that point in time they are key loggers and other things that they track you and see what you do. If you've been talking to people you're not supposed to talk to they will come knocking on your door and GSG9 is very unfriendly with the MP5s as they kick in the door. It's a general rule. Don't do that. That would be bad. But govware is certainly a thing. We have actually found one special thing that I can share with you. There was equipment from a large manufacturer of equipment of computers and this particular large manufacturer had sold a whole bunch of computers to a defense contractor and the defense contractor as part of their normal modus operandi take the equipment and put it into a sandbox. They basically put it into a small isolated network enclave and then they start dorking around time. And sure enough after about six months of time it beaconed out and went back home to China and said hello they of course asked the manufacturer what was that the manufacturer said well it came from the original design manufacturer that way. So the fact that they were actually working out to a place in China that was doing all the biases and everything it was a weird situation. We actually tracked it down a little bit further than that and what it turned out was they were using the system management mode on the x86 and in system management mode it's built specifically to allow you to take a look at you know temperature and power rails and things of that sort. It's made for the server world. It's a non-maskable interrupt so you can't turn it off and what was happening if you switched into system management mode there was a TCP stack in there and it was phoning home. So little things like that you should probably keep an eye on them. I always am suspicious of any new equipment that I get but I tend to be paranoid anyway at least that's what my wife will say. Alright, so what kind of tools should you have in order to be able to just simply disassemble the device? First of all, you're going to need a Torx driver set Torx is one of the more popular weird screws that are used in a lot of these cases but you'll need screwdrivers in various sizes a precision utility knife not to use the trade name that you all know so we don't want to do that but a precision utility knife spudgers these are basically little pry bars plastic pry bars guitar picks work really well suction cups small hex drivers thankfully thanks to the right repair movement these are all available as a kit you can go to ifixit.com and you can get that little kit that you see there in the upper right hand corner that little thing right there that is a really incredible thing now one of the things that you don't really probably can't recognize is that little black thing that looks like a hot dog about that length that's a really cool device that is a piece of it's a squishy slug-like thing you put it in a microwave for about 30 seconds and it heats it up you slam it on the back well, you don't slam it but you place it on the back of the device you're trying to unglu and what it does is it transfers enough heat to the glue to soften it so that you can then pull the back off of machines, off of devices like, I don't know, iPhones that you're not supposed to pull the back off of and that's also where the suction cups come in by the way if you use suction cup to pull it apart those are all things that are definitely handy to have another thing that you can do, of course, if you happen to have something that you know has an adhesive in it of some sort, heat guns if you happen to have solder equipment and you've got a heat gun that'll work blow dryers if you don't happen to have a heat gun or this microwavable gel you'll heat the adhesive up and then that will make it pliable enough that you can get the box that you can actually get the box open another thing that's really handy and this is one of those things that you don't really realize until you get into the business and that is an inspection microscope you really need to be able to read those teeny tiny little letters on certain parts and even with my bad eyes with three or four X magnification I still can't read them so these little inspection microscopes they're USB powered you can actually use them to take pictures of things so if you need to prove to somebody that it was a particular part you can actually take a picture of it and they go up to 600 power magnification so it does a really nice job and definitely it's handy to have one of these things around now electrical test equipment especially when we're trying to figure out exactly what kind of circuits are in the box and what voltages they're running a volt on meter is a must don't scrimp on these spend a little bit of money, get a decent one they'll typically cost you a really good one will cost you anywhere from $70 to $150 the one that you used to get at the old Radio Shack for those of you who are old enough to remember Radio Shack the $9 meter that one doesn't work very well I still have some of those and I just use them to tease here try this oh what's this meter it's got a little needle on it how do I read that the problem with LCD based meters is often times the event that I'm looking for happens so quickly that it can't update the screen fast enough so there's a voltage change there but it's so quick that I can't tell what the voltage change was if that is the case and often times it is then we drop to a digital storage oscilloscope now if you're a software person go into your hardware developer one day and ask to hook up the digital storage oscilloscope and he will look at you like you have two heads and why do you even know what one of these things is let alone want to hook it up to something a DSO is a really handy application we'll see a couple of pictures of where I used a DSO to take a look at a signal and it turns out that yeah you can spend a lot of money on these things I mean tech's got some wonderful scopes 4 channel 2 gigahertz scope that will set you back $30,000 but the reality is 50 to 100 megahertz that's good enough for most applications most of these kind of applications at least so that means you can probably pick up a couple of hundred dollar scope and it'll do okay now some of the little PC scopes that are USB powered some of them are not quite fast enough because they're only in like the 5 megahertz 10 megahertz range usually 50 megahertz or so will allow you to catch most things that might be of interest another little gizmo that's handy to have is an 8 or 16 channel logic analyzer this one is one of the ones that is powered with USB this is my salie I've got a wonderful I love my salie it is a USB 3 I've got the salie pro version so it's got a USB 3 interface on it when you got to transfer a lot of data really really quick USB 2 doesn't cut it but USB 3 does a great job now another thing that's handy is a sig rock with a location interface aka bus pirate or something similar these are often will use these as a way of just being able to try and identify the type of signal that it is is it a J tag is it a serial port is it an I squared C what is it because if you start plugging things in without really knowing what it is you'll likely let the magic blue smoke escape and that's bad not to mention the fact they come in and go no I took a shower it's not me okay now logic analyzers as protocol decoders most of the high end logic analyzers these days have got protocol decode in them they've got I squared C, spy, can bus asynchronous serial, etc the tricky part of course is just identifying the signals also these devices tend to have their own limitation as to what kind of power you can put across them five volts is a maximum and we have seen 12 volt compatible RS-232 interfaces so the old RS-232 standard actually used to go to 12 volts and you'll find the equipment sometimes that still does so definitely you got to watch out for things like that and some of the really expensive units of these will decode both PCI express and other high speed buses this particular one this happens to actually be from my salee the salee will do 10 mega samples per second or something like that so it's fast enough fast enough for most things now I have a couple of patents on doing video backup out of your video port back in the dark ages we used to do things like that and I had to use high speed echo logic to do that so 2 gigahertz logic analyzers and scopes where the empty mainframe of the logic analyzer nothing in it just the mainframe is $35,000 so fortunately you don't have to have anywhere near that kind of investment in order to be able to do this by the way the salee the 8 port salee I think is $495 something like that it's relatively inexpensive for what it is maybe that's the 16 is the 495 the 8 maybe a little bit cheaper I don't remember right off hand it's been a while since I bought mine okay so now now that we got all these tools collected now what's the thing that we want to do first of all we need to do a little bit of research we need to find out is there an original design manufacturer some place an ODM and if there is an ODM who is it for instance with laptops it's common to find laptops made by a company called clevo they'll be branded with Dell or HP or some other badge on the outside of them but they're actually made by clevo why is that important well it's because you're looking for the FCC ID if you look in the device you will find somewhere an FCC ID associated with it now the Federal Communications Commission here in the United States has requirements for any device that could potentially emanate in the in the spectrum they're doing a certification for a Class B or Class C type device they will have an FCC ID associated with them the FCC ID is you can look it up on FCCID.io among other places you can actually go to the FCC but it's so hard to find it there but there are other websites that you can actually look up the FCC ID the advantage of finding the FCC ID is that you will find that a lot of devices use the exact same components in particular radios radios I actually talk to a manufacturer a Chinese manufacturer one time about what would it where it's the breakeven point for being able to make my own radio at what point does it make sense to do that and they go well after FCC certification and all the stuff the registration and everything you have to go through breakeven point is about a million units for your own radio and what happens many manufacturers don't use their own radios manufacturers will go to another vendor that has an already approved radio set and this already approved radio set has an FCC ID associated with it you look up the FCC ID and you find out it's the exact same radio that's being used in like 10 other devices that's an advantage because one of those other devices may not necessarily be as locked down as the one looking at it may actually have more open source code associated with it there may actually be some you boot versions and things of that sort floating around for the other units that you can then look at those and try to figure out what does that mean to me for this device that I'm having a look at another one which is kind of an odd one that you wouldn't normally think about and that is are there patents involved does the manufacturer of this particular device have any patents associated with it patents are a wonderful thing when you go to the US patent and trade office or you go to the EU patent and trade offices you can the patents are public record and when you bring up the data sheet of the packet of the patent you will find all kinds of interesting stuff in there circuit diagrams box layouts which parts are being used for which motherboards there's all kinds of really cool stuff in patents and another one that you want to take a look at is who are the patent holders and see if you can find out any information about them the reason I say that is because when we get into the disassembly of the protocol we'll find out that it was something specific to the patent holder that actually made its way into the protocol interesting problem with that so patents and a lot of people who are just starting to do reverse engineering forget to check the FCC IDs and forget to check the patents both of those are tremendous sources of information alright so now opening the case just opening the case can be a problem sometimes manufacturers will actually sonically weld the case and if they sonically weld the case getting into it is a dremel that's one of those tools that I didn't mention before but sometimes it's handy to have a dremel just to cut through the case now why do they make it so hard well sometimes they don't want you to see all the goodness that's inside and in other cases they don't want you to see all the badness that's inside like that one right up there that is probably one of the worst solder jobs I have ever seen in a commercial device and that's actually considerably cleaned up over the first two versions that they made cold solder joints I mean it was just horrendous inside and you just go well if nothing else I might just fix it just while I'm here because I just I can't stand to look at such bad solder joints alright other techniques that are designed to keep the casual user out of their hardware one is special screws we see there a picture of the pentalobe screw that was one that Apple made that would keep you out of the iPhones turned out there were only two pentalobe screws in the iPhone it was in the base where the USB connector was but it was a special screw that they had custom made and it took at least two weeks to fab one out of a 3D printed material so usually you'll see torx screws are very popular make sure you have a complete set of torx especially the little tiny torx they are really tough to get a hold of and used a lot there is a secure torx which has a little post in the middle of it that keeps a normal torx screwdriver from going into it so if you go to a fries or you go to not so much you can just buy but Microcenter some of those places they'll actually have security tool kits that have all these weird little screwdrivers inside of them definitely worth the 10 bucks that it cost to buy one of those the tools themselves are not all that fantastic but when you need something that's got a secure torx you need it, you need it then and not have to go try and find it yeah yeah yeah absolutely they are of course sometimes they'll use special adhesives that was what we were talking about earlier when you use the heat gun to kind of melt the adhesive down a little bit to make it a little more pliable anti tamper sensors now anti tamper sensors are really tricky what they'll have is a case that has some special switches inside of it that if you open the case it triggers the switch and zeroes out the flash that is particularly troublesome we'll talk more about that in just a minute and the other trick that they do is encasing the device in epoxy otherwise known as potting the device that's that black glue gooey thing that you see right there and they do that for several different reasons one of them is to keep you from figuring out what the circuit is other times it's to keep you from getting wet if it is going to be in a moist environment we'll want to keep the circuitry dry so they'll do an epoxy coat like this or if you happen to be going into space it's called conformal coating and it'll keep your devices from outgassing we do a lot of work in the space world so it turns out that that happens quite a bit so you have to basically conformal coat it in order to get it to keep it from outgassing and then blowing the chips up now dealing specifically with anti tamper switches and potting any well equipped reverse engineering shop will have an x-ray inspection capability so it's an x-ray machine and basically you just simply put the device under the x-ray and you take a look at it see what it looks like in the x-ray world that will typically let you know that there may in fact be some tamper switches inside of it one of the things we have found with tamper switches is liquid nitrogen does a great job at slowing them down you hit it with liquid nitrogen through the case and then you pull the top off and the switch is now basically frozen you can now put tape on it and the switch won't trigger so liquid nitrogen great to have around not only that but it's fun to play with and you can make great ice cream with it when you are when you're bored now potting comes in several different varieties including special polyesters and epoxy resins when you get to the potting you can actually kind of put your finger in it you can determine whether it's a hard pot or soft and depending on the type of material it is made from often times heating will make it pliable so this is one of those cases where just putting it in an oven at about 200 250 degrees fahrenheit will make the potting pliable enough that you can actually pry it off now be careful with this because as you heat this thing up first of all understand that you're using most likely today you're using Rojas solder so Rojas solder is a new type of solder is lead free but the melting point of Rojas solder is around 700 degrees fahrenheit or so so you can heat the board up to about 250 degrees fahrenheit and it's not going to have any damage to the actual solder in there the other thing that you may find is of course when you do that heat make sure you have it well ventilated because some of these epoxies have toxic fumes that come out of them when they start heating up so definitely watch out for that there are solvents that will often do this WD-40 actually does a pretty good job against several different types of epoxies dichloromethane nitric or sulfuric acid isopropanol all of those are types of materials that can be used to dissolve potting the problem with a lot of this stuff is it may require special permits in your jurisdiction you can't just simply call up and say hey I'd like to get a gallon of sulfuric acid most of the chemical supply manufacturers take a damn view of that sort of thing and start asking a lot of questions why do you want sulfuric acid and what are you planning on doing with the sulfuric acid and when you start trying to explain to them that you're reverse engineering a piece of government equipment they then take a damn view of that and they don't ship you the they don't ship it to you but nitric acid certainly is something you can typically get your hands on sulfuric is a little bit harder this stuff that you see there called attack that is actually material it's not cheap that little leader can there is almost a hundred dollars but when you need it you need it and it'll actually do a really nice job of pretty much dissolving the material now when all else fails you go to cut and scrape cut and scrape is a dremel and a an exacto knife excuse me a precision utility knife and it's not pretty you see an example of it down there cut and scrape but unfortunately if you can't get the epoxy to melt any other way you may not have a choice and of course be careful as you're using the dremel to scrape off you know sand off layers because sometimes it produces fumes sometimes the little particulate matter if it gets into your lungs is really bad stuff so make sure you wear a mask when you're doing this kind of stuff in general any time you're messing around with epoxies and polyester resins you want to wear a mask it's really pretty much generally nasty stuff all the way around now let's assume that we have managed to get ourselves into the device now that you got it out of the case let's take a look at the device to see if you can identify the parts often times if you look at the parts some cases they actually I've had manufacturers black out the part number with a sharpie other cases and if you hit it with just the right light especially if you're in different modalities like infrared you can usually read it even though they've marked it out with a sharpie or if you I have seen them actually scrape the part numbers off to try and keep you from figuring it out and again in that case a little bit of nitric acid will usually bring the part number back up so you can figure out what it was so being able to read the part numbers is easier said than done but if you can identify the part numbers try to get a hold of the data sheets in some cases the manufacturers will be more than happy to give you the data sheet Marvell give you a data sheet or Atmel or one of those guys will give you a data sheet others not so keen on giving data sheets intel for instance anybody from intel in here I'm sorry Intel if you want to get data sheets from intel it requires your first born child and you then have this book that is basically handcuffed to you and you can't go anywhere you can't leave the book because the book gets lonely and you know you have to carry it with you all the time it really is kind of a nuisance in some of the cases for some of their parts but absolutely take advantage of teardown sites like iFixit if they've already done the teardown for you that's fantastic saves you a lot of time and effort in some cases they'll even identify some of the really weird chips you know it's a Samsung part that's not commercially available it's only available for company XYZ they'll give you a little bit of additional information about it so that's really handy if you can get a hold of that now getting that sample data sheet is important and the reason it's important is because it'll tell you capabilities of the part of the major chips that you may need to know for instance in this particular case we were doing a work we were doing a job that used an AVR it had an AT mega processor in it and the fact that we were able to take a look at the data sheets and find out that there were actually two serial ports involved in that particular part we then could figure out a little bit more about what the serial ports were doing what they were up to with them additionally they may well they may very well talk about spy flash interfaces and things of that sort to give you an idea of how the board is supposed to boot and then if it's a spy flash we can then hook on to the spy flash and try to read the flash directly but more importantly it gives you some information about voltage levels logic voltage levels we see today we don't see much in the way of 5 volt anymore it's usually 3.3 volt or even all the way down to 1.8 or 1.2 volt logic so you definitely need to do a little bit of homework there because if you try to plug in a 5 volt RS232 port into a 3.3 volt logic level port it'll toast it so definitely a bad thing data sheets may also outline a lot of information about the algorithms that the part supports if it supports things like CRCs or anything special that may be built into the hardware sometimes the manufacturers will take advantage of that and it will explain a lot of what you're seeing when you're looking at it on the oscilloscope alright so now repopulating the interfaces of course a lot of manufacturers will depopulate the debug in the interfaces so the JTAG interfaces in particular will be depopulated at a minimum sometimes you'll find the serial ports are depopulated as well JTAG interfaces often have a pretty familiar look about them so it'll be a 10 pin or a 14 pin or a 20 pin connector those kinds of things kind of stick out on the board and they're fairly easy to find also again kind of a caution here about the logic voltage levels in this particular case we put this scope on it if you use a digital signal storage oscilloscope digital storage oscilloscope we have high voltage probes that are available for that so when you put that on the circuit and you're taking a look at it you know you're not going to toast the scope and the scope is made to be able to sample the voltage level so we'll be able to see it in this particular case for this one this is actually looking at a communications channel and what you're seeing there are ones and zeros going across and also because of what it looks like if you actually could see it very well you'd see that that is 3 a little bit over 3 1 volt markings so we were looking at 3.3 volt logic in this particular case also of course signal tester like a bus pirate can oftentimes determine what kind of signal it is whether it's a serial port or a JTAG or something along those lines they actually have some pretty decent tutorials for bus pirates in similar sorts of devices and of course sigrock.org is a great site for kind of information about how to do this kind of signal analysis of course for those of us who do board ports where we're trying to port Linux or VxWorks or some other operating system to a new platform we need these kinds of things because we need to be able to see what the circuits are actually doing and whether or not our code is actually doing what it's supposed to do so definitely really handy to have this kind of stuff now other cases we'll actually see a lot of examples here's some examples of devices that have had to be repopulated that's a router up there and it turns out that they had 4 pins normally for a serial port you'll have transmit receive and ground so that's 3 pins occasionally you will find a power pin sitting next to it so again this is one of those make sure you test the pins before you start plugging anything in to know which one is the power and which one is the ground UART serials j tags here we see this particular one over here happens to be a j tag signal we have another serial port in here this is the pocket beagle if you look on the back of the pocket beagle they actually have a big interface there now this connector that you see this is referred to as a needle connector or a pogo pin connector the little pins that you see there are spring loaded which is why they call them pogo pins and what you'll do is there are a couple of holes on either side and you clip this thing in and then it stays in contact with the connector so we see this being used a lot in microcontrollers so whether it's an ARM Cortex-M3 Cortex-M4 type microcontroller often times they will have some of these pogo pins additionally we're starting to see more prevalence of serial wire debug SWD SWD is basically j tag but it only requires two pins instead of 1149.1 IEEE 1149.1 I think it's 1149.7 so they have a specification for this thing generally the serial wire debug is targeted at microcontrollers but we're starting to see it now on some of the larger ARM Cortex-A parts as well and if we had a requirement for serial wire output which is basically a print line it's a serial port that requires just one more pin so in three pins we can absolutely get basically debug interface etc now some of the manufacturers will actually blow the e-fuses so what the heck is an e-fuse an e-fuse is an electrical fuse that's on the chip itself and when they blow the e-fuse it basically breaks the connection to the debugger if they've blown the e-fuses the only way to fix those is to decap the chip and with a laser you spot weld the e-fuse back in place that clearly is not something that the typical user has at home because they basically have to cut the top off the chip and then look at it in an electron microscope figure out where the fuse is and zap it and bring the fuse back so that's not something that most people can do but nonetheless hopefully you don't get to that level now here we actually see two connectors side by side one of them is a 14 pin connector and the other is a 10 pin connector the 10 pin connector happens to be two serial ports the 14 pin connector is a J-tag you'll also notice that they all the holes are nice and neatly filled in in the wave solder so when we get ready to repopulate the connector we're going to have to get through that and actually solder the thing back in place so that can be a little tricky alright so now why bother repopulating the interfaces well of course for serial ports we want to be able to watch the boot cycle we want to know what this thing does during the boot cycle we want to know if it's using U-boot we want to know if it's using some custom home rolled boot loader of course for J-tag and serial wire debug the goal here is to be able to read the firmware out of the boot flash we want to be able to grab all of the firmware because that's going to have some interesting binary blobs in it that we probably want to take a look at and we'll show you an example of what that looks like here in a moment now once you have the firmware you can now start the reverse engineering of the boot code which boot loader is it using is there a device tree blob of course for those of you who do a lot of work with the Linux kernel you know how important it is to get that device tree blob if you don't have the blob man your life is living hell so being able to pull the device tree blob out of the firmware can save you a tremendous amount of time obviously if you're trying to upgrade the device let's say it had a 2.6 kernel on it now you want to move to a 4.14 kernel you're going to have to have the device drop blobs because they started using device blobs in about the 3.10 3.14 time frame so in order to move up to a significantly newer kernel you're going to have to figure that out for those of you don't know what the device tree blob is it's basically a way of describing the motherboard and the components that are on the motherboard so you'll know exactly there's two serial ports located at this address there is umptiumpa megabytes worth of memory located at this address there's an I squared C there's a spy bus all that sort of business GPIOs all of that will be laid out in the device tree blob so if you can get a hold of the device tree blob that's a huge win and of course which OS is it using it may not necessarily be using Linux it could be using VxWorks or FreeRTOS or some other flavor out there and if you find out that it's using something like Linux then which version of Linux is it there's been significant changes between the 2.6 kernel and the 4.4 kernel so as a consequence knowing which version of the kernel it was running is a huge thing to be able to take into account now another thing is of course being able to repopulate the interfaces and use a J tag to read the flash and get yourself a hold of the binary that's all very sexy and wow that's incredible you were able to do that but sometimes it's not that hard sometimes you just go to the manufacturer's website and they will have updated firmware or even old firmware if it's an older device they'll have the firmware sitting out there and you can download it and when you download the firmware well okay now I just save myself all the trouble of repopulating the J tags and all that sort of business but I have the firmware if you do have a case for firmware update then download it take a look see what's in it and we'll show you some tools that help you do some dissecting of that here in a moment but depending on the vendor you might just be able to download the update directly occasionally they will require you to actually do the update through the device so if they're going to do it through the device then make sure you've got your wire shark turned on so you can capture all the packets that go through and reconstitute the actual firmware image from the packets that got transferred in order to download it to the device a little bit trickier to do that but certainly if you are dedicated it's not that hard alright so now that I have the image here we have an example of this little guy right here this is an action tech MI424 this one happens to be I think this is a revf or something maybe but it's one of them that I had laying around the house because Verizon forced me to upgrade and therefore I had this extra router sitting there and I might as well do something with it and when we took a look at the firmware we actually found that action tech does not supply the firmware because it was developed specifically for Verizon and as a consequence we had to go poking around in other places to try and find a copy of the firmware we finally did it actually only took us about an hour or so to research and find out where the firmware was and then we downloaded the firmware having the firmware now you'll notice that this is in a weird format a .RMT format which is some sort of remote format it's a strange format doesn't correspond to anything it's something that's proprietary but that's okay because it turns out that our good friend is familiar with Benwalk Benwalk is a great tool for looking at binaries and kind of walking through the binary to try and find out what's in it we took a look at Benwalk and sure enough there's a U-boot header out there which identified the type of processor it was and exactly what kind of kernel it was running then we saw a GZIP compressed data segment which is probably going to be something like CramFS or some other sort of compressed file system and then we found that there was a wide area network firmware interface sitting out there that if we were really interested in going a little bit further in there we would have dug into that Landcom firmware interface to try and figure out what was in there but we'll see that we didn't really have to get that far in order to be able to do some interesting stuff here now we know where the edges are of the different images now we can use a tool like DD to simply extract the pieces out so here's an example of separating the pieces out of that firmware we went out and the first part of it of course was U-boot so our block size is one and we told it to skip the first 163 bytes which is what this told us to do in the decimal here 163 so we skipped the first 163 bytes and then basically rode out until we got to the next block that incorporated all of the U-boot stuff did the exact same thing for the compressed file system and of course the Landcom firmware just for completeness so now we have all three of the pieces that are laid out now let's start poking around inside there because it identified itself through Benwalk as a GZIP compressed image we simply extracted out and then G-unzip it and see what's in it and guess what once we did the unzip and then we did a Benwalk on that we found GZIP compressed data we found two KramFS file systems a whole bunch of security certificates we found CRC polynomials for doing the checks we found that it was a Linux 216 16.14 release of the kernel when that was made that was dated back to 2014 and we also saw some of the paths for the actual files inside of it we cut it off because there was a whole bunch of stuff in there that we couldn't make fit on the screen even that was a little hard to get fit but you know that was a really fortunate find where you managed to simply by grabbing the whole of the firmware doing a little bit of poking around in it figure out that it was a GZIP environment unzip it and then run Benwalk on it again and you find all the good stuff that's inside of it so now what's the general approach to reverse engineer a binary well we got really lucky with the MW to a 424 part that firmware was just fantastic because of the way they laid it out most firmware is not so forgiving now you will find that some of the firmware from manufacturers if you can download the firmware it may be encrypted if it's encrypted then that represents a much more difficult problem sometimes it's just compressed and often times you know it's one of these security through obscurity kind of things they compress it and of course they don't figure that anybody can figure out that it's compressed and therefore they don't know how to decompress it but when we take a look at it we run Benwalk and we have it print out the entropy now the entropy if you have an entropy close to one that means it's typically either compressed or encrypted and when you see entropy like that you go but at least it didn't start at one there was some little piece ahead of it that was not one so that gives you some hope that there's some way of being able to figure out what's inside of it then another tool we'll use is the strings command once you've compressed it and you run strings across it you'll find all kinds of stuff in there like this thing here start section rg hardware ferocion distribution the vendor Verizon the product version etc etc that was all in there just through looking at the strings the question is how often do we find the manufacturer strip the strings out if the manufacturer is doing the due diligence the way they should they should have stripped it sometimes they will sometimes they won't and it's just catch is catch can you may get lucky and if you get lucky take it if you're not lucky and you get nothing but gobbled a gook out of the strings command it's like well okay we tried that and that didn't pan out for us so let's move on to the next thing now the next step once you've got it separated out a bit then we can take another look at the entropy after decompressing that image we see a spike right there which is the separation between the u-boot image and the rest of the cram fs these interpes close to one and those are the cram fs images and then we see the rest of the operating system setting out here so again if it's not solid at one that's good if it's solid at one that's a bad thing that means they've either encrypted it or they've got something else sitting out there that makes it look like encrypted data once we of course in this particular case we did the bend walk on it we found out there were two cram fs images in the front we know what their offsets are because we know what their offsets are we can simply mount them and take a look at them that's a really nice thing about the way the loop back interface works in linux if we can identify what the file system is we can simply loop back mount it and then take a look inside cram fs is compressed but not the headers so the data is compressed but the headers are visible so you can then look at the names of the files inside of cram fs when you go to squash fs it's a little bit different now we'll see a lot more squash fs being used a little bit harder to do the dereference on but not impossible but in this case we were lucky we got cram fs and it was easy to go ahead and mount it now normally we would then dig a little bit further into that go ahead and mount them and try to extract things we'll do that in some other time we just don't have time to mess around with it here in this class so now let's take a closer look at the binaries let's assume we've managed to be able to get the binary out of the image what can we do with it well first of all there are bin utils like obj copy and obj dump which will allow us to do a disassemble of the data if you look at obj copy excuse me obj dump obj dump has the dash s option that will allow me to disassemble the data so this is really useful if I know the manufacturer if I know it's a mips or an arm or a power pc whatever then I get the obj dump for power pc and I do an obj dump dash s against that image and it will then disassemble the image for me understand that disassembling the image is useful but in some cases not terribly enlightening unless you speak the assembly language of the native processor if you speak the assembly language of the native processor then you've got all kinds of stuff there also understand that when you run into something like that and you've disassembled it there's not going to be any nice neat labels there's not going to be basically the object code is stripped so you will find a jump instruction to some random offset or some offset from where it is right now a relative offset rather so when you're in that situation you basically have to try and create a map of the executable from the assembly language now this is definitely a long poll to try and go through avoid it absolutely try to this is why we love open source it is a wonderful thing to be able to say well I need to get the open source from you and then you can do a mapping between the open source output and the actual executable obviously when you're dealing with open source you can get a hold of the compilers GCC is often the compiler that's used simply because it's free and the manufacturers don't want to have to spend empty up thousand dollars for the development environment so it works out really well that way and everything just comes out as normal ELF with GCC as the primary compiler alright so now let's assume that I have my executable and I've taken a look at it and go okay this looks alright now what can I do well I can run it if I happen to have QMU for instance for that particular processor architecture then I can actually put it inside of a QMU session and try to execute the code using tools like S Trace and L Trace we can get a pretty good idea of what the code is doing, what functions it's calling up in the kernel, what libraries it's trying to call as it runs there are some disassemblers there's the Ressie project this one is a fairly nice disassembler, does a great job for x8664 and it tries to figure out exactly what the jumps are to help you put it back together again there are also professional tool chains and disassemblers like Ida Pro now Ida Pro is the hardcore reverse engineers friend Ida Pro is not cheap it's like six or seven thousand dollars for a copy of Ida Pro per CPU architecture so it's not cheap but it is awesome what will happen you get the assembly language output over here it will construct a calling graph for you and allow you to label things so if you look at a particular piece of assembly language code you can clearly tell that it's a print statement you can actually label it as print so when you're creating the calling sequence here you can actually try to figure out what the calling sequences were what code is calling what the inner relationships between the individual function calls inside of it understand this code was probably C code or C++ code at some point and now we're looking at the assembly language so a lot of the things that would have been control structures inside of the C have to be mapped into assembly language control structures we can also see the actual data itself down here and in some cases we're looking for very specific byte patterns that indicate the end of a jpeg image for instance if they happen to have a boot logo or a splash screen in the unit and you want to change the splash screen to make it really cool well you want to find out where the splash screen starts and where it stops and figure out how much space you have and whether or not you can actually go out there and reprogram it so all of that sort of stuff comes into play with IDA Pro as I say it's not a cheap although you can get I think a 30 day evaluation of IDA Pro if you want to play with it but they have training classes that teach you how to use IDA Pro because it is a complex enough tool set that you really need to take the training class this is used often times by larger name government agencies to disassemble malware and figure out what the malware actually did or what it's supposed to do always use protection that's a general statement in life but nonetheless never run a foreign binary on your test platform without taking some significant precautions first of all QM use a good start make sure you keep it bottled up and of course there is support for most of the common CPU varieties out there especially MIPS PowerPC, ARM and x86 or you could use a VM if you've got KVM certainly use that to keep the application bottled up add a minimum use a charoute LXC type environment LXC rather than Docker simply because LXC looks like a real operating system to the application as it runs Docker tends to be so stripped down that the application says well I'm missing this I don't have this library blah blah blah blah I can't run well okay if you're running LXC chances are you can actually trick it into thinking it's running on a real operating system yeah oh yeah yeah yeah it turns out that some malware will look at certain registers to find out if they're there and Apple calls that the UEFI BIOS I'll probably get in trouble for that cut that out but they actually look for special registers in places and if they don't find them then they say oh I must be running under VM and therefore I don't run so yeah there is certainly legitimate pieces of code as well as malware that will look around inside the environment that is running and determine whether or not it's running on real hardware in that case you have to go in and dork around with your QMU register sets to make it think that it looks like real hardware sometimes easier said than done of course capture the run with S trace and L trace we want to see what it's doing we want to see what kernel functions it's calling we want to see what libraries it's accessing note any anomalous behavior it could be a legitimate application and because of the fact that it's the third Tuesday of the month it does something weird on the third Tuesday of the month they call that Microsoft patch Tuesday and now cut that out to you I'm going to get in real trouble here alternatively you can transfer the application to a small platform let's say it's arm then we can take a beagle bone or a raspberry pi and we can transfer the application to that platform and try to run it there if something goes wrong pull the plug simple enough and you can always format the SD card and start over again but if I'm running malware I probably want to destroy the SD card and get another one I don't want to take any chance that there's something left over even though you reformat sometimes the bad block list and everything doesn't quite work the way you think it should and it's also I've seen malware that actually manipulates the bad block list so definitely be careful about those kinds of things with SD cards in general don't run with internet protocol enabled don't have it connected to the internet pull the ethernet cable or go into IP tables and block it off there block all the outbound traffic until you have a warm fuzzy feeling about what it's doing again putting it inside of a kind of a sandbox and watching it go back and forth it turns out that in that case I was talking about with the major manufacturer and the bios chirping back out to China that was no operating system running that was just sitting at the bios and it chirped out so they call that beaconing and this is a characteristic of a lot of malware it'll beacon back home let them know hey I'm here this is my IP address do you have anything for me to do and the controller will say go to sleep and call me in a week or a month or six months and then it will wake up again go back out check to see if there's anything new to do and then things will do some really interesting stuff this is what you call intelligence surveillance and reconnaissance operations with ISR operations we're interested in looking to create a map of your network and creating a map of the network is really valuable because it tells me that you have printer X and router Y and devices made by manufacturer such and such and by that information it helps me understand what sorts of attacks might be viable against that so understand that the malware writers are trying to do that against you in which case you definitely need to be careful with that kind of stuff now we'll show you real quick here the example reverse engineer of a protocol in some cases it's not the device that we're really interested in we're interested in the communications protocol that the device uses ethernet cabling of course just because it's got an ethernet cable plug on it doesn't necessarily mean it's ethernet we see ethernet cabling those 8-pin RJ45s get repurposed they're repurposed for serial ports they're repurposed for all kinds of stuff and it may not be standard 802.11 I mean 802.23 802.3.2 it may not be that it may be ethercat or it may be some other industrial protocol which happens to go across ethernet cables so when we're trying to understand exactly what that's doing it's great to be able to kind of sniff the protocol and see what's up serial protocols themselves are also significantly difficult to reverse engineer some of it is just because we're dealing with ancient technology and very few folks still remember what a serial port is or how to hook it up a lot of the deals with serial ports today it's just a USB connector you just plug in the USB connector and it magically works but in a lot of this older equipment we actually have you know data terminal ready data request to send clear to send kind of signals that we need to understand exactly how they work in order to enable the device otherwise it won't work now the use of logic analyzers with protocol decoders is your savior here the big mumbo daddy of all protocol decoders is a device from Agilent it is a decoder that understands how to do like E2, E3 signals T1 signals you know telephony signaling mechanisms and they're expensive I mean they're you know $20,000 $30,000 for one of those things so if you can use some of the low end logic analyzers like the salie units to help you understand what's going on that's a good thing saves you a lot of money now obviously again we can't emphasize strongly enough check your voltages this one happens to be the device that we were getting ready to do the RE on and what voltage does it show 9.6 volts now why was it running at 9.6 volts turns out it had a pack of AA batteries that were running it and that's the raw voltage off the AA's and then it would go through various buck transformers and step downs and filters and everything to get it down to the 3.3 volt that the actual circuit used but that meant that you had 9.6 volts floating around in the system and you had to be very careful about what you plugged into because that will definitely toast a serial port in a heartbeat now again remember voltometer may not necessarily do what you need it to do if the voltage is very quickly you may not be able to see the spike in the voltage in which case we have to drop back and take a look at it through a digital storage oscilloscope or something similar now once we have it hooked up and we think we've got it at least the voltage that we're trying to figure out then try to see if you can get the device to send any data and watch what happens does the voltage change significantly or does something else weird happen we'll see we'll talk about this particular one where something else really weird happened alright so let's capture some data this happens to be the output from the sale unit this was the transmit side this is the receive side so they're not lined up exactly like that you had the transmit side and then you had the receive side it was a half duplex connection the bit width when you actually set on and measured these bit widths you found that it was roughly 56 kilobits but not exactly 56 kilobits it was 59 kilobits in some cases so some of that has to do with they're not using a good solid crystal reference sometimes if they're doing phase lock loops and they're trying to generate the voltage I mean they're trying to generate the clock using a phase lock loop sometimes the phase lock hasn't really happened yet and therefore it will be a little wonky in terms of the frequency coming out of it but it was roughly 56 kilobits and there was some drift in it but that was something that we could easily take into account in the sale unit now when you're trying to do a protocol decode if your logic analyzer supports multiple protocols try and switch you've captured the data just simply switch the interpretation if you look at it as CAN bus, if you look at it as I2C if you look at it as normal RS232 async those kinds of things can be really helpful just to be able to see it how many pins are involved with the signal is it something that I can see in just two pins in which case it's probably an RS232 but does one of the pins look like a clock when you hook it up to the sale unit you're going to see this thing going that's probably I2C so when you start doing interpretation for I2C you'll be able to decode the master and slave interfaces you'll figure out what the I2C ID is and little tools like Arduino's are wonderful at being able to do decode on this kind of stuff generate the actual I2C clocks, the spy clocks and then be able to get the device to talk to you even though it's maybe kind of isolated from the rest of the circuitry of course when we were taking a look at that particular protocol there was some really strange things about the protocol if you take a look here you'll notice that the let's see if I can get this to show up over here whoops wrong way you'll notice that there was no voltage over in this section originally the voltage went high for about 2,000 milliseconds so about 2 seconds and then it started clocking data now this looks a lot like something called a mark after break so when you're looking at mark after break protocols there are certain types of protocols that use this particular technique so we were originally thinking that it may be a mark after break protocol then as we got a little bit further into it it turned out that it didn't match any of the known mark after break protocols and when we took a look at the data sheet it turned out that this particular part didn't support mark after break communications protocols so that helped us a lot by being able to rule out what we knew it wasn't we didn't know what it was but at least we knew it wasn't mark after break now this and taking a look at it because of that weird it was off and then suddenly it went high and then it started clocking data it looked a lot like it might be bit-banged and when you're bit-banging a port like that that would also explain a lot of where we were getting some of the drift in the clock frequency if you're bit-banging a port you might be off a little bit in which case it might look like 56k plus or minus a couple of k bits per second and so it looked like maybe this is bit-banged okay well this is going to be interesting to try and figure out what this protocol does fortunately there was some weird things about the protocol that told us that it was not quite bit-banged it turned out that of course the voltages started at zero and because the voltages started at zero we looked at the part and we found out that the part had two serial ports on it so we suspected they were using one of the serial ports for the communications channel but serial ports when you enable serial ports there's a voltage that's constantly available on either the transmitter or receive pin depending on whether it's connected as DTE and in taking a look at that we go okay well that's really weird because there's no voltage at all for extended periods of time and then when we trick it to do something the voltage suddenly comes on and it starts clocking we took a closer look at that we found out that it was although it looks like a break signal that high signal that would come out it really wasn't mark after break and then we have it meant that basically if we wanted to inject a new command we're going to have to figure out what the pattern is and follow the pattern so the goal for this particular case was to be able to get into the communications protocol and start injecting commands that in some cases might confuse the device in other cases might do something special in the device that we can't really get into the details of. Alright so we used a microcontroller to do a lot of this testing turns out in this particular case it was a CC3200 from Texas Instruments the nice thing about this one is it had the right voltage level it was all 3.3 volt so we didn't have to do any level shifting we didn't have to get any special circuitry out there to handle 3.3 to 5 volt level shifts also this gave us Wi-Fi and JTAG on the embedded board so we could actually have some options for interfacing to the micro and then kind of watch what's going on we did opt in this case to use the Arduino like Energia or Energia depending on how you pronounce it I guess this looks like Arduino code and it sits on top of TIR toss so we can get it to do fast things when we need them to be fast but not have to sit there in eclipse and our minds go numb as we look at the eclipse code to basically boot the board and do all the rest of the stuff with it so Energia was a great solution for this particular case now in the first try we said okay well we think we know what we need to do we need to inject some serial port like stuff we knew we wanted to isolate our circuitry from their circuitry so they would not know that we were there so we basically made our equipment look like a diode well we used a diode to keep it from being able to be detected by the devices that were communicating with each other and it turned out that if we didn't do that or if we used a diode but then didn't do any additional isolation we ended up with this weird thing you'll notice those are not nice square edges that's bad that says that we would sometimes get some things through but other things wouldn't come through and it was all mystery meat until we put the scope on it and then once we saw the scope we go oh that explains it also weird thing the voltage levels here instead of 3.3V this was 1.8V so it was a little strange that we would see some big change in the voltage there so the fact that the voltage got cut in half the edges looked horrible this is not a good thing we have to figure out some other way so we went to a high speed shot key diode and the high speed shot key diode can handle the frequencies so that seemed to fix the problem for us again because it's a diode we basically isolated ourselves from the signal they couldn't tell we were there we could inject commands into the system and see what happens we also needed to power the microcontroller so what we wanted to do this we wanted to have this a box that would basically just plug in line with the thing that we were doing the testing on just to make things easy that way we could upgrade the system in a matter of like 20 seconds and put our box in the middle and then start looking at signals going back and forth we needed to power the micro and to keep the signal at the same reference ground this is a tricky thing we definitely want to make sure that the ground being referenced in the controller the real controller is the same ground we're being referencing in our circuitry otherwise we get ground loop problems and things just go to hell in a hand basket so what we did was remember that 9.6 volt power we pulled that 9.6 volt power ran it through a buck transformer that's that little white thing you see down there and converted it to 5 volts to power the microcontroller so if you powered the microcontroller through a 5 volt USB connector if you powered it with 3.3 volt it didn't work at least not for our application if you powered it with 5 volt through the USB connector everything worked fine we basically took the 9.6 volts we pulled the power off the connector we ran it into this power supply we then ran the power supply underneath into a USB adapter you can get from spark fun they're great a little PCB that has a USB connector on one side and just through hole solder pins on the other and so we ran it into that and it turns out that these let's see whoops wrong button these pins that you see here that's actually the pins of the cc3200 sticking up through this PCB through this perf board and when we did all of that it was amazing we could actually inject new commands and watch what happens you know as we were going through doing the decode on the protocol we actually found out that there was some places where it was representing 100% so you would see 100% on one thing or 100% on the other thing and then the mix of 100% so you would end up with 100% if you added the two of them together those were both being represented in two bytes for each one of those values also now this is the thing we were talking about earlier where you did a little research on the manufacturer and the people who held the patents turns out that the people who held this particular patent were both ham radio operators and we were looking at the protocol and there was a weird thing at the end of the protocol that was a 16 bit value that changed drastically depending on what the input data was and so we said well this is probably a 16 bit CRC and the problem is 16 bit CRCs there's 24 different 16 bit CRCs in this particular case because the two people who hold the patent were ham radio operators we said ah AX25 which is a 16 bit CRC that's used in ham radio protocols let's try that sure enough that's what it was so I mean as a ham radio operator I go ah you know I bet you they're using AX25 let's try that and sure enough that's what they were using and when we went back into the data sheets and actually dug down through the data sheets for the at mail part we found that AX25 was one of the modes that it supported so it's like ah cool so this is I mean this is esoteric detective work but this is the kind of thing that you have to do when you're trying to be serious about reverse engineering something now but with this particular solution we were able to inject commands complete with the CRCs and lo and behold the controller and the device it was controlling was unaware that we were there and we were able to do our particular job in this case inject commands and make things do light up and do funny things now reverse engineering kind of a summary here reverse engineering is an incredibly challenging problem we have found some weird correlations and that is people who are really talented at reverse engineering and also like to pick locks I'm not sure why but it's just a correlation and some of the people who make malware they will have like sets of locks sitting on their table and that's what they'll do for you know a break a mental break they'll sit there and try and pick the lock I don't know why but it just happens to be a strange correlation there it's a problem solving and if you like solving really really tough problems and you're familiar enough with the technology the hardware underneath it that you can then try to figure out okay so this is a serial port but it's not quite a serial port you know why is it doing this then you'll be successful make sure you gather your tools the hardware tools from places like iFixit again the right to repair movement is a fabulous thing because being able to source suction cups and pentalobe screwdrivers is a major pain in the butt but thanks to iFixit we can just simply buy it as a kit yeah it's a couple hundred bucks for the kit but it's worth every penny understand what your goals are and when to declare victory is your goal just simply to figure out oh this is a Linux machine I had my wife got upset with me with this one we had a Samsung television and there was a firmware update for the Samsung television and remarkably enough when you did the firmware update you saw things go across on the screen like u-boot and you go oh okay we're running u-boot on this platform and there was a maintenance connector on the back that happened to be a serial port of course the wife took a dim view to our TV set stop that hands off I'm watching that but it's definitely a cool thing when you understand the technology behind a lot of what's being built today and certainly here at the embedded Linux conference many of the devices that are being constructed are in fact using Linux anything that is an internet of things kind of device it's got to talk to the internet somewhere and if it's got to have a protocol stack protocol stacks you can get them for microcontrollers they're expensive but if you can get a free one for Linux okay it may be worth it to pay a little extra more money on the processor side and not have to worry about all this software that you'd have to write in order to handle repos and firmware updates and dealing with you know certificates and all that sort of business so the other thing that I would say is understand the legal implications of what you're doing as I say in some jurisdictions just taking the firmware off the platform using a spy flash reader is illegal definitely check that before you start going down this path for most of us I mean we think well I bought a PS3 and because I bought a PS3 it's my PS3 and I can do anything I want to use on my PS3 I can do that no the DCMA says if I go in and change the boot sequence the boot firmware to boot Linux instead of booting the PS3 operating system I violated the DCMA and you know it's like well I own it no no you don't if you read the fine print you really don't own it you have a unlimited lease on the device so always make sure you read the fine print now because I like to live dangerously what we're going to do here is I'm going to see if I just basically I re-populated one of the serial ports on this board and we're going to try it for the first time and of course again I didn't sacrifice a large amount to the demo gods this morning so I'm not sure exactly what it's going to do but we're going to try it anyway so what we'll do is because I've re-populated the serial port and I went through and I did measure some voltages to kind of know what sort of things to expect we'll run Minicom on this and we will set it up for 115.2 which is usually the speed of these guys work it'll be either 115.2 or 9600 those are the two that they tend to go to now the one that we did the protocol RE on that was 56 kilobits that's a little unusual 38.4 is even more unusual but again just by looking at the signal we can figure out what the bit width is and then calculate from the bit width we can figure out what the board rate is but let's assume that we have here 115.2 running in 8.1 that's the other problem that we run into a serial in most cases it'll be no parity, 8 data bits, 1 stop bit but that's not guaranteed it could be 7 data bits 1 stop bit 7 data bits, 1 stop bit even parity or odd parity odd parity turned out to be one that they used a lot in older equipment so again kind of look at the signal it's coming across the scope and see if you can figure out whether you've got start bits and stop bits and how many of those things you have so let's assume that we're running here and I'm going to go ahead and power it up and see what happens now with any luck let's see what we get here oh! yippie-skippy so we've got u-boot sitting here and we can do a print env and there's all the u-boot settings so we now know, based on this version of u-boot, that there is no binary blob no device tree structure because of the way the u-boot is put together and we can then tell it to let's just go ahead and live dangerously let's reset it so it's uncompressing so this is coming off of those two CRAM FS protocols here let me blow it up a little bit here Ethernet's running at 125 megahertz Wi-Fi interfaces let's see this one actually it turns out that this is a MIPS processor this one happens to be using a cabium processor and it's bringing up the Ethernet at this point let's see if we can get it to ah! there's a user prompt username and password I haven't gotten to the point where I've figured out what the username and password is yet any questions? is anyone still awake? yes unfortunately I cannot say yeah both under NDA and classification issues I can't say but it was really tricky it was like you implied there was some real time encryption happening and some decryption that was going on you had to basically differential power analysis has anybody ever seen differential power analysis? so with differential power analysis we can basically monitor the power rails and it turns out it takes more power to decrypt a 1 than it does to decrypt a 0 so if you're watching it you can actually pull 1024 bit RSA keys off of the device without too much trouble now, downside is you have to have physical access to the device and stand off a couple miles and figure it out I have done things where for instance back in the day of CRTs you used to be able to reproduce with a good receiver I could reproduce everything that was on the CRT from at least from almost a mile away and so I could watch exactly what you were doing what you were typing all that sort of stuff on my screen it was an oscilloscope and it was all in green typing and when you entered password I could see what that was that was back in my day when I worked for some large name government agencies doing tempest which was a way of being able to look at emanations and try to figure out what was going on out there other questions yeah oh I don't answer your questions I'm sorry no no ok a little bit have not had once you pay the license it's kind of like what you got and you know what you got that's another possibility you know there are a lot of additional tools I mean I developed some really nice disassemblers in the past and tools for doing analysis of code we did have a large manufacturer printers ask us to reproduce a piece of code that they had done it was originally written in Italy there were like 60 people who worked on the code for three years to develop this product and then that division went under and they said can you reproduce the product based on what we have here we have the source code the source code was 17,000 lines of source code and they were actually what it was doing is it was taking a scanned image converting it into a pdf and then emailing it to somebody and it turned out that they were instantiating every line every pixel as another object this was like and so you can tell the difference between Paolo's code and Giuseppe's code because they all had their own particular coding styles and we took a look at it I have some tools that do reverse engineering of large code bases like this and it broke all of them it was horrible but we said you know what we think we know what you want us to do let us try and we were able to reproduce it with open source code two people six weeks and we had 95% of the functionality working and we knew how to solve the other 5% and they said that's enough we've seen enough oh by the time we finished 50 lines of code that was it total 50 lines of code everything else was just pipe this to this and do this thing and it was just a shell script and we could reproduce this thing and the 50 lines of code was some java that somebody wanted because they wanted to have a display okay we'll give you a display sure success what do you want and that was the display that they wanted to have success question that's a good point often times I should have mentioned this you always get two of whatever device you're going to have I guess it seems obvious because the first one you're probably going to destroy and then once you learn anything you need to learn then you do the second one but yeah you always want to make sure that at least two of them fall off the truck so that you can figure out what it is question given enough time and money anything is possible now the question is how much time and money is it going to be understand that when you're dealing with typical black hats that are going to be trying to take things apart their motivation is primarily money and if you make it sufficiently hard they will give up and go on to a softer target now at what point is it sufficiently hard well that's up for debate in the case of state sponsored hackers they effectively have no budget and they have the best and brightest that they have been able to find and there are some really clever people that are in that business and if you're going against state sponsored they will own you eventually it may take a little while but if they really want it they'll get it absolutely you want to know who's trying to hack your system and unfortunately today a lot of companies don't spend enough time doing threat modeling when you're trying to develop secure code and that's the other thing that we do we help people understand what it means to develop secure code and so when you're in that particular business you're trying to explain to them that the security engineering can cost at least as much as the total product if not more if you really want to lock it down so now your threat modeling becomes a factor we're talking about risk mitigation here what risks do I want to accept what risks do I want to transfer to a subcontractor so it'll be their problem I'm talking to you Toshiba I mean Toyota so you're going to transfer some of that you're going to accept some of the risk you're going to try and mitigate the risk by adding additional security measures in place obviously when we're using things like TPMs and secure data stores that makes things a little bit harder obviously and then of course using TPMs we need to understand which type of TPM because the Chinese have their own TPM that have their own encryption algorithms in them they do the standard TPM 1.2 stuff but they also do something else and it turns out that a lot of machines a lot of laptops and things that come out of mainland China have the SX-88 has the SX-88 chips in them and that's a different kind of TPM and then you wonder okay is there a back door in there but they the reason they created that chip is because they were afraid that TPMs from Infineon and all these other places also had back doors in them so I figure any other questions? Occasionally what we find are some really interesting things they call them doorknock protocols so basically you hit this port this port in this order and it unlocks something doorknock protocols are really interesting to see them implemented one of the easiest ways to avoid that kind of problem as you're trying to engineer a secure solution to the peer review I mean it sounds stupid but having more than one person look at it go what the hell is that why are you doing that is a great thing I mean you take a look at the numbers between proprietary code versus open source code proprietary code tends to have nine times more errors in it than open source code does simply because if nothing else it's because of the peer review now that does not mean that open source code is impervious that was a smart bleed that was a big one and it had been in there and everybody was using SSL for five years before anybody ever found it it had been peer reviewed but they just didn't happen to see that mostly greed understand that when you're trying to build a new device that nobody's built before or something that's supposed to be better than anyone else has built before the market is either number one, number two or nobody knows who you are we can easily say okay who's the number one bookseller on the internet Amazon who's number two Barnes & Noble, good who's number three maybe once you get to number three nobody knows you you might as well not even bother so the problem here is the problem to market is everything if I've got this really cool thing and I need to get it out of the market I need to get it out there before anybody else does we take shortcuts we don't do the due diligence we don't do the security engineering understand what's the mindset today the mindset today is let's say you just bought your brand new iPhone X what's the first thing you do plug it in and then it yeah the garbage but I use Sony Tim Berg would actually be happy about that but in any case the issue here is the very first thing you do is you plug it in and you update the firmware so manufacturers know that the very first thing you're supposed to do is update the firmware so they ship it with firmware that has security problems all kinds of stuff just to get it into the distribution channel once it's in the distribution channel they figure the first thing you're going to do when you get it home you're going to plug it in you're going to update it so that gives them an extra six months anywhere from three to six months in the distribution channel before they have to worry about anybody actually using this really buggy software so that gives them an opportunity to fix it and then preposition the update so that as soon as you update your phone everything's good now that doesn't keep you from jail breaking the iPhone within 24 hours of them coming out with a new firmware but that's a different problem but that's the kind of thing and that's the mindset that a lot of manufacturers are in these days so I had an OLED television set from LG and the first thing you do is you update the firmware when you plug the TV in really? I mean most people wouldn't think to plug in an appliance into the internet it's firmware now I always am suspicious about things like that so I kind of turn on Wireshark and I watch what it does but that's a different story anything else? Yeah question yeah there are going to be cases like that I mean for instance the third peoples liberation army which used to be the name they've changed the name now because everybody knows who the 3PLA is but they had a captain that had been working at Seagate in Malaysia at one of their plants and every hardware that got shipped out had something special on the hard drive they found him, they caught him he pulled out his black passport and they deported him but now we don't know how many of those hard drives got out before they found him so those kinds of things that's kind of a one off I mean it's a very specialized system you're not going to see it being done very often and so fortunately that's been so long ago now that most of those hard drives are already dead but it's still the possibility you find a box an old box not very common because it's always what's the payback how quickly can I achieve the goal that I'm trying to achieve and if I got a custom create some thing to reverse engineer this stuff it's just like is it economically feasible? if somebody's paying me then absolutely I'll do whatever you tell me to do not that I'm a bad guy or anything but the reality is if you're paid to do reverse engineering that's what you're paid to do and there are a lot of companies out there that do reverse engineering now obviously you're trying to do the quickest solution because nobody wants to hear that nobody wants to figure out how this protocol works especially with encryption encryption is really really tough encryption is not the end all be all because it all becomes an issue of whether or not you can retrieve the keys data in use is a real interesting problem turns out that what we see in data centers a lot of times as you start migrating VMs from one machine to another they take all their memory and they write it off to disk so passwords keys things like that that happen to be sitting in memory are now on disk and it makes it real easy to find them not necessarily not necessarily and this is again to get it out quick they did a kind of crappy job of doing their security engineering so you find weird stuff like that we have a cyber security class we teach a cyber security class and in that class we actually do this we run a VM we snapshot it and then you look in the VM and you can actually find the password yeah reverse engineering is almost always done as a as kind of a temporary blog sort of thing and the sites will pop up and then disappear as people start getting angry with them you know I've done that to my machine I'm going to crush you so unfortunately there are some books that are out right now on reverse engineering recently came out on reverse engineering Linux binaries definitely an interesting one you can pick it up on Kindle actually I just got it myself I'm still going through it at this point I forget right off hand send me an email I'll tell you who it is okay we're out of time I appreciate it, thank you very much