 No, I thank you and and there's a reason for that. I'm gonna show you in a few slides. Why? First a question how many Printers do you think HP makes a year? Wait another question. Are there any HP folks in the audience? If you're here, you can't answer but for the rest of the people how many printers do you think HP ships a year? Shout it out a Billion okay, that's awesome Okay, so It turned out it was very difficult to get this figure from a technical person at HP But if you just follow the money Wall Street knows exactly how many printers Everyone ships out and if you look at the precision 879 I mean down to the single printer. So this is a how many units? HP shipped second quarter of 2010 If you multiply that out, that's about 40 million printers per year If you multiply that out, you know since 2005, right? We're talking about hundreds of millions of these laser jet printers This slide's answered answers why I chose HP printers for my project because they are 41% of the global market as far as printing goes right so they're the biggest player And they're everywhere So I like to read an excerpt I found After I finished my project. I was looking through HP's security solutions white paper from 2006 a Concerned citizen asks our current HP multifunction printers susceptible to worms and viruses and HP says no silly Because most of the viruses and worms on the internet attack Windows-based operating systems and because HHP MFPs use non-standard operating system Okay, these worms and viruses are not going to affect our products And as if this person didn't really believe that first answer you wrote a second question follow up. Are you sure right? Does this mean that HP MFPs are completely safe from worms and viruses and HP says well, no, okay, but Hackers are more likely to be interested in exploiting vulnerabilities and workstations and servers Since they are more widespread and require less expertise Okay, so this is really my first slide and I have about a hundred slides after this that's going to make this first slide really funny Before I do that I would like to give a big thanks to Jethin Kaderia my advisor self-stofo and John Boris who helped a lot with this project and made this presentation possible a Little bit of background history. I finished most of the technical portion of this project at the end of August or no October right and You know, I showed the vulnerability I showed the impact of the vulnerability to a few folks in Columbia and we actually sat down and realized that the impact was a little bit More dangerous than I even thought you know when I first started the project So we agreed that it wouldn't be responsible to come to a place like this and give out all the technical details About the exploit without doing two things right so first Contacting HP and getting HP's attention and second really contacting HP and really getting their attention So we did one of two or we did two things we first You know got ahold of a few good people in the HP security group and you know We got to work explaining the vulnerability to them But we also reached out to a resource in MSNBC who came to the lab You know, I showed him my demo I explained the impact of this vulnerability Right, I didn't tell him really any technical details So I couldn't count that as a disclosure of the vulnerability But you know, he got enough of what was possible and he eventually went home and wrote a newspaper article and This was the headline right millions of printers open to devastating hack attack. I was really excited You know, I wanted to read this paper read this article when it came out So I set an alarm for 7 30 in the morning, right? I got up. I read this article You know the title was a little pizzazzi, but you know the article actually hit most of the points that I wanted to to get people to understand More or less satisfied. I go to sleep, right? I don't wake up till 12 30 and I look at my phone and my phone's blown up just from random people You know messaging me saying OMG on you're on Gawker, right? I think oh this can't be good So from 7 30 to about like one in the afternoon this story turn into this story Okay, could hackers And okay, this is Gawker you would expect something like this, but flaming deathbomb is a little bit of a hyperbole And I found this one of that I liked because it was it starts out really reasonable You know, it says can hackers really use HP printers to steal your identity thinking. Yes This is what I'm talking about somebody's got it and then it goes to and blow up your house Okay, so that's that's how the first day ended it was it was a headache You know, I'm just thinking like we're missing the point here and when you see the actual vulnerability you realize Not only that fire is not possible and I'm gonna tell you exactly why but if you had Perfect control of this printer the last thing you would want to do is destroy it, right? It's a very valuable asset For you now that was the first day. Here's the second day Okay, and this this is when security gets really exciting. I had no idea that I could just There was a lot of smacking and spanking and oh, I'm not gonna read all of it But it ends with HP memo spanks Columbia researcher over flaming printer flap. You have to say that really fast So I'm thinking oh geez I'm getting spanked. It's awesome And then I just want to show you the one more. This is my favorite one. Okay It's funny for two reasons HP hit with lawsuit over flaming printer hack Turns out first reason why this is funny a lawyer had read all the articles from day one and day two Okay, without contacting any of us at Columbia File the class action lawsuit against HP just by reading Gawker and such That's funny, but I actually read the class action lawsuit that was filed And it really had nothing to do with flaming printer at all, but it didn't prevent wired from writing this Article so, you know, there's that saying right don't let the truth get in the way of a good story And this is a good example of that To be fair there were some folks who did get the gist of what I was trying to say and they did write more or less level-headed articles about what the actual vulnerability is with these printers and you're gonna see for yourself We're gonna have live demonstrations of everything that I've done and hauling these printers from the United States is Pain in the arse and we're not gonna haul it back So I'm gonna donate these printers to the hardware hacking area downstairs as my housewarming party to Ccc so if you want to get your hands on some hardware and play with it come find us after the talk downstairs So this is why I need to thank you guys Okay, my original security disclosure was done on November 21st on December 23rd HP released 56 new firmwares. I'm not talking about device drivers Printer firmwares for 56 of their printer models ranging from devices that are introduced on the market from 2005 to 2000 now and I think you know, this is a pretty speedy response And I think this quick turnaround probably had nothing to do with the fact that I'm standing here in front of you guys Talking about the technical details about this vulnerability on December 29th. So You know this happened in part because you guys are here you exist and you care. So this is great Here's a list of all the printers that are affected by this vulnerability At least according to HP and these are the printers that have new firmware released on the December 23rd So if you learn anything about what I'm saying today learn that go home check your printer update your firmware. It's really important Okay, so let's talk a little bit about this research in context who I am why I'm doing this and You know why this is all happening I'm a fourth year PhD student in the intrusion detection systems lab at Columbia University And here are here's a list of my past Publications in various academic conferences and if you read the titles of these these papers You can quickly see a pattern to what I'm doing Right for the last three and a half four years I've been basically quantifying and qualifying the nature of embedded security embedded insecurity on the Quantification front. I created this thing called the vulnerable embedded device scanner It basically continuously monitors IPv4 space for things that I call trivially vulnerable embedded devices Long story short. There's about 1.4 million embedded systems that ranges from backbone routers pop routers home routers and IP phones teleconferencing units printers, etc That are publicly reachable on the internet and are configured with their default route passwords Right in other words one in five embedded device on the internet right now can be exploited without any Exploitation you just log in with your default password and you get root on that box So there's 1.4 million of these And after I did the RFU vulnerability RFU attack on HP printers I turned the scanner onto the vulnerable printer population and it turns out that there are over 75,000 vulnerable unique vulnerable HP printers out there on the internet right now And I'm going to show you guys some interesting distribution of you know where these things are who's using them, etc and on the qualifying front I've been working on various exploit or offensive techniques for exploiting Embedded systems and I've basically done this in a bi-directional approach right looking at things from the top down in the bottom-up perspective Top-down I'm talking about things that make up the internet substrate Right the big iron routers that hold the internet together and from the bottom-up perspective I'm looking at you know common embedded systems that are connected on the periphery right on Networks that you have at home at work in your school. So The top-down stuff I did some interesting work on Cisco iOS Shell code techniques and that was presented in black hat of this year and today we're talking about Exploitation for printers right this is something that you all have you've all used in the past year and chances are if you've used a printer This year you probably used an HP printer and All this work is done so we can answer some really fundamental questions about the nature of embedded insecurity Right like can embedded systems be compromised? Okay? I'm standing here. I'm talking to you I don't think I have to work very hard to convince you that the answer to this question is yes, of course these are written With code that's vulnerable. They're they're much like general-purpose computers except there's a single function Right, but if you answer yes to this question a few other tricky questions necessarily follow Like you know have embedded systems been exploited in the past and more importantly have your embedded systems been Exploited in the past or right now and really how do you know for sure right? So just imagine your office you have your computer you have your printer You have your little LCD picture frame doodad that's on the wireless network You have a Tivo and a top-set box and a wireless access point and a teleconferencing unit They all seem to work just fine, right? But how do you know that there's no malware running on that device right secretly sneaking Information out of your network or you being used as a backdoor for other people to come in and hone your entire network It's not like we have antivirus for embedded systems, right? So I want to pause on you to think about what that means and why it is that we don't have antivirus or embedded systems We're gonna get back to that thought at the end of my presentation And here is even trickier question. Okay, so suppose I told you that your printer has been owned Okay through maybe the vulnerability that I'm gonna show you today Can you be sure that you can really remove the malware once the malware has been Installed on your printer now cleaning up a general-purpose computers is tricky, but you know more or less it can be done For most things, but I'm gonna show you some techniques. That's gonna make cleaning up printer malware very difficult to potentially impossible So with that said let's talk printers Okay, this is the printer that I did most of my research on and we have a pair of these on stage and we're gonna show you some live demos This is the HP 2055 DN And just for the record we bought these in what middle of September or October of this year. So You know, they're definitely on the market and they're definitely still vulnerable Here is a puzzle a co-in. Okay, how does printer update firmware? So you have to imagine that your printer you're on the network sometimes you print pages out and once in a while you update your own programming What is the most Zen way to do this? You print to it Right, I'm flipping through HP's manual and I see this remote firmware update using LPR command So I look at that and I say hmm. It's probably not a good idea Right, I click on the link and it's basically a page that explains how to use LPR in the most User unfriendly way possible, right? But basically all it's saying is if you want to update firmware on your magical printer Just type LPR and some magical dot RFU file, right? If it prints to the printer then the firmware is magically changed So for folks who know what happens after you press LPR, okay, you can already see where this is going and it's not gonna be good Okay, so I look what is this magical RFU file that you're talking about I go to HP's website I download this file right from you know, this is for my version of the printer, which is p25 or 255 dn I look at it and I say ah, this looks a lot like just a bunch of PJL commands The first one is a comment. You can throw that away. The second one is this The PJL upgrade size equals blah and then this is followed by just a third PJL command that enters into this mysterious ACL language, right? So if you play this stare at the binary blob game for long enough You realize that the thing that actually changes the firmware is a PJL command Which sends for printer job language It's a single PJL command and it's a single 7 megabyte long PJL command and that 7 megabytes is Compressed and not encrypted and I figure this just by looking at the byte distribution looking that at the entropy of the data And it's very typical for a compressed file and not an encrypted file And I started flipping bits in this command and sending it to the printer and it turns out this entire command is covered by integrity checking But the big question now is is there a mechanism that stops me from crafting my own PJL command because if I can do that I can just print to a printer and update the firmware in a permanent way, right? And one way that you know that would stop this from it working is digital signature verification So do they use signatures? So here's a lazy way to sort of approximate Instead of reverse-engineering the whole thing right away I just look at the the service manual and I look at all the error messages that this printer can produce, right? So if it does check for signatures, you would expect to see something like, you know signature verification failed, right? Didn't say anything like that. The only thing I was able to see was code CRC error send full RFU on port Now this is a fairly technical output, right? So I'm going to go on a limb and say, you know CRC doesn't mean digital signature and CRC actually means CRC Okay, so let's just review something obvious about what we've just talked about When you hit LPR, right? that RFU file is sent to the raw printing queue usually On an HP printer and that once you go to that queue the file is sent unmodified Unconverted to let's say port 9100 on the printer, right? and This mechanism TCP port 9100 raw printing has absolutely no authentication Mechanism right if you can print a page it'll just let you print there's nothing that says oh you want to print a page please enter username and password doesn't exist and This thing that updates the firmware is a PJL command now It's very easy to embed PJL commands inside post script files, right? But if you just thought about it a little bit harder You can probably figure out all sorts of ways of sneaking this valid PJL command into all sorts of different file formats and If you put one and two together, right suppose you're able to Craft your own malicious RFU. What you would get is printer malware, right? It's bad It's not that bad, but if you have malicious RFU plus some sort of a document attack vector Okay, you get something much worse Or much better. I don't know Well, so first of all you can use this as a spear phishing attack, right? So suppose I want to penetrate, you know super duper secret code, right? And they have Just hardcore perimeter defense and when gun guys with guns guarding their base, right? But they're also hiring so I put a resume, you know with along with his malicious RFU I package it together and I send it to HR HR says oh this guy is not an idiot let's print out his resume and consider him right what happens then is the printer will print out my resume and Again begin to update its firmware and from that point on I have a persistent foothold into super deeper secret code and What if I made a backwards TCP connection out to the internet to my laptop all of a sudden I've just penetrated the firewall I have a persistent foothold from internet through the firewall to the printer and now I can run all my Reconnaissance and attacks from a printer right, which is usually not monitored by IDS So this is the perfect thing to do and this is why if you had this you would not burn it Okay, you would save it and you wouldn't destroy it and Another thing to think about is so suppose I had this attack vector plus I combined it with an existing botnet code for let's say General-purpose computers right all of a sudden we're talking about multi species propagation right you can imagine let's say You know I get onto your PC using some sort of you know an old day And then I'll find all the printers on the network then I own those printers and from there Right, I would use the printers to scan and own other PCs to put my malware back onto those PCs So this is something that can spread from Computer to printer printer to computer and something like that. It's gonna be really very difficult to eradicate Okay, so all of this depends on Ability for me to my ability to reverse engineer the RFU format and craft arbitrary code into this thing If you want to do something like this, I'll save you a week of your life. This is what didn't work Okay, so you can just skip all that stuff Staring at the blob and work okay, but it only gets you so far bin walk is a great tool that supposed to Interpret firmware in all sorts of different ways and give you hints about what is actually potentially in this data For whatever reason bin walk actually failed in this case trying to find standard file system headers not there because the data is compressed right and Googling didn't work because if you just Google HP PJ L ACL, it's not gonna come up with anything At this point, I think this is the middle of September. We actually went to HP Little background story the reason why I wanted to do this in the first place was because I have this project That's essentially antivirus for embedded systems My thing is that I can inject this antivirus in an OS agnostic way into embedded systems now We contacted HP to say, you know, we're trying to put antivirus in your printer Can you a give a super proprietary RFU format and be? You know get back to us soon and of course they said no, thanks go away. We don't want anything to do with you so We went back to the drawing board and I noticed it so I started twiddling with this RFU and it's pretty easy to break the printer But it was also really easy to unbreak the printer All you have to do is turn the power off right hold these two special keys and into combination on the front control panel and the boot code would actually boot back up and it would rewrite all of the MV ram Content back to a pristine factory reset. So it basically just does a clink slate factory reset So I start getting this idea. Okay, so if I can get physical act I mean the answer is in that box, right? So if I can get physical access to the boot code, I should at least be able to see how I can write the MV ram and If I get lucky, I should probably be able to see some code that tries to parse or validate an RFU file And if I have that I should be able to get some sort of specification for this thing and then Figure out how I can pack my own code into this format. That's very mysterious All right, so I hit this service manual This is a page of the circuit diagram for the P 2055 and you'll notice that there are two Big rectangles right the formatter board and the control PCA and there's this little thing here And I want to just take a moment to digress and Just say okay the all the printer fire stories just got it totally wrong If anything we proved in our lab that you cannot Turn a printer into flaming death bomb and here's why okay? This is something I wrote to my advisor slid it under his door and say this little thing here is a ceramic heater It's good for fire right but here are these thermal switches that cut off when things get too hot and that's bad for fire and in fact I Brought the original sheet of paper. Okay that you maybe you saw in the YouTube video I put the fuser under in this printer hooked it up to a 24 volt 10 amp power supply put this piece of paper on Stood there and watch for 15 minutes and this is how it burned it just browned the paper Okay, so fire on this thing really not that possible You can actually look at it if you want and pass it around in the crowd You want to But I think my advisor do wants it back we're gonna make a t-shirt out of it I think but anyway, so We looked at you know I went through and looked at about six different models of printers and generally this is the design that I was able to figure out Okay in the HP printer There's actually two microcontrollers at work at least two the the formatter board is where most of the General purpose computing happens. This is where your web server runs This is where your talent server your format conversion blah blah blah right your SNMP all that sifts just everything So if you wanted the right malware, that's useful This is probably the place where you want to live, but there's also a second microcontroller called the engine controller board Okay, this microcontroller is also Programmable it seems like through the RFU format and this is the thing that actually keeps track of you know Turning on and off the the engines and the little rollers putting the paper at the right place Turning on the laser turning on the heater and the fuser and all this stuff So this is where the real-time printing mechanism is controlled And on this formatter board you have your usual neck controller some piece of memory that you boot from Volatile memory and in some cases persistent storage Okay, so now I'm starting to rip apart the printer right this is how I destroyed the first one But I got this information the formatter board is is arm. Okay. It's this chip here 88 PA 2al 2 dash TAH1 Long story short, I spoke to dozens of yield pirates in Asia Trying to get the datasheet for this chip, but this is protected by NDA So I can buy thousands of these chips and they can send me pictures of their stock and they can sell it to me for You know just a few cents per chip But nobody can even give me a pinout of the chip that they're trying to sell me And there are lots and lots of these vendors. It's really fascinating but The rest of the board there was one thing that actually made my job really easy and it's this pension fls. Oh six four P Right and it turns out that that's a flash chip That's a flash chip that the printer boots off of when you do a factory reset So here's what the formatter board looks like You know in real life This socket thing will Bob. I soldered on myself, right? And this is after I learned how to do surface mount Soldering I'll show you some pictures of what I did before I had a higher rework station But I did this so I can actually switch different flash chips in and out to have the thing boot up in different ways to poke at the hardware Okay, so this is the datasheet that I was able to pull off of digit key or somewhere for the flash chip So this is just a very standard SPI interface, right? This is a stock component So for example if you want to read some memory address you just send a one byte command followed by three bytes of target address and you just wait on the output pin for The contents of that address, right pretty simple I also want you guys to keep to look at this the right control, right? You can actually lock pages in this flash chip keep that in mind. That's gonna be important for later So, you know, I take the chip off. I look at the pin out It looks like exactly the same chip I have on the board So I basically you know send my read commands in here and I wait for the output here, right? And if I could do that I can just you know dump the The content of the boot chip and then figure out how this thing actually boots up and write the MV RAM Etc. So I bust out my trusty or Arduino. Okay, I built this very simple. I spare a dumper Basically just 40 lines of code. I mean, it's really not that big wrote a small Python script to control the Arduino So I can get the data back save it to a file on my laptop and analyze it in Ida Pro Okay, this is what I built first you see how I just have no idea how to solder and PC board This is my monkey soldering. I basically just got the pins straight to the Arduino and It worked But it was it worked terribly and it wasn't a lot reliable at all and just got all sorts of noise So I gave myself a b-minus for this effort And then I have attempt to Same thing same monkey soldering got the chip off of the board taped it duct taped it to my desk Throw up a lot of duct tape And it worked perfectly right so this is how I got a plus with just massive amounts of duct tape So after a couple hours, right I come back to my laptop But I slurp this into Ida Pro and I started looking at the strings and the code in the boot sector and it makes some Very interesting observations So first the code thing said that the chip that it boots off of is a boot SPI wrong gate wrong But we it's obviously not because this is writable. It's a flash chip The chip has eight megabytes of capacity and it has a small boot loader and right after the boot loader actually is a Small factory reset RFU. This is just the minimum feature set RFU That's meant to get you back in the network the web server back running so you can actually start, you know updating the firmware that you want and Look at there. So the RFU parser right because it needs to you know Read this RFU and the RFU parser is actually in the boot code Right, so if I can just sit there and reverse engineer like this much random arm code I would be able to figure out the format of the RFU and This is the content of the ROM right right after the boot loader and I noticed This funny little sequence here the uat sequence, right? I remembered maybe seeing that before so I think hmm Let's look around Here is the slide that I showed you, you know five minutes ago in the beginning, right? Can you guys spot where uat is? Yeah, it's right here. It's right underneath a CL. So I'm thinking ah, you know if this thing is parsed Right and it has a uat header and it looks a lot like the same format as this So maybe the boot loader code. Okay, well parse the thing I downloaded from HVs website, too, right? So I start reversing the code and I started getting these hints and Staring at it a little bit more. I figured out some structure to this code First of all, okay, so if you look at upgrade size, okay, this is obviously decimal represented in ASCII You convert this thing into hex and you get 7 9 0 0 3 2 which is super close to the nice round number, right? 7 9 0 0 0 0 so I'm thinking what what is this? 32 all about right 50 You shift in real line So I throw away the first two commands, right? I throw away the pgl comment and the upgrade size and then I jumped to byte 50 inside the actual command that parses the RFU Sorry, no, this is That's an image. I can't sorry Look closer But anyway, so right if you jump to your byte 50 you get uat So things are really starting to fit together, right? So there's a 32 at what 50 byte header and the rest of it is the rest of the payload Shift again for realignment. No, I look at the first two words before the uat header I read 0 for the first word and 0 0 7 9 0 0 0 0 right? So things are really starting to fit these things are starting to converge So this is this looks like you know the format of this binary string, right? You have the start address and address this mystery header and then various payload which describes The structure of the rest of thing which contains multiple compressed files for you know the update package Now at this point we've talked to call me or we've talked to HP and various Columbia folks and we've agreed to Not release a checksum algorithm and not release the uat header format Until folks have more time to go home and update their firmware like I said the firmware released on December 23rd But that's you know, not enough time for people to do this and roll it out and test it My goal for this presentation is to convince you that this vulnerability is real, right? It's exploitable and I want you guys to go home and tell everyone that it's time to upgrade your printer firmware So that said I start you know scrubbing through the rest of the data or the data section in the bootloader And I find this you know, I'm thinking Yes, right verify firmware key super secret bypass of crypto key enabled like I don't care what this thing does I have to use it in my project Because that's just such a good freebie It turns out I didn't actually have to even use the super crypto Super secret crypto bypass, but here's what I did. You know, so I figured out the RFU format I was able to unpack and pack all of the the form words that are available for download and Made some observations about what's inside this RFU now I can't tell you the compression algorithm right but I can tell you that the Specific version of the compression library that is compiled into this printer has a known stack-based buffer overflow Whether that's it you know exploitable for arm, you know, that's another story but at least there's a known bug in the boot code of this printer and What's inside the RFU is essentially just a Vx works operating system, right? It doesn't have the MMU edition doesn't have any memory separation whatsoever no kernel level security, right? And everything basically runs a supervisor mode on the CPU which means if you have a vulnerability that gets your arbitrary code Execution you win and you win completely on the whole device, right a vulnerability like this, so But then I thought oh, you don't even have to do that Right because we can just walk through the front door and use and not exploit a bug But just use the feature right the RFU update Standard is a feature that is supported in all of these laser jet printers So this is actually a much better way to do it because it's more you know reliable than you know exploiting a very specific version of a library Okay, so let's talk about the proof of concept. You know John's gonna warm up and do his thing The actual packer that packs arbitrary RFUs right for this demo is 200 lines exactly 200 lines of Python This includes comments spaces Comment it out crap, and I just want to brag a little bit that I actually wrote a unit test for this thing So I'm really proud of myself What you're about to see is essentially a vx works root kit now I know there's got to be better ones out there But you know I didn't know anything about vx works when I started so I want to write my own This is basically 3k of arm assembly, and it really has two Features that we're gonna show you in the demo the first one is a print job Duplicator, I mean it intercepts But it basically duplicates the job that gets sent to the victim printer and then it forwards it onto another IP address And the second one that you're gonna see is a reverse IP proxy, right? Like I said before suppose I got this malware on a printer in SuperDuperSecretCo but I can't get access into the Environment inside because of the firewall this reverse proxy will well when the printer boots up the reverse proxy would activate It'll connect outward through the firewall to my laptop and from that point I can use the printer as a launching off point I can you know do start doing reconnaissance and I can actually what we're gonna show you we're gonna show Metasploit working through this IP tunnel and we're gonna get root shell on that XP desktop laptop thing about over there all through the printer So to put all this things all this stuff together there's two components There is a RFU packer which is about 200 lines. I just talked about it takes an arbitrary bind elf binary And it spits out a pgl command that you can send to the printer And the second part is just you know my symbiote stuff that I reworked to make this thing happen And really all it does is cross compilation the malware code right does some binary rewriting some function of hooking and that input It takes the original firmware for the printer uncompressed and it spits out a modified version of this firmware Right, so obviously you take output of this you feed it back into this and you get a pgl string that you can use I included some details for people that may want to do this but it's really not for reading But I want to show you this okay Some mystery programmer types and the data section is full of this gold I mean you got you have to just read through it. I think do not type plough you will end up Inside a small building with keys on the ground. So I saw this Three o'clock in the morning, and I thought maybe some mystery programmer wrote this three o'clock in the morning many years ago So like I hear you you're out there So this is a slide that I want to show you okay, this is important detail. This is my main make file Okay, it's two lines all it does is call pack the firmware right with the prefix that you want and then LPR the firmware to the printer So this is basically print a poem right do this and it just sends the thing to the printer, and it'll work Okay, so we're ready for the demo, and I just want to say we did sacrifice the demo gods Because I got so I got checked all my equipment is okay with 240 volt except for the little Power strip so I plug that in smoke everywhere in my hotel room the electrician from the hotel comes up And we're just like I don't know why like none of my sockets work in my room Like can you take a look so we actually bought this guy which I also blew up we returned that and bought two more So we're gonna show you the demo now all right the first demo. We're gonna show is the print job replicator Okay, so Basically what's gonna happen is oh So first let me explain a little bit about the network topology here right so this makes sense this red cable here represents internet Okay, this is the only thing that connects this stuff to this stuff down here is a Cisco router that we're gonna call our perimeter firewall Right, this is a compromised printer. That's behind the corporate environment We have this XP machine that we're gonna try to own through the printer and John is an employee of SuperDuperSecretCo So he's gonna print his tax return on that printer because he likes to pay taxes Right Yeah, hi, and now what's gonna happen is that the tax return is gonna be sent to this printer But then the printer is gonna forward all the packets that it gets to my laptop over there Which is why where you seeing this scrolling and once I read the end of this file I'm gonna forward that To a printer on the internet and it's gonna print the exact same document now in our original demo We showed that we can tweet your social security number and whatnot But I don't have internet activity, but hopefully I can convince you that right doing this is not all that difficult I can scrub out whatever information I want and Exit trade it any way I want right so this is all done through this internet cable here second one It's gonna be a little bit more exciting So I just want you just give away your tax return All right, so we're gonna show you the second one where we're gonna use the reverse IP proxy right to Make this printer compromise this machine to get reverse shell back out through the firewall Here so what happened here is John just activated the oh You touched it wrong I know what's going on Yeah, okay So John just activated the reverse IP proxy right he's gonna use Metasploit to Essentially send the good old MSRPC vulnerability to the laptops IP address right which is I'm gonna forward that packet To the printer right and the printer is going to be instructed to forward that packet to this machine here Okay, and Now this machine is going to make we just find a interpreter shell back to my laptop through the firewall out on the internet Do you see that? Okay, so just review what we just did we made a printer Pwn a machine to get a reverse shell out in the internet to me. Okay, so this is what can happen if you don't lock down your printers Okay done. So we did the sacrifice paid off There was demo guide Wait a minute Don't die me now There will be a short intermission We're gonna try this again. I do full screen on this thing All right, no more PowerPoint when just do this PDF style This is reviewed. So you don't forget about what I just talked about Totally plan Case we go back to Yeah All right So you saw what the the malware is capable of doing so let's talk about Attack vectors. How do you get this RFU this malicious code onto the printer? Okay, there are two obvious ways to do it the one I'm gonna call the active method and one I'm gonna call the Reflexive method active is very simple. Okay, so if you can connect make a network TCP connection to port 9100 on your victim printer You can do this. You just connect to the port push this malicious RFU through kill the TCP connection and the The firmware start updating and this is true because prior to the December 23rd patch RFU update is turned on by default on all the printers that you saw on that list and Well doing this to port 9100 has absolutely no authentication mechanism So if it's on then anyone who can connect to this printer can do it Reflexive attack is a little bit more complicated, right? So here we're embedding the malicious RFU in a document Let's say in a resume and mailing it to someone and hoping that they'll print it out But then again, you know, if you're sending a resume or like a gift certificate or something, right? It probably wouldn't be all that difficult So let's talk about the quantitative scope, okay This is the PR message from HP right after that the public disclosure and it reads The specific vulnerability exists for some HP laser devices if placed on a public internet without a firewall You know as I'm saying like who on earth would do that, right? Actually 75,000 people did on the planet So we scanned using our scanner for the last month and a half and we found 76,995 unique vulnerable HP printers on the planet and 43 of these printers belong to governments and 16 of them belong to the US government and Here's the thing I love. Okay. There are nine printers on the planet named payroll And they all belong to universities and I'm just gonna skip save the question Yes, one of the payroll printers is Columbia and yes, we locked it down. So Okay, and it says and this is something that you rarely ever read right linux So in some Linux and Mac environments and maybe possible to do blah blah blah right this bad thing But implies that windows not possible So I'm not gonna sit here and try to convince you guys that you can't do this in windows because anything that prints can do This and windows can print so windows can do this but if I have time I have some really funny backup slides about a Support case that I had with windows to get a bug in word fixed. So this attack would work inside word But it's not really that it was a feature that it was supposed to work But so let's talk about the quantitative quantitative scope of the reflexive attack Okay, so we're not talking about, you know 76,000 printers anymore We're talking about all the printers that HP shipped that are vulnerable to this attack between I don't know like 2005 and 2000 now Right like I said before, you know, we're not talking. This is not a hundred thousand. This is not a million Right, this is not even 10 million. We're really in the order of hundreds of millions of potentially vulnerable printers And like I said before if you think about a malware that takes this combines it with, you know Takes attack factors that involve general-purpose computers combine this with Let's say a firmware update attack for embedded systems, right? You can imagine how difficult this thing would be to eradicate So let's look at just I want to give you a feel about how easy the reflexive Post-script attack is okay. This is a standard post-script file somewhere in here describes my resume, right? And when it's done, I just do this very simple thing. I say EOJ end of job Okay, and the printer is going to receive this file and says ah, I am processing post-script file I am printing post-script file. I have reached end of job. I am going to process new job I am entering language ACL. I am now updating firmware because that's totally reasonable. I am done updating firmware And the nice thing about it is it's actually gonna print out the resume So it's going to be fine. And I was just staring at the printer You won't notice that you know jobs aren't being printed for a minute and a half Which is basically how long this takes to work, right? And for most of this update process your printer actually responds to pings So it's not like if you constantly ping the printer, you're gonna see oh this thing went down for a minute and a half It's about like a 45 second downtime on the network Okay, so everything I talked about here applies to the 2030 and 2050 model Like as you saw before HP released 56 firmware So there are different types of RFU formats out there. They're slightly different But you know so if you repeat the same exercise that I did it really wouldn't be difficult to figure out the format for the printer that you're interested in and the other day John and I threw up this little diagram just to see what So we unpacked a lot of HP printer firmwares, and we just wanted to really quickly see you know What's underneath what are I say is right and what the operating system is? So, you know, it looks like most of them run MIPS and most of them run this thing called links OS Don't know very much about either, but you know, that's what we found So this idea that you know HP printers have a lot of diversity in the software Maybe but you know it looks like there's a lot of shared code, right in these models here So here is how you can go home run home and verify to see if your printer is vulnerable to this attack or not Right lock down your printer any way you can every way you can Maybe according to the HP this guide, which is actually fairly useful Download the printer RFU for your model and just print to it. Okay, if your firmware changes, you're not okay And if your firmware doesn't change you're probably good And some obvious immediate mitigation, right you want to disable RFUs This is actually harder than you think because for most of the models I looked at you can't actually disable the RFU feature from the web Interface or the tone interface you have to download this thing called WebJet admin or some other tool, right and use this massive control software to actually disable RFUs and Well, you probably want to act whole off your printers so that only your print server can connect and send jobs to the printer You probably want to filter some jobs filter the jobs on the print server But if you saw Andre Costin's talk yesterday and you can think about how you can put that together with this and see maybe You know post script being a touring complete language have to post script generate on the fly an RFU which is then printed So you really can't filter this easily without right emulating post script on your filter, right? So this is not something that's really going to be that easy to prevent and of course, okay most places you can you can Well, you can't cut off people's access to the internet because they'll probably if they can't Twitter They're gonna burn the building down But your printers don't complain and they don't need to talk to Twitter Okay, so you should probably segregate your printers on a network that is away from the sensitive stuff on internal network And also can't talk to the internet, right? So this is just a good thing to think about Now that being said on the 2055 all of these steps actually proved to be Irrelevant prior to the December 23rd firmware update. I haven't looked at this update, but With the older version of the firmware before the release There was no way of disabling the RFU update feature, right? either either in the WebJet admin tool or any of the interfaces that I saw there was also this thing called the PGL password Which is supposed to prevent unauthorized PGL commands, but it didn't actually prevent This command so you can set a password on it and this one just flow right through Long story short, there's just no way to disable this attack Prior to the December 23rd firmware, which is why you should go home and update your firmware print printer firmware and You need to do this quickly. Okay, because it's a race So whoever gets on the printer first can probably win forever And here's why now from the bad guy, right? I generated malicious RFU I have it on your printer. The first thing I probably want to do is just disable RFU updates, right? So you can't get rid of me. I could be a little sneaky and just update the firmware version strings in the right places So you think you updated your firmware, right? But I probably won't let you do any more firmware updates, and that's not it. That's not at all. I can actually Potentially write my malware into the boot flash chip I remember when I told you about the right controls here, right? So if I could do this I can probably just lock all the pages and on some of these flash chips There's a feature where you can write or you can lock once permanently Now if you can do that then your malware is physically resident You know inside this printer and the only way to get rid of it is to desolder the chip from the board But realistically you're gonna want to buy a new printer basically But it's not like you're gonna do that because we don't have any antivirus from embedded devices So you won't even know that you've been compromised Okay, so I'm mostly done with my presentation I want to leave you guys with you know a talk on the bigger picture of embedded defense Okay, so let's not just think about this immediate vulnerability But let's just think about the nature of embedded and security all together now HP's reaction Which is very predictable and pretty knee jerk is to say ah you are arbitrarily crafted malicious RFU So we're gonna prevent you from doing that. We're gonna digitally sign all the firmwares Which is one of the features that they rolled out But I don't have to convince you that you know sign code doesn't mean secure code, right? Because you're gonna go ahead and sign that compression library that has the buffer overflow in it That's just gonna be assigned vulnerability, right? So it's like you know putting up your thumb to block out the Sun, right? This specific vulnerability won't work anymore, but we can just go back to buffer overflows right to own the printer You know let's use a general-purpose analogy Let's say if Microsoft said you know we're gonna cut off all third-party antivirus and everything that runs on the kernel It's gonna be just signed okay, and but that's okay. Don't worry about it You probably say no thanks, but if HP says the same thing right right now. This is sort of just accepted that is okay now I'm gonna say that that's probably not in the right direction, right? So if you really think about real embedded defense and what you want out of it It actually ends up being a lot like just real regular defense, right? You first want host-based defense for these things to exist, right? They don't exist now they it should exist You want this thing to be a well-known defensive mechanism, right? You don't want any more obscure secret sauce. Don't worry. You'll never guess the magic number type of defense and You want this thing to be essentially decoupled from the operating system, right? Because the operating system is the untrusted code that you're trying to protect so you can't really say my operating system isn't secure but it protects itself, right and You know OS fortification is a good idea We should continue to do that, but it shouldn't replace independent security software living on the host on an embedded system And this is my plug. Okay, this is what I've spent the last three and a half years of my PhD career working on I've created this thing called the software symbiote. It's something that can inject Generic host-based defenses like root kit detection into arbitrary programs binary firmwares in an OS agnostic matter And we've shown that this does work. We've injected root kit detectors into many different types of physical Cisco routers Across tens of thousands of different iOS versions And if you actually want to censor it you want to test it out Please email me privately because we're actually trying to get people to test out our defense and see you know what's good and what's bad and I'm hoping that we can do this I'm hoping I can get back to my original project which is put defenses onto printers So this type of attack can happen and hopefully I can do this in 2012 and actually demonstrate that this is not just you know I always version independent, but operating system independent, right? I can get the same defense to work in a router as a printer So that's it. That's right funny, right? So we've got 10 minutes for questions and I believe All the angel back there has the microphone for questions in the audience. So if you have a question, could you put your hand up, please? Okay, we'll start with this gentleman here right at the very back what sort of formats can you actually embed the Religious code into could you false and put it in word doctor PDF file which are come Supply for printing Okay, so, you know for my demonstration and I only put it in postcript But this is close enough to my backup slide that I want to show you this really funny Microsoft story So I'm looking at you know all the legit ways I can put PJL commands inside a word document And it turns out that there is an actual feature in word that does exactly this it's called print field Okay, and it just does they just put up whatever PJL command you want in the dock and it can be printed now Problem is there was a bug anything that was longer than 240 characters introduces random weird character in it So I'm thinking well, this is not the specification I'm gonna open a support case with Microsoft and have them fix it and they tried. Okay, we spent this guy spent two months Trying to figure out why it was not okay What didn't work when I tried to put a seven megabyte PJL command in something that great in something that it was supposed to super 260 characters and You know dozens and dozens of exchanges later never once did he ask why are you putting seven megabytes of data in this thing? It's not supposed to be the way But actually HP released a driver update on December 1st after we made our disclosure to them And this guy poor guy comes in the office and tries it again And all of a sudden this big scary pop-up box comes up and says you can't do this anymore go away. Thanks Long story short Well, I haven't tried any other document formats yet But I think that if you think about the way that the cup server converts your binary right as long as you can Have your pgl command survive the conversion intact or have it be generated somewhere in the process and they have it So I've intact when it's once it reaches the printer. This is definitely possible, right? Exactly which Formats as possible right now, you know, I can't really say but I would expect that many different file formats Will be possible with this type of tech Okay We'll take this Seen it have we got any questions from the from IRC. Yeah, that's stupid question Right Okay, we'll go with these guys here because hopefully your questions aren't dumb Okay, first of all, I have a small note basically the print field command and the word document where Practically demonstrated like in 2010. Yeah, so that's not news but the question is is like From those 56 firmware updates actually the reason why you cannot disclose the compression algorithm is because they just deployed the fixes with disabling there are a few and still working on the signature thing or They deployed the signature thing, but they it's not quite so in my slide I have the the link to the exact SSRT There were there are two revisions the second one actually details I think something like 20 different printer models have introduced the feature for signature verification Right the rest of them that had signature verification. It's my understanding I just looked at it, you know, I just read it not an HP product Person, but I think the rest of them or all of them have disabled RFU update by default Right and some of them it's actually now possible to disable RFU updates on those printers like this one So that's been patched to Printers where you couldn't disable this feature before you can now So this is really important for everybody to go home check your printer and just update the firmware Okay, and I think You've shown this Job control language command where you can choose which language the following Blob is have you actually tried using the post script console for Right, I just saw the talk yesterday. I'm gonna go home and check it out I'm actually so I'm gonna take these printers down to the hardware hacking area and probably just try that so You know come find me after the talk and we'll find out, right? Okay? Okay more questions brought it back here. Oh Okay. Yeah, sorry. Sorry. So one question is about Okay, one question is about So don't don't put it to my mouth. Okay. I would don't want to eat the microphone Okay, so my question is about self-modifying codes. So thanks. I'm for example, you make they made a pop-up box that Detects out that you you want to do something evil. So I at least I understood it that it like that I Understood like this driver set in the pop-up box. Now. It's not possible anymore Go away. So if you make self-modifying codes, for example, you make this payload Maybe x or by some some some some short string and then you make the self-modifying code into post script and it Then it these ciphers in the printer memory and then gets executed. Exactly. So, I mean That's not the only way you can get this thing to print through, you know word I'm sure there are lots of other ways I just wanted to use the pgl print code thing because it was cute and it was an actual Feature that I thought it would be great if it worked, but it didn't so but yes, you know, like that's the thing You know, I'm gonna try out what I heard yesterday and maybe potentially compute right the RFU on the fly So that filtering would actually be very difficult And maybe have the thing print to itself, right? Who knows maybe that's possible Hi, what's the chances other printers have these kinds of vulnerabilities? I Can't really I mean I Don't want to say I mean I don't answers I don't know but it wouldn't surprise me if other printers have this vulnerability, right? I don't think I think Andre mentioned yesterday that Xerox printers would share a lot of common history I think with HP printers also can update their firmware through you know post script their pgl interface So maybe Xerox printers, but I don't know. I mean it wouldn't surprise me if others have the same vulnerability basically Hi, I was wondering if you could they'll operate on your fingerprinting. Are you relying on like a version string or what else? What do you mean? What do you mean by fingerprinting like you were your fingerprinting all the printers on the internet? Oh, so I did that by a few different ways, you know by looking at the web server the town that Prompt banner and by looking at there's this actual command that's um, I think PGL info prod info on some models of printers. It would actually just return the entire Chassis ID and then the model number etc. So we put all that data together To find to figure out this figure and we have one question here from IRC that's Got been voted up a few times and doesn't seem totally stupid The other one was can you make the printer blow up and like how do you actually watch the talk? Okay Okay, I've got a question is there an attack in the wild You know, this is a really great question And I don't think we know why because you don't have detectors Antivirus for embedded systems to find this type of thing, which is why my original project was to inject Right this type of detector into the printer to fight figure out if printers have been exploited in the wild, right? And that's exactly what I've been trying to do with the Cisco router Sensor because if you think about it, you know, this code is very old We've talked about vulnerabilities for these things the question is not can they be exploited? But have they been exploited? And if so how sophisticated is the payload, right? We don't really know for sure because we have no good way of finding these things Which is exactly why I've been working on the software symbiote project Okay, and I think that's I think we're out of time. We've overrun by a couple of minutes So I think it's time for a massive round of applause for the whole team Especially John John get up, man I think you're I think your supervisor here needs to stand up and take about a sponsoring the work