 As it's the beginning of camp don't really have a lot of announcements, but real quick to the 12-year-old out there That just got your first Wi-Fi pineapple Yeah, that's cute Stop it, but I guess without further ado. We'll get started this morning It's my unique pleasure to be able to introduce a longtime friend of ours up here Bunny's been with us for a long time, but absent also for a long time Unfortunately due to things like geography and whatnot, but we're space and time But we're so happy to have him kick off our conference. It's a great start for the cons. So please welcome bunny to the stage Everybody Thanks. Thanks for having me. I'm like super excited to be back and and with good friends and in a great location Dave asked me to sort of come up with a talk about sort of hardware hacking sort of Essentially like the hardware hacker book but in a talk and so I started thinking about it He suggested the title making and breaking hardware Origin of the hacker ethos and it kind of turned into more of like why I like hardware hacking and how you can get into if you Haven't gotten to already how many people here would say that they're like kind of a hardware hacker or play around a bit with Harder that sort of thing. Okay, so actually a really good number of people are into it And how many people would say that they're more of a software hacker? Also a good number of people. Yeah, so so I mean just to get started There's a valid question of like does this harder even like matter anymore because we have Supercomputers in our pocket and what we can't put in the pocket. We can go to the cloud and You know get had you get whole supercomputers essentially at least You know sort of through the internet via the cloud if if we could containerize a teaspoon somehow We would have no more mountains because people would terraform for the fun of it. It seems at the end of the day And so like, you know, there is a sort of illusion of possibility that comes from from software, right? There's a sort of it's a bit of a fractal thing It's like you have an infinite surface area to explore There's a bunch of things you can do but the truth actually, you know as with a fractal is despite the infinite Surface area you're there's always a finite volume that volume is bounded by the hardware that that you're given so If you were to have you know an army of men you could take over countries You could build pyramids all this sort of stuff But they couldn't do the most basic thing for species survival Which is to have a baby because they don't have the right hardware to have a baby So don't forget that the heart is actually extremely important There's some things you can't do without the right hardware So this is this here is a picture of an infrared view of a circuit board I use this often times to see if I have a problem You can find hotspots and so without that infrared sensor You don't have visibility into that spectrum of light and that's a very helpful thing to have Just to have that one sensor so this is an extension to what you might have on a normal camera And there's some things are just easier to do with the right hardware So you know if you're trying to hack a system and you have some crypto keys You're trying to extract I mean you could build a supercomputer and hook up to hydro plant and wait many many years Or you could build a piece of hardware drop it into there and sort of pull out the actual keys themselves so There's kind of two big reasons when I was thinking about it for why you would want to hack hardware There's basically a security reason So maybe you want to extract some secrets and there's another one which is sort of the interaction reason So this is sort of like extending the capability of a piece of hardware beyond its existing capability So I'm going to talk about both of those facets a little bit So we'll start off with the security side of things. So why Would you want to go ahead and hack harder from the security side? So people have gotten better over the years at protecting secrets and hardware from hardware security modules To sort of like whatever Bitcoin wallets sim cards people now sort of Putting stuff at the chip level burning things into e fuses. These are sort of like micrographs of of What the the fuses now look like they're hiding secrets inside of a chip and so Now that people are getting good really good at burying the secret keys inside of silicon and inside epoxy that sort of thing Sometimes it feels like the only option you have To get those secrets out would be to build a big super computer and a hydroplan and wait a billion billion years to sort of brute force it out And that's the point at which you want to go ahead and you want to take the red pill and change the rules Right like you know we you if you exist within the rules of the hardware you can't you can't you're just up against a you know sort of Impossible computational tasks to get the get the keys out So there's a whole bunch of things you can do don't worry about the the the list I'll talk through it in pictures to try to extract Hardware secrets so the most obvious thing to do is like direct observation You can go ahead and you can take a microscope or a scanning electron microscope and look at the piece of hardware And the cool thing about sort of atoms is you can't encrypt them You can try to obfuscate them, but that's very different from encryption And so if you get a better microscope you can basically always find the secrets on the inside Actually at at tour con about 12 years ago or something like this I was here with a probe station and we had chips and we took them apart and show people how to get the keys out and that sort Of thing it's actually it's actually as easy as knowing where to look the problem is is that When we did the chip at At the last tour con we printed like a 10 by 10 Sort of picture of it and you could actually hold all there You can actually point at the transistors of course today with a billion transistors on the chip It's just it'd be like the size of a city block and you really have to walk around it for days before you could find Where the security keys are? And so there are other ways they're easier to get it out for example a direct measurement So this is a throwback to a hack I did a long time ago where I pulled the keys of the Microsoft Xbox on the left-hand side of the picture here this is the Xbox motherboard and there's a six some Secure keys that were held inside the southbridge chip there I said to the CPU and At the time, you know, we weren't sure how to you know, basically brute force the keys out So I built a little circuit board that was that sat on top of the bus and just sniffed the keys out as they went by And so that's an example of a passive method. So there's not we haven't really Directly modified the function of the computer. We just observed the traffic as it went by So this is a little very simple board that we built and we're able to pull out the the keys that way There are other sort of really cool passive techniques. So it turns out that a transistor Whenever it switches emits a photon in the infrared spectrum. It's a very it's a very small amount of light But it's detectable and so there's a paper that was done Which shows this is a picture of a chip in the optical range And this is a picture of the chip in the infrared range and as the transistors switch during computation You actually can see them turning on and off And so they had demonstrated that they were running an AS algorithm And they're able to actually pull out the keys by observing the pattern of lights coming out of the transistors as they switched on and off This requires a little bit of like a manipulating of the silicon You have to grind off the backside till it's very very thin But it gives you an idea of sort of the power that they have now to sort of extract secrets from chips as they're running through These sort of passive optical emissions If you don't have access to the crazy, you know microscopes and stuff like that There's the other things you can do like just observing the amount of power that a chip uses during a computation So this is a example of so if you have it if you have a chip and it's doing a computation If it has more computation it draws more power, right? And so if you have a round of AES or a public key algorithm You're running depending upon which s-box you're doing which part of the key you're doing you'll have a different signature of power And this here is an actual sort of picture of the trace of the power going through as it goes through different rounds You can actually see they have a different length of a signature and so you can actually extract the the secret through Observation of a power site channel and then and this is actually becoming so commonplace Now you can actually just pick up a piece of Harga the chip whisperer and it has great docs online and teaches you how to do that There's also RF site channels so chips in addition to having power Fluctuations and photons being emitted. They'll emit stuff in the railway Domain and these sort of round things here are RF probes. You can buy them. They're actually They're called near field probes And you just hook them up to a small amplifier in a oscilloscope And you can get similar traces to the ones you saw of the power if the if you can essentially get the chip quiet enough So it's doing just the computation or you can do a ton of averaging That's another thing you can do if you can get a good trigger on it And so there's a paper here of a sort of people pulling out encrypted keys for Xilinx bit streams And that's where I pulled the picture from So if you if you if all else fails on the passive side you can start to get active So the ultimate active thing to do is called a focus IM beam. It's a fib That's this is what it looks like sort of like a little desktop size machine and it fires Atoms of gallium at the surface of the silicon and if the atoms are at a high enough energy They'll hit the silicon and knock it away if it's low enough energy to add gallium atoms will settle onto the surface and create a metallic film and you can inject Other gases and at the same time to do deposition and so you can see an example here This is actually a micrograph of a chip where they've actually cut holes into the chip itself And they jump would wire over it and rewire the chip to do something different So this is actually an example. This is actually wasn't a security example This is someone who just designed a chip and done fucked up and they had to fix it and And they and they went ahead and rewired it So they didn't have to go ahead and like pay millions of dollars for the mass set But this type of technology was actually originally developed because making chips is hard developing chips hard turns out that same technology can be used to Rewire your chips if you want to go ahead and have them divulge the secrets when they're not supposed to If you don't have access to a fit, there's other ways to sort of actively modify the way a chip runs You can glitch the power you can glitch the clock you can glitch other pins like the reset You can actually you know how I said the transistors emit light Physics has this duality property things that emit light can also be sensitive to it You can actually like shoot a laser at the chip at a very focused location and flip a bit if you want to These are all these are all possibilities. The only problem with light is that actually photons are really big So like the infrared photons are about a micron in size and as you're well aware Transistors are now like 28 nanometers And so when you now today when you want to glitch a chip you're actually glitching like a whole Field of transistors is what turns out But full injection itself is is interesting and um, you know, the whole idea is that you go ahead and you just Let me see if I can't reposition this so I Can Okay, so the idea is that you want to try to modify or corrupt the memory or code in some predictable type of way So a lot of times like for example when you're powering it on you'll go ahead and as a Code is being pulled into the cache You just sort of glitch power a little bit any meal flips and bits of code That was supposed to be secure that was going to compute like a branch target or something like this Another thing is is as you're doing the actual cryptographic algorithm itself if you can just like You can get residues out of their cryptographic algorithm that can leak portions of the secret key So this is a common attack against RSA Particularly in plantations that use the Chinese remainder theorem or you can just change the reset So like you know you can just say instead of running code from the intended location It goes from some other place and this this picture here is actually that it's literally called the tweezers hack This is on the Nintendo We where they have the tweezer between ground and the address pins and during reset hold it and so what happens is that the code that was meant to be secure gets loaded into another address bank and then they can go ahead and Execute different code as a result of the of that hack So it's sort of like a small injection can be actually very simple because a pair of tweezers at the right time hitting reset Can get you to the point where you want to be? There's also some sort of In between active and passive for example coupling is a new thing that's come out And there's a thing called row hammer pretty interesting if you were to look at a DRAM chip so this is a micrograph of a DRAM chip these are the these are the row the row wires that the data sort of like being read out across and It we're looking at one tiny portion of chip this wire actually will I go from one end of the camp? It's like super super long and so when you have two wires are this long next to each other parallel Go ahead and In effect each other and so this is that this is an attack You can go ahead and if you have access to two wires They're adjacent to one that holds something that you want to modify you swing them up and down You can go ahead and corrupt bits on a page of memory Through something like this and so that this is a relatively new form of sort of active attack And I think there are probably other Row hammer like attacks out there that exists in particular as you start to push into these deep subnan or subnanometer processes the margins get thinner and thinner so that should be an interesting area to watch in the future I mean is also a kind of an analogy I guess for row hammer that happens in nan flash But these nan flashes so much ECC on it typically those things get scrubbed out before they cause any trouble So that's sort of like the the security half like that's a the quick summary of why you might want to hack hardware if You want to get secrets up but actually a lot of my time recently has not been so much focused on the security side But more about extending what computers can do sort of on the interaction side What what can you make computers do that they couldn't do before I mean if you remember if you without the right hardware There's some things you just can't do so you really want to try to put more hardware capability Into computers just things you can't be aware of and so in in terms of thinking about You know what you could do with a computer. I came up with a sort of ontology of things I'm sorry it gets a little bit academic, but that's like sort of my my tendency So there's three major areas that I think that you can divide the realm of interaction to this Computers interacting with other computers. So that's the top bubble here That would be you know things like the internet your computer interactive the computer is Wi-Fi Bluetooth that sort of thing there is Computers interacting with physical matter. That's this bubble over here and I've written sort of like the seven SI units position time mass current intensity temperature and mole which is sort of an odd one but The there are sensors and actuaries we can use to go ahead and affect physical quantities So, you know your earphones turn current into the motion of air a robot arm turns You know electrical impulses into position and mass we have GPS that tells us position in time We have optical sensors that can detect the intensity of light over time so and so forth those form your base set of sensors And it's really interesting when people develop a brand new sensor like an infrared sensor or or something like this That can can unlock a whole brand new set of functions for computers that wasn't available before and then on this bubble Here is sort of society. How computers interact with society itself the ways computers can change society is for example cryptocurrency or surveillance Artificial intelligence changing the way we feel about other people There's self-driving cars which can change the way we compute There are sex robots that change the way we have sex and interact with other human beings There's cryptography that can limit the what we can do with our own hardware through sort of the norms that we agree in society And then you know, maybe not too distant future though But the actual direct human computer interfaces where computers are like literally plugged into our brain and that will be Sort of a very interesting time indeed But you know when I was looking at sort of The space of things you can do and an observation came up Which is that computers are basically defined by not by the CPU but by the sensors and peripherals you put on it So for every base, there's kind of like three tiers of computation I kind of put out here the sort of like the Arduino level thing which is embedded microcontrollers There's sort of like the mobile level which is sort of like represented by a Raspberry Pi But also if you look at the Raspberry Pi and like an iPhone, they're basically the same sock Remory sort of combination on the inside and they have sort of the desktop server class motherboards So between those three levels if you're a hardware engineer and you're designing your schematics We're going to start from basically the same spot every single time and then really it's all about the sensors you put it about It's all about the interaction. So these are two Different products to actually use the exact same CPU a PXA 168 from Marvell I didn't know this at the time, but the PXA 168 is used in the Xbox connect Which in is also used in a product I designed called any TV So and you guys all know what the connect does the any TV is a device that allows you to put Sort of your own your inject your own content on top of encrypted video So as a sort of experiment to see what we can how can how you can get around sort of the roadblock that DMC puts on Your ability to manipulate video. They both use the same sock. So that's the that's so see that's the Marvell sock They both have the same kind of DRAM architecture. They might even run the same flavor of Linux. Who knows But then the sensors that we put on this on the side of that it really changed the product overall I put an FPGA that allows me to interact with encrypted video feeds. They put some amazing Sensors that allow you to sort of you know sort of look at you know Sort of space and measure the distance people are from the camera And so it's all about that it's all about the interaction layer that changes kind of defines what a computer does Another example is you know, they're a tweet Omega. This product is in both of these Devices one is a power supply. The other one's a 3d printer It's all about sort of you know what they do in addition to the core computer at the end of day Or like for example accessories so adding or extending interaction So if you have a phone which is like this amazing super computer in your pocket and has sensors and it has stuff There's some things they can't do like if you want to measure the wind You have to get a little thing to plug in and they can measure the wind if you want to measure your body There's some sensors that let you interface your phone to your body and measure that It could be as simple as like a selfie stick, right? So a selfie stick is not a fancy piece of hardware But the fact that you can now take a phone and put it in a position where you can take a picture of you and a friend Unlocked a whole bunch of new social interaction capability with the phone itself So there are there I mean, you know kind of hardware hacking can take some very subtle things that just changed the way the computers Where they're located or how they're how they interact with humans and can create sort of new social interaction Themes so interaction is is where it's at. There's a incubator called hacks that I Mentor for in Chen's end every a few a couple months I see about 12 startups and I would say like 70% of the startups that go through all hinge on the existence of a new sensor or an actuator So they're not about like oh, we're going to you know do a new microprocessor core or something like this It's more about like hey, we have You know, we have this new actuator That's actually really good for people who can't walk so we're going to build like prostheses or we're going to we have a New sort of sensor that's good for measuring the you know Sort of the temperature of a human's build a fertility monitor these sorts of things These are the sorts of stuff that are the focus of of that group and then I also know a research affiliate the MIT Media Lab and 90% of the research there is involved sort of computer-hardware human interaction again, so that extending computers is space of what they can do and What's interesting about computers and if you look at this ontology they put together a lot of these buzzwords and harder that you That we all talk about are at the intersections of these three circles So between other computers and physical matter. There's the Internet of Things Between other computers and society. There's all this AI and machine learning hullabaloo, you know vote manipulation and whatnot there's a physical matter and society that's like you know You know people 3d printing and or drones moving things around now autonomous cars pushing things around So you know what happens here at the middle of this is it is it the robot revolution? What happens? You know actually it's it's the the right hardware at the right pace can be very simple But if you can sort of like over you know kind of hit that perfect job of getting all those different things you can actually You know create something very meaningful, but it doesn't have to be very complicated So this this here is a square which now it takes a phone Allows it to interact with physical matter that couldn't be for the IE your credit card to other computers and changes the way we do payment right and this very simple piece of hardware because it's sort of Hits these three different areas changes interaction society and physical matter and networking Has unlocked sort of a new business model a new way that people can can other people can do business so Given that you don't have to you know build a robot revolution to do something interesting Why aren't there more hardware hackers right if it's so compelling how come there aren't more hackers? Well going back to that analogy of the red pill in the blue pill the problem is when you take the red pill It you know you end up coming out in this like goopy world the desert of the real right? It's not it's not like you you're coming out of that wonderful sim in the dream of the things are and the thing is The desert is literally real right so like you know if you you know Arn software you want to get rid of your stuff arm-ar star great It's all gone right this and then there are actually places in the world where like it looks like this because of e-waste This is an e-waste processing facility in Africa where they have you know a desert that is covered with the stuff that we threw away We have this abstraction called a garbage can that we throw things into it seems to just disappear Where did things go? Well they have to go somewhere their atoms They don't just arm dash our star and so Atoms are difficult and they actually in a lot of times you try to abstract atoms We I mean the more we can abstract in the belly. That's what Amazon is Amazon sort of an abstraction of atoms can just appear at my house That's amazing, but there's like a huge amount of Overhead and a lot of costs that's involved with creating the abstraction that we that we all love and and use every day So the problem with harder is that it's made out of atoms and Ams are really annoying so atoms Owned by people people like to say that these are my atoms or these are your atoms and I have to protect these atoms We have to buy these and then trade these atoms That's actually one of the most annoying parts about atoms is that it's not like a computer I need more computation resources I go spin up into the cloud instance and Whatever it is and then if I don't need I just turn it off and it goes away at you know If I need more you know if you need more minerals or whatever it is You have to go and mine it and and and make it available to to build your to build your things Atoms are imperfect right so in nature They don't exist as perfect crystals of silicon And so there's a lot of effort that we have to go through to align them position them put them in the right place to Create the things that we want and after you've gone through all the effort of buying atoms and Perfecting them and refining them. They just don't move around on their own right You can have just warehouses full of atoms that do nothing if you can't get them to the place that people want them to be On their hand on the other hand bits are perfect and way more profitable. This is you know This is in that the NASDAQ you know Sort of valuations to the dot-com boom if you guys remember that you know back in the day of like coasts and Akamai Alta Vista the days right And so what happened during that time of the of sort of the dot-com boom is that we end up people were like bits are way More profitable hardware sucks. There's a lot of outsourcing mania that happened during that period of time So it's sort of the 1990 to 2008 range There's sort of like this notion of why would you own a pick-and-place machine when you can rent one right? so there's you know if I have a pick-and-place machine that's building blackberries and Another person has a pick-and-place machine building Motorola razors and if you know supply and demand is in balance on them Well if we both rented the same machine, they actually are both busy all the time And so we get more economy of scale by not actually having to own that pick-and-place machine And there's also some some accounting benefits This is the actually sort of it's interesting to sort of see the valuation of Foxconn You'll kind of rise up with the with with with the outsourcing boom There there's certain accounting of benefits actually that actually really drove out as well And I want to talk a little bit more about that so the there's a cycle that a business goes through where they take money You buy parts right then you put the parts together You ship them around the world you sell the parts you get more money and you buy more parts Right, that's a very it seems to be the intuitive cycle that you expect for hardware And this is like a 20 to 60 day cycle and investors who are very clever people like to make metrics And they say you know what are your days of inventory and was your inventory turnover and that's a measure of the healthier company Well, it turns out that during the outsourcing boom people figure out a hack on this where they say okay if we don't own The parts and we don't own the SMT machines We just buy them as they come out of the factory and we just ship them to customers We can make it look like like our inventory supply chains are really really really really lean and so at sort of apples height They actually had a three-day Turn of inventory so the fiction made it look like you know They could take somehow snap their fingers and three days later all the inventory is moved actually turns all they did is that They would they would buy the stuff from the factories they would hold them in a place in Shenzhen very temporarily and You would then buy it online and because of the way the counting rules work if a if a tracking number is emitted That's the point which they can say the inventory has been has been shipped out So actually the inventory didn't move it was brought into a truck into their facility They signed some paperwork that said that we now own the inventory three days later You bought it online it could still be sit in their in their in their factory But because they sent you a tracking number they realized the revenue and so on so forth So like this this is how sort of How much day outsourcing has has moved in terms of you know sort of the accounting cycle and why people want to get rid of all The hardware it's so currently an apple like you know Sitting at 11 days worth of inventory in the industries it still at 70 days because people some people so I'm picking place machines So there's a whole bunch of outsourcing that happened people want to get rid of all their hardware and then in 2001 there's sort of strange coincidence is going back to the Xbox hat hack I did I was sitting in a IRC chat room and we're pondering about how do we get these secret keys out and literally There's like the question like does anyone here remember how to solder anymore like we've we've we've put those soldering irons aside about a decade ago And we all went to start our companies does anyone know how to solder I like to raise my hand like hey I know how to solder I can maybe come up with something and and and that ended up leading to this hack which I led me to write a book which got me a little bit of notoriety and then there's this irony of I Went to an emerging technology conference because of the work. I did there this in 2004 Where they wanted me to tell people about this emerging technology of hardware hacking Okay, I mean like I've been doing this all along. It's not really emerging you guys just forgot about it, but But you know a guy named Bill Doherty was at my talk in 2004 and he says that this was one of the talks He saw in 2004 they encouraged him to do You know the make magazine which ended up leading to the maker fair which sort of brought out the sort of this whole side of like make Demonym whatever it is and then maker fair gets big and goes to Shenzhen and Then there's this huge criticism that happened in the first year than Shenzhen that now this the Chinese They're not real that it's too much like a trade show right like you know What's going on with this this maker? You know Chinese don't get maker They're just using it as a trade show that kind of thing and then you know I sit there I'm like is this you know is a tail not wagging the dog now at this playing time that we're like Criticizing you know the Chinese ecosystem for not you know being maker enough and so and so just just to sort of Calibrate some expectations about sort of Shenzhen. I'm gonna spend a couple slides just talking about the the area This is a picture of electronics district in Shenzhen and I Guess a lot of people haven't been there yet But this is if you can see it these are like really tall buildings that go back for city blocks And every single one of these tall buildings is from like ground floor to the spire full of Companies are involved in hardware and little small businesses like mom-and-pop businesses They're involved in the trade design and and manufacture of hardware if you were to go into one of the buildings This would be a very typical scene You have rows and rows of stalls like this and each of these little stalls is kind of like It's actually model on the old thing where you would buy fish and meat and it's like a wet market a Chinese wet market except people are now Instead of like hacking up pork. They're hacking up iPhones and and they'll sell you parts and stuff like that And and there's like tens of thousands of these little stalls you can walk in there And you can just you know with five bucks walk out with ten thousand resistors and go to production Whatever it is cash and carry a tray of chips Whatever it is this these are all this is like sort of sort of the density and that's sort of the fecundity of that the ecosystem This is a very typical scene There's a guy here with a bunch of iPhones on his desk and he's just you know fixing and reworking iPhones This is not this is like not a thing that you you always imagine that if you're gonna work on an iPhone It'd be like in the clean room because those things look so immaculate whatever is this is this is actually what it looks like When people work on that stuff out there and it's to the point where it's like literally in the street There's like use a little mobile phone just like pouring out of a bag onto the street and people are and it's e-waste And people are going into those things and pulling out parts and figuring out ways to recycle when we sell those bits and pieces so that's that's kind of the the density of electronics in that ecosystem and And it turns out that when you have that kind of density of electronics You can actually like dumpster die for factories and that and that this is a this is a picture I took while I was driving on the highway This is a SMT line that's on the back of a truck and they just sort of like put danger tape around it to Secure it to the truck because whatever. It's just an SMT line And it turns out that like you know if you think about it, you know, maybe rim went there You know 15 years ago put a million dollars into buy a top of the line assembly equipment They got a business that gets sold a scrap for a hundred thousand dollars a few years later The next guy goes a business self scrap for ten thousand dollars And so now you have these tiny little factories are picking up with things that could build mobile phones for like super super cheap And turns out that when it's easy to have a factory anyone has a factory This is a picture of a of a vendor that I actually was working with to I wrote a book about a guy to Shenzhen and on the backside I want to put these little Cardholders so you can put your samples and business cards and so I had to get these custom Sort of plastic sheets that were stamped and molded And so I actually went to the guys factory and it's like this old gentleman here owns these machines And the young guy in the front owns the printing presses and so forth And so all these guys just have factories and they just whatever you can go into them And it lead it led me to sort of like this insight that the difference between What we have here in America or in the West and what we have going on inside an area like Shenzhen is a difference being inventory and capability So to a little illustrate a bit more Let's take the hypothetical case that you want a USB cable and you want a USB cable. That's 1.8 meters long exactly 1.8 meters long for whatever reason and you go into Best Buy or wherever it is and you talk to the gentleman You say hi, I'd like to buy 1.8 meter USB cable. I said well We have 2 meter USB cables and 1 meter USB cables How about you buy the 2 meter cable and just wrap up the little piece? You don't need put a little twist on it and call it the end of the day Like no, I want a 1.8 meter cable Can you make one for me right and the guy will look to you like your crazy says I have inventory of 2 meter cables You should buy my inventory right so it's it's so basically your space of possibility is defined by what that's what the retailers Have inventory are willing to sell you However, if you go to an ecosystem like Shenzhen you have people who have rock wire and they have cable endings They have this machine here, which is the oval molding machine and it cuts the cable It's a little you know dial your paddy can punch in 1.8 They'll cut the cable to length put the moldings on the ends and you can have your 1.8 meter Cape 1.8 meter cable So this is the sort of ecosystem we can actually go to a guy and say like hi I want a 1.8 meter cable. I want to be blue and I want to have these endings I want to do this and they say okay sure how many want like you want okay? Just a few okay come back tomorrow and we'll have it right so that's the difference between an inventory ecosystem and a capability ecosystem So this sort of notion that hardware is hard is is driven a lot in large part because The right tools aren't necessarily available for the job. So this is a picture of someone trying to pull off a BGA Which is who BGA soldering that sort of thing right and and there's like lots of solder and all sort of stuff I have to do a lot of work on the board to clean it up Right well, you know how to remove BGAs in China well They built these little CNC machines that you can put your motherboard into and correct for For sort of the tilt of the board They just literally take a bit down and they mill out the chip because it was a bad chip anyways Why'd you want to want save it right and and they mill out the chip and you're left with like a clean They they mill it just to the point where the solder balls are visible So you now have the old solder those there You don't even have to stencil on new solder and you just put the new chip on top and you go ahead and you Reflow it so it's like it's like it's actually a pretty cool machine to watch if you can check out the YouTube video on it That's you know, if you have the right tools, it's not that bad You just have to have that tool or another example is like if you have your phone and you broke your screen People say oh, I broke my screen actually you didn't break your screen you broke your touch screen Right if you notice like the screen is actually okay Most of the time 99% of time you're broken a touchscreen when you go ahead and replace it They want to charge you like 200 bucks because they're going to sell you another perfectly good screen to go with your $5 piece of glass it was on the touchscreen. Well out there, you know, you don't waste any part of the cow You also don't waste any part of the phone You know if you go to China, they have these machines that this is a heated vacuum chuck plate You put the screen down and heats it up to the point where the adhesive is soft You have these things that look like nunchucks have a wire on it You pulled across and it splits the adhesive off between the LCD and the touchscreen and then now you have this You know sort of screen you have to bond your your LCD on top and the little bubbles Inside of it. So they have this little vacuum chamber you can put it into that pulls out the air and then you know Voila you have for $5 you've now have fixed your touchscreen on your on your on your phone And it's and it's because I have like this sort of the right the coin cold right equipment You can these are these are things you can go to those markets I was showing you before you just buy a cash and carry like it's like 30 bucks for the Screen mover and like a hundred bucks for the vacuum chamber or something like this cheaper than buying the whole assembly together with anyways, so One other thing is just like Another example sort of chip on board if you've looked at the backside of some circuit boards Yeah, like these black blobs of epoxy. They're not actually chips the black blobs of epoxy those blobs of epoxy are covering An actual silicon chip that's unbonded directly to the board with little wires that go to it And this is this is done to save cost because packages are expensive right so at the super super low end of things They do this to save cost and if you try to do wire bonding in the United States It's super super expensive, but then if you do it out there It's really cheap and sort of the secret to it is that if I can get this video to play Is that they have humans that just go ahead and take the chips and put them on to the board before they do the wire bonding right it's and and if you can see here there's actually a pair of chopsticks and part of a This is a box cutting blade It's actually not even like in what they did is they just wrapped it in tape So they don't cut themselves when they use it and they and they go ahead and they slice off the chopstick to the right angle And it has enough surface tension that it can serve like you know Pick and place them onto the board and that sort of thing And so a lot of people I show this to people and people go like like what it's not all done by robots I mean freaking Tony Stark has a robot to put his robot armor on so like why don't we have like you know Robots doing this sort of thing In terms of that, you know manual skills are still skilled at the end. I this is another this is another picture of a woman who's building prototypes for For me and she's just taking BGAs and putting them onto the board by hand She sort of sticks it down looks at it and then puts another BGA down sort of nonchalantly And I and before I actually saw this with my own two eyes I was like there's no way right this is not Something a human can do and then and then I was like okay Maybe I should try it right and I try it's actually she's surprising the easy to do It's not that hard to put BGAs on and to reflow them down. And so you know Not everything has to be done by robots a lot of it can be done by humans and having this big Huge base of people who can do stuff is really important to create a diverse ecosystem So if you look at sort of the the sort of ecosystem in China, it started as a bunch of raindrops, right? Billion of people of one percent of them went to go work as assembly laborers who then became technicians and engineers One percent of them become managers and designers and then now today we're looking at like oh my god Where did Xiaomi and Tencent and Taba and Ali Baba came came out of nowhere, right? You know, this is crazy China It's just they have this enormous ecosystem of people to support this Capability whereas what happened if you were to look at analogous timeline in the United States You had these engineers who we invested a lot of money in the Cold War to create because we had to fight You know the evil Russian powers or wherever it is and they built amazing things We sent you know people to the moon and stuff like this and then the dot-com boom happened And we're like well screw the soldering irons. We're going to go and make money You know selling computers and there's this sort of like decade of where we tried really hard to push everything to China And then we you know sort of this rediscovery of the maker movement happening And people saying like no hardware is hard But it turns out that actually it's not hardware the right ecosystem. So these Wonderful phones that you see we we oftentimes focus on the system integrator like an apple or a fox con or something like that And we think that yeah apple makes the phone or fox con makes the phone actually fox con Puts the phone together right and maybe they put the parts on the board But like you know there's people who do the metal stamping for them people who do the injection molding people who do The test jig people do all these different pieces. It's a huge network of Factories that feed into it that go through sort of that traded district of the of the electronics area to feed the The factory to go ahead and do these sorts of things And so you need to have that sort of rich mulch to have a you know a Healthy forest at the end of the day Now the problem with ecosystem loss is that is that if you are in an ecosystem system that has lost capability You can't even describe What things look like so you know what do what do dinosaurs look like they've been extinct for a long time a lot of time Long time we thought they were Scaly creatures, but maybe actually they have feathers right we don't know we actually can't even Draw what a dinosaur looks like we have we all have the canonical thing that we think it looks like These are Jurassic Park, but that may can be completely detached from the reality What actually did the dinosaurs work or for example like you know dodo birds right here's a picture Which one of these birds is the dodo bird right? We haven't seen a dodo bird in years We don't know which one a dodo bird is turns out the one on left is the dodo bird But a lot of people think it's the one on the right And so and so ultimately the issue with ecosystem loss you end up with the situation here Like of course everything's done by robots because that's what we see in the movies And that's our impression of how things should be done and therefore like you know hardware is hard We don't have the robots whatever it is And it's actually you just need to sort of practice and get your skills up and you can you can you can do it And of course the other problem with ecosystem loss is a few people who are playing in the area can go ahead And create a monopoly around it and create further barriers for people to go ahead and even get started So but despite all that the really good news For people who want to get into hardware is that hardware kind of doesn't change physics hasn't changed Since a very very long time. There's no like physics 2.0 That has forced me to go ahead and reformat my oscilloscope and my soldering iron And I have to like throw it all away because I'm not compatible with a new physics a bi And you know humans which is the other half of why we build hardware They also haven't changed fundamentally I mean we you know we still have two arms and two legs and we have our desires and our And our fears and so the good news is if you consider You know the study of hardware and how it interacts with humans There's a there's a limit of space of things you actually have to learn and it's cumulative It just goes up. So like if you look at the programming languages I've had to learn right basic c assembly of various layers assembly of various flavors c plus plus Pascal Pearl java python javascript rust verilog vsh of basco, you know, it just goes on and on So it seems like every freaking year I turn around and say another framework that I have to learn another language Like the what are the cool kids doing today? You know the equations I had to learn for hardware I learned in college and they haven't changed there's max with equations I actually had this compressed onto this max was a fixed first love I said be generous and that like max was as four equations gauss's law fair days and peers law You learn those and it turns out ome's law and kirk elskert law just derivatives of maxples and fixed first law which which teaches how Carriers diffuse through silicon and so you you learn those two things together now You know how to transistor works and then everything builds up from there, right? You just it's hard to learn those things but once you learn them they've served me for my entire life I had there's no max well 2.0. There's no, you know Things that change on these sorts of things and so like you know also the hardware tools I had to learn hasn't really changed much either like software tools It's like in a d system d busy box open embedded make pip app young conda docker genetons travis github All these different frameworks like you know I can't even keep track of the frameworks that I have to learn To keep a simple web server running these days, right? It seems like like every every year all I want is I want to have a web page that shows some static content or something And I'm constantly having to go back and take care of that thing You know Harvard tools I learned how to solder I learned a hot air gun That's basically the tools you have to learn and learn how to use them well I got generous and said you also should know how to use a microscope and a oscilloscope and a spectrum analyzer And that's kind of like it for the actual bench tools And then if you want to actually design the hardware there's some software packages You need to know and those things are like surprising stable too So you're going to board design pick your pick your poison altium eagle whatever it is, but it's not like You know altium Isn't good enough and you have to switch all these different packages You pick when you stay with it and they're good enough for you to go with solid works for 3d design If you want to do actual silicon design Maybe there's cadence magic and synopsis and that's that's like it like I've kind of described The world of hardware tools that you've had to learn So it may be a little bit difficult the learning curve may be a little bit hard But once you walk up the curve, um, you actually get up the mountain So it's you know hardware might be hard, but at least it's not Sisyphean, right? Right People got this one You know the old story of Sisyphus where you know He's cursed to carry up a rock to the top of hill only to watch it roll down again That's what I feel like every single time I have to learn a software framework At least when I you know do my hardware stuff. I'm like, okay today I'm going to learn how to reball a BGA, you know that That took me a day and a few bad BGAs and now I know how to reball a BG. It hasn't changed Like maybe I can do it again 10 years later So getting started the basics is surprisingly cheap. This is a snapshot of my workbench my wonderfully messy workbench in Singapore Basically everything in that in that photograph I have got on a trip to Shenzhen put it into my suitcase and checked it and carried it for about 500 bucks Including the microscope the hot air gun So that's the microscope the hot air gun soldering iron There's a little hot air plate that I use to serve as cheat mode This one here is expensive. This is the multimeter But that's because I really like lots of digit precision on my multimeter. You don't need Five digits of precision is just I have an obsession With very precise measurements. There's some cool things you can do that talk to me later But you know we that's that's that's that's for that's for me to sort of my my porn So um So and then getting and starting a hardware if you want to sort of start playing around stuff The dumpster diving is a really good way to get your hand on things and of course as I alluded to in China It's very easy But there's facilities in in this ecosystem like ebay and craigslist to have lots of stuff you can Equipment you can buy plus local universities and institutions tend to throw away a lot of really amazingly good stuff You know when I was MIT I would sort of routinely cruise the reuse list where they would throw away equipment and you would get like top-notch You know multimeters and and soldering iron to the lab just aside the grad student graduated left this equipment behind And now they're throwing it away. So guess what like you you can you can have it too So the dumpster diving cycle basically uses virtual cycle you get equipment You fix the equipment you learn how to use the equipment right then you get More equipment that might be a little bit harder But then it may be a little bit borked you fix the equipment and now you can use the equipment to fix even better equipment And so and again, it's it ratchets up. It's not like an investment that goes away It's not like I'm going to write a framework to make another framework to do another framework And then someone tells you oh we left those frameworks behind a long time ago It's now the cool thing is now whatever distro that you know pick, you know today So it does ratchet up and also in the process of fixing things you can learn through reverse engineering So this is on my blog bunny studios.com. I'm on a competition called name that where And the reason why I do that is I want to people to encourage people to look at Hardware people say how do you get started in higher or it seems so hard? The first thing you do is just look at it Right. I mean there there are patterns you can learn those things are really interesting things you can get out of it You may not understand it today But because the the the themes and motifs are consistent if you get those those patterns in your head When you see something's different your your eyes drawn to and you say why so like You know, there's this really weird white thing sticking out here. This is actually picture of the inside of my five-digit multimeter And and that's that that is that is the that's the really ultra high precision resistor they use for for setting as a reference standard for for For the measurements and so and so you see you're drawn to these things you look it up and you learn about these things And you say oh that's a really cool technique That's how they do it and then now it's another little fact you put in your head You hang it in the tree and it stays there and it doesn't have to go away or go bad The next thing is there's the fear failure right people often like but what if I break it right? What if I avoid the warranty? What if I you know this sort of stuff the thing is the truth is I break hardware all the freaking time right? So if I'm going to hack something my rule is I get one to break one to hack and one to test right and so You you want to get a piece of hardware that you can just completely trash take all the parts off Whatever it is at the other day So if I'm going to hack a phone or something like this I'll I'll go to ebay find a broken one and then I'll just start taking the parts off like you know where You know where are the parts that I shouldn't be putting my probes that would break things right? Later on down down the road oftentimes all you need is just one sample like that and you just you just completely trash it Right you learn a lot doing that so now now that you've you've learned what not to do You get the piece that you actually do the hack on right? So this is this is back to the piece that right installed that That motherboard that the tap board inside the xbox So when I had the xbox I had one xbox from a broken one that I took all the parts off of to Figure out and I measured where the trace pitch was and then all I had to do was build one On the xbox I actually had to hack it and then I could always find a friend's xbox To just check to make sure things are going well if things weren't working on my side So and it's important to have one to check because you know a lot of times It's easy for you to think that you're crazy or you're not you're not thinking straight You just want to say okay. Is it working or is it just me right and so there's a rule of freeze for hacking one to break So serve minimize that fear of failure the fear of loss one to hack We do the experimentation on and one to check sort of the base a baseline to ground your experiments for making there's a kind of a similar Pattern to it. So you know you look at a phone you take it apart. You're like wow it's like so Immaculate and put together and so perfect like it seems Impossible to build these things of such complexity. It turns out they it's a multi-stage process too Phones start not as tiny little phones. They start as really big boards This is an example of a qualcomm dev board. This is just a bezel with a screen, right? And then this is the actual cpu that's on a card that you can swap in and out in case you break it Or something like this or you want to change it to another one And you have this huge array of connectors for yourself to plug in diagnostics and peripherals and all this sort of stuff So, you know, there's a multi-stage process to making these complicated things as well There's a in the industry. There's a there's a there's a terminology for it that ebt dbt pbt process So vt stands for evaluation test the e is engineering. So your first step is does it catch fire, right? If it doesn't catch fire, then you have a design and you say it does design meet our requirements, right? It meets requirements. Great. Let's try to produce it does it yield, right? That's sort of the three basic stages And the observation is that like the bill materials stays 80% the same through all those stages ideally and hopefully you're not like You know at stage two and a product manager comes and says actually we need to like put a you know A coffee maker inside your phone. So now okay Great throw everything away start over again. So that accepting, you know, sort of dillbert mode Sort of changes things don't change too much. So usually what you do is you buy 10 or 20 parts at the beginning and keep in mind that parts are cheaper by the dozen So you're not actually buying You know 10x or 20 it's the amount of cost and a lot of it's a setup fee And so if i'm if i'm going to do like an iot style like you know single board You know embedded microprocessor kind of project at the ebt stage I'll build a minimum of three because you want to have one to break You know one to test one to check right so that you're actually kind of at the very first stage You're hacking your own design because you it's not working Okay, what I do. Okay. Well, let's try to reverse engineer my thought process and figure out and You attach the jtag debugger and you finally get code loading and you're actually kind of owning your own hardware at the very beginning because it's not booting So that's sort of like the it does a catch fire stage And then once you to that point you want to build a few of them So you can start handing them to other people to you know Check your design assumptions and also put them in you know machines to bake them and shake them and then To the production side you want to build as many as you can afford So you can like give it to friends and actually do like a soft beta to see how things are going So this is some exact actual examples of how this comes along. There's a product I made A long time ago called the chumby. It's basically a squishy little internet of things alarm clock At the end of day. This is what it looked like at the end, but it started as like this large Board on my desk and then to see if it would catch fire Literally we we've got some squishy pillows and some hello kitty things and cut them open And then like put the circuit boards on the inside and ran them for a long time So it literally doesn't catch fire was like the the ebt stage Of of the of the project, right? Okay, great. It doesn't catch fire Let's go ahead and try to refine the design and stick into something and then we figure out like actually the You know this original design was bad like the electronics would fall out There's all kind of small problems and then finally once you hammer out those kinks We get to the production stuff which is end up being photographed and people You know, that's what people talk about But like there's a lot of the but it was a process getting to there, right? And so well-defined process and the key thing is it doesn't happen if you don't try You just need to sort of get started and and give it a try to make it happen so another example of a Project I work on is chibi tronics. So chibi tronics is a project where we put electronics onto paper It's sort of an Interaction of sort of physical matter and people to get see how people can learn and and create differently if they have electronics integrated with paper It kind of looks like this you have stickers that we can put down onto paper And we have like clippable microcontrollers and go on the edge of a page And you can like then draw on top of it do art that sort of thing So it you know, there's a lot of problems that we had to solve getting to that in the very first prototypes We're literally just like a large flex pcb Then we just put everything that we could think of it on it like just stupid stuff We had like accelerometers and microphones and boost regulators and battery all the sort of stuff right And and huge chains of high power leds and we saw like what caught fire and what didn't catch fire What worked and what didn't work and we sort of winded it down to a space of things that we could reasonably produce And then for the clippable paper stuff it literally started with a paper clip that we like taped Up board on and then like stuck some foam behind and said oh, hey, this might work, right? You know that's that's the evt stage that's where it all starts We just have to try and see how it goes and then we matured it to a dbt stage where we did a small crowdfinding campaign and some early stickers and some 3d printed models to sort of refine the clip And then after about a year of effort we end up with these I'm sorry what you see now like on on amazon or whatever it is of the actual product It you know looks finished and refined, but it but it's not you know It's not like it was just born that way or another project that i'm working on called any tv2 I mentioned the earlier any tv which sits kind of the intersection of other computers in society Society says that it's bad for you to break crypto. That's a crime today. You're not allowed to go ahead and Circumvent crypto to access copyrighted materials and a tv2 tries to Challenge that notion And so the basic idea of any tv2 is that if you have a raspberry pi you have an application You want to show on your tv you can go ahead and just you know plug it in line with your cable on the tv And then whatever shows up, you know on your raspberry pi is now like a little window on the side on your tv You can do sort of like a video overlay and the reason why it's a sort of Sits on the intersection of society is that there's actually a law the digital million copyright act which makes it illegal for you to go ahead Play around with the video feed So if you want to do something even really simple like what you can do with your phone where you want to go ahead and do Live video translation of of text on the screen or something that that's actually not lawful to do right now On a tv feed because those things are encrypted So when I first started the project, this was the ebt one I thought maybe I would build something in like a little pc a mini pc express card form factor And I floated around and again, you know, did it catch fire and built some stuff for it? It turned out it was I completely missed the mark Like went back to zero again and ebt two looks a bit more like this It's actual full-sized pc i card because that's you know where people actually had the power to do the video And where the interest was in the market Um evolved it into a dbt Prototype with a 3d printed taste and the raspberry pie mounted on it to see you know And we're that's currently actually at the stages right now What the pbt will look like will actually be the outcome of it's actually an active crowdfunding campaign shame is plugged if you go to crossplay.com alpha max slash any tv2 You can check out more details about this project But the but the idea is that like you know based upon the feedback of people the users and what people want to do with it The pbt will come into shape. So this is actually that process happening live And you can participate in this process through the magic of crowdfunding So, you know the high level sort of message i'm trying to get out here is that you know hardware is interesting And hardware isn't hard if you know the method right so just you know hacking as a methodology you get a couple pieces One to break when to hack one to test it's not it's not scary If you wanted to do making there's also process like does it catch fire right doesn't meet specs and can it yield And if you can Hack hardware you can go ahead and bring new capabilities to the things you can do whether on the security space If you want to go ahead and pull out keys or if you want to extend the interaction and capability of a computer Um, and this ultimately is sort of like the core of the hacker ethos. I think when I was thinking about this talk It's about redefining your volume of possibility It's not being caught in that sort of infinite fractal surface area and thinking wow like but there's so many things for us to look at Let's just change the rules and change the volume of possibilities So you have literally a blue sky of ideas that you can go ahead and explore and that's my talk enjoy the camp Okay, thanks everybody. I really look forward to hanging out with you guys and getting really trashed tomorrow soldering It should be a really good time Oh, that's right, I think they wanted to arrange like a book signing or something like that. I don't I've never done with these so You're interested Sure