 The talk for 12.30 will be starting as of now. So please give a round of applause to Lee Brotherston. I dread that for the first time today. Hey, thank you. So this is, well, it's my glorious celebration of IoT security as you can see. These are the objectives I kind of want to cover today. An introduction, which in part is this. Then we'll go into some challenges of IoT, why problems occur, and about how we're doing a security. So the background on this is very much that as I was showing a minute, there are certain IoT talks and IoT types of talk that come up frequently. But I think that very often the root cause is not always addressed. So this is aiming more to talk about the root causes and how those problems get established in the first place rather than popping shells and actually demonstrating exploits. So who am I and why am I talking about this? Well, I'm a security person, not an IoT person by trade. I typically work on the defense side of things. You know, day job is being a defense person. But more specifically, I'm a security person in an IoT company. Specifically, Kobe. But this equally applies to things I've learned from sort of industry type events and that sort of thing. And really, this is a reason I want to talk because I think that a lot of IoT companies do not have security teams and a lot of people that give talks or conferences are not from IoT companies. So I feel like it's hopefully a unique perspective. Apologies if it's not. So at the minute existing talks to me fall into a few sort of three broad categories a lot of the time. And that's great, but I wanted to do something different. The categories being, hey, like there's an old thing because IoT devices run like Linux from five years ago. So here's a redoing the five years ago Linux exploit talk. There's no more web app stuff that just happens to be on an IoT device. Or the common one, which is the so with physical access to the hardware, I can do a thing. And these are legitimate in a lot of cases. But if we're talking about threat modeling, you know, with physical access to big, some big industrial thing, like that's great. But if you've already broken into my house, I don't really care if you can reflash my DVD player. That's like not really the threat model that I'm going into. So got DVD player showing my age now. But the point is that that's often not a threat model for most people. IoT maybe not exclusively, but for the most part we're talking about consumer devices in a consumer home. We're not talking so much about industrial control. We're not talking about production line. We're not talking about like airlines and public transit and all that kind of stuff. So because of that perspective that a lot of people take, it often focuses on the impact of the hack rather than the root cause. I know I'm generalizing. I'm not meaning to shoot anyone down there. It's often largely presented by people outside of IoT just because there are not that many people in security in IoT. And there's also guesswork because people are, by definition, they're having to reverse engineer things. They don't work in the companies. They don't have access to the source code. They don't have access to schematics. So they're taking devices apart, dumping firmware, that sort of thing, which is awesome. But it does involve a certain level of guesswork and inference. So we've seen the headlines. This mostly covers the sort of stuff we're seeing. I'm sure everyone in this room has an interest in IoT and so has seen these. Although these are slightly dated. So to revamp my slides, these are now the headlines that we see, focusing very much on the Alexa's and Google Home's recording things and oh my god privacy, et cetera. A pretty good base for the main problems, and not a lot of people seem to realize this, is the OWASP have done an IoT specific top ten. And it's actually pretty good. And based on what I've seen both within my own organization and other organizations, I would say it's a fairly representative and characteristic overview of the core issues. I'm just going to switch the slide because this was a screen cap of their pretty graphic. This is the same top ten just in text you can read. You probably know them, but we're just going to super quickly skim over them. So weak hard coded or guessable passwords. I'm pretty sure you'll have seen the talk here already today where some hard coded token can be reused. Insecure network services, so that's like old versions of demons, old HTTBDs and stuff. Insecure ecosystems as a whole, because most of these devices, I'm guessing, when you buy them you're not using an isolation. They often talk to a mobile app to some like web service or something. Lack of secure update mechanism. This is a big thing because IoT devices often don't get updates or they've got a terrible update mechanism when they do. Insecure and outdated components. I believe this is software components. We're not really talking about outdated fastest and such like. Privacy protection is fairly non-existent in many, many areas. And the same with sufficient data transfer and storage protection. Device management is mostly referring to the fact that you can't adequately do things like network config, being able to set passwords, being able to set cryptography appropriately on Wi-Fi connections and that sort of thing. Insecure defaults and lack of physical hardening. Like I said before, physical hardening is very much threat model dependent, so I'm not going to delve into that one too much. But really the crux of this is like why. So there are a few areas and I'm just going to delve into each of them. I've broken it down into sort of four broad topics. So first of all, there's the company. I'm going to get a little bit more into detail on that, but there are effectively three types of companies that make IoT devices right now when we're looking at consumer IoT devices. So you've got dedicated IoT companies. The companies where their core product or their flagship product lines are IoT devices. That's what they do. You've got tech companies who often make larger sort of commercial grade equipment. So think, you know, big network provider who's like, I'm going to do a consumer offering, make a smaller, simpler IoT device. And then you've got a third type, which is a traditional player in some other market. So I'm talking when a lock manufacturer decides to do IoT door locks or a kettle manufacturer does an IoT kettle or whatever. It's like tech is not actually their product. They're sort of diversifying into that. I'll come back to that. There's a product development lifecycle. That causes a bunch of issues just by the very design of how it has to work. And we'll go into that. There's actually some logistics issues. And finally, second to finally, there's economics, so I know this is not really an economics conference for the rest of the economic reasons. I'll touch on that. Hopefully it'll enlighten a little bit. And then there's just straight-up technical stuff. So to look at the third company type, the people that work in a non-tech company that are diversifying. So this is a real photo from Dave's Toaster Manufacture Company. They make toasters. And they realize that, you know, they've got to diversify their market a bit. Toasters are kind of for old guys. And they want to broaden out. So they start thinking about the millennial market so they can get their avocado toast on. And they think, like, OK, so how can we do this? Well, we know how to make toasters. And we need to appeal to, like, the kids. And, like, I hear that guy in IT can do an internet. So we can probably, like, stick these things together and make an IoT device. We should do an IoT. Oh, we should do an IoT. There you go. So that's cool. We know how to do that. We'll kind of, like, duct tape a Raspberry Pi to it. Or some similar, some Arduino or whatever. It may be prototyping, but I've seen things that end up literally like this for real. Yeah, and then we'll just apply some duct tape. And, like, it'll all be fine. And we can, like, enter into new markets and become ridiculously rich. So what does that work out like? Something like this, probably. But it's not so far from the truth, worryingly. And at least they've taken it a little further. They've put an Alexa on there. So voice activated. Awesome. And of course, that's never caused a problem before, ever, for anyone. There is the incident of a door locking manufacturer who didn't really know about uptime and didn't really know about whether to fail open or fail closed. So the locks just stopped working during an outage, which was amazing. And obviously they haven't thought through issues, partly safety and legal issues as well. Like, do you want to fail to unlocking everyone's door or locking everyone in the house during a fire? Neither of those are sort of great situations. But yes, that's how they work sometimes. This is from Tesla, where you can't drive your car because you need an update. Again, you know, probably not exactly the user experience that people are looking for. And yes, this was the backlash from Yale specifically from people who are not super-duper happy about the situation when they couldn't use their door anymore. And so how does this stuff happen? So this is actually the hardware lifecycle for people that work in hardware design and manufacturing. So the little loopy bit at the beginning is the closest you will ever come to agile in hardware. This is where people are sat in a lab and they're taking components off and on and off of a breadboard and like moving wires around and that sort of thing and iterating on their design to make a hardware design that works. And then it naturally goes through a few stages. People make real PCBs. You know, they 3D print a chassis. They make little prototypes that are good enough for you to test functionality in-house, but they're still not quite the real thing. And they go through this loop until you land on a fairly, fairly solid design. At that point, you go into EVT. That's engineering validation and testing. So this is the hardware equivalent of functional requirements testing. You're going through and saying, does it do the thing that we designed it to do? And you're also doing things like getting certifications because hardware manufacturers don't just have to make it do what it says it does. Unlike software, there are things like electrical safety standards testing and thermal testing and RF emissions testing and all that kind of thing. So that all happens during this phase. And what that means is that there are certain changes that will have to happen to hardware and certain things that cannot change in hardware because you have to meet these requirements or fix a shortcoming. What that means is that you effectively, from a hardware perspective, have what is like a software change freeze. The reason I mentioned this is that until this point, people don't like developing the firmware. So the firmware development lifecycle is not as long as you may think because during this lead-up point, hardware is changing. If you're trying to write device drivers in software and you're constantly changing it because someone's like YOLO and you're just switching some component out that no longer, you know, different chipsets and stuff, it causes all sorts of havoc with that. But then once you've frozen it, then the reverse is true. You can develop all you like, but if you find a problem in your software that requires a hardware change, good luck to you because it's not happening, certifications have happened, things are not being retested. Then there's design validation and testing. This is a little bit more like QA. And then PVT, which is product validation testing. So that is, sorry, not product, production validation testing. So that stuff coming off the line that gets tested to say, okay, the samples worked. Are they coming off mass production looking what we expected? But what that means is that firmware needs to be developed after the hardware is constantly changing, but be on there and working before you're testing stuff coming off the production line. So you've actually got a relatively narrow gap during which you're shopping out firmware because firmware has to be on the device when you're shipping it to consumers, and then you're into mass production. And so like I just said, building hardware is a few areas. So there's PCB fabrication that's physically making the boards, sourcing and shipping components, which if you've read any headlines on supply chain stuff, that's its own part, bit of fun. There's assembly, which is often completely different plant. Building quality testing is literally just like, well, exactly what it sounds like. Making cases and accessories, so that's the pretty touch screens and the boxes are in and everything, and then packaging and shipping. So getting a little bit more technical, but not super duper technical. Hardware actually changes what language you can use. So I'm sure you're all aware you don't run massive like 32 core X86 processors in IoT devices. They're normally ARM or MIPS, for the most part. There might be some other stuff, but that's predominantly the architecture. That leads to various constraints, not just in hardware, but you've got processing power. You've got things like the amount of addressable memory. You've got the cost of getting the memory into the devices and all these things. So you're very often resource constrained. Additionally, you're probably running some embedded Linux, like a busybox or whatever, which actually writes their own OS. Everyone takes a base OS and tweakifies it a bit. Same as in big vendor land where they inevitably take Linux or BSD or something and just tweak it. So then you've got the resource constraints, which means you've only got so much memory and so much processing you can do, which means that things like encryption and not writing things to volatile storage and that sort of thing gets a lot more difficult, which in terms of things like managing keys and that sort of thing is also challenging. And I'll come back to that in just a sec too. You've also got kernel drivers and black box drivers, which is super fun because some chipset manufacturers, even when you're putting their chips into your hardware yourself, pull the thing that any of you that run Linux with a graphics card are used to, where it's like, source? No, binary driver. Off you go. Good luck. So you end up running bits of code that you may not even understand yourself on your hardware. And as I mentioned, key management is hard. It's like super-duper-duper hard. And if key management is hard and you're used to key management being hard, IoT key management is harder. And why is that? Well, it's what I just mentioned about the hardware lifecycle, right? So how are you going to ship the keys? Am I going to say, I'm not putting any keying material on devices? Or am I going to say, I'm going to put a shared key on device and I have to give that shared key to some fabrication facility that I don't own in God knows where. Are they going to have that key and put it on the devices? Or do the devices get individual keys, which we know is like way better all the device is having individual keys, but does that mean that I have to give them some kind of root set to sign all the individual keys? Because that also sounds like a fun idea. Or you can go the tofu root, which is the trust on first use, devices do not have any keying material at all. And they just set on first boot up to go grab some keys, which is great because you've not exposed your key material to an external third party, but it is a little bit of an issue in that when your device is plugged in, it has no keying material. And then you're at risk of things like if things are man in the middle of first use, you know, and that sort of thing. Next up is software supply chain. So as you're aware, these are typically like your Linux boxes and they run some processes that rewrote, they'll run processes that are part of the OS. They'll probably run some random open SSL version or whatever too. And what you start to realize is that you have inadvertently become an OS vendor because you took out that base Linux and went like, that's great, but the open SSL is out of date. I'm just going to update mine. Suddenly you're no longer using the base version. You can't just be updating the base version. You're updating whole firmware images. Also, unlike things like Mac OS or Windows or something, where you're used to vendors pushing updates, if you compile, build and install an embedded Linux system, most of them are not doing OS updates at all. Package management, nothing. Which means that the IoT vendors have to do those updates and package it in, which is sometimes I think a thing that people don't anticipate, that they are now becoming like a small OS, but a whole OS vendor. And then you get stuck in the dependencies route. Another fun thing, if any of you are developers, is water agile full development. Because most IoT companies are like these young startup like agile is awesome, micro services and AWS and everywhere. And that's great. That is a great way of doing things right now. However, you can't modify hardware on the fly anywhere near as easily. The closest you can come is re-flashing firmware. There is no, hey, would you mind yanking your thermostyle of the wall, switching these chips out? No, you're great and ready for the next version. That's not going to happen. You somehow have to manage agile development at the back end. So like APIs, mobile apps, all that sort of thing. With what's effectively waterfall methodology for the actual hardware itself that this is running on. Which leads us into the pain of logistics and this is actually where I think a lot of IoT companies get their issues. So you have things and they are in places. Those things being the IoT devices that as a company you're shipping to consumers. And then places are places. So there's the factory they're built in. There's things like the warehouse because we don't all ship direct people. They sit in like Costco or an AWS warehouse or some AWS warehouse, an Amazon warehouse or something like that. Because consumers want to buy from a variety of places not direct from vendors. This poses a small issue because the time to thing being online is unknown. That means if I write firmware that has an inherent vulnerability in it that needs to be fixed by disabling something on it or killing an API endpoint for me I can't do that because there may be some device in a warehouse that's not going to be plugged in for six months. If I kill the vulnerable API endpoint and then someone plugs it in in six months and it's a brick I am going to be shot by someone. So you have to keep that online. You have to find alternative methods of securing it. Similarly devices themselves we send firmware updates to them and everything but on first connection they're still going to be running that six month old version of firmware was that was put on at the factory. The other thing is you don't know the geography of the thing at the time. By that I mean if you have various methods of updates they don't all work everywhere. So for example if you're putting things into a corporate environment there's often things like proxies and firewalls and stuff in the way. When you're getting big commercial gear put in there isn't a whole easier requirements for opening up your firewall that people read. Like consumers plug it in and hope it goes. If they're blocked they're blocked and we probably aren't going to get updates to them. And speaking of which if anyone remembers what this is so that's about as frequently as users will update if you don't make that automatically happen for them. So if things are not updating on their own then all those previous problems I mentioned they're just going to stay like they're not going to change. And then people like Bloomberg pull this stuff which was fucking annoying. So as a heads up if you start working in an IoT company and this headline comes out as surely afterwards you're in for like the least fun week of your life but effectively this got debunked not debunked that it could happen but debunked that it did happen. There are a couple of talks out there that are really worth watching if this is an area you're interested in and the TLDR is this is like supply chain attacks like fabrication plants, switch chips, oh my god the NSA's or Chinese government or whatever panic. And people proved that they could happen but FalconDarthStar did a talk Shmucon which is really good that shows that there's probably a bunch of other areas in the supply chain that would be way easier to achieve this with and actually some of the QA checks catch way more of this than you would think they would. So on to the economics part and the Bordens this is because we're a Canadian company and we don't have Benjamins so this is our equivalent so you've got the retail price of a piece of hardware so whatever just imagine a number. You're going to take some taxes out because we have to pay some taxes the retailer takes a chunk because they don't do it for free. Shipping and packaging take that out at the price take out the physical parts and the labor to build it recouping some R&D costs because of course you're not making any money whilst you're designing a device you only make money when you sell it so you've got to recoup those costs there's marketing because it's all pretty blinky ads all over the place there's Facebook banners to pay for and whatever else and oh yeah hopefully some profit at the back that's like you fix costs if you're a subscription service you're lucky you can probably roll the rest of your costs into the subscription but if you're buying a device where you buy the device and that's it the following also has to come out of the cost of that device oh sorry wait no subscription revenue so one tech support you probably want to be able to call someone and get support those people need to be paid as do the phone lines customer service for complaints because people love IoT and then you've got replacements and returns those cost money hosting bandwidth and APIs and all that stuff in AWS building the free mobile app you're planning to use and the free online portal and then having deducted this entire list and the profit that the company probably wants to keep with what's left you can patch an update and do for your future development your security team you can pay for your pen tests and your bug bounties people sometimes wonder why security work doesn't get done this is actually a large part of it for people like you have to pay for that stuff and there's only so much and if you want to do more you're going to jack up prices or cut from somewhere else so let's quickly talk updates because I've just seen the time this is BMW's update process as published on the website how many of you do this with your car regularly I'm guessing probably close to zero maybe when it's in the shop or something so let's talk updates ideally you want to do over the air signed updates that can't be rolled back to previous versions so people can only move forwards they know it's safe from the vendor and it's over the air and works and great PS that's what we do yes a lot of the time that's not the case often it's over the air manually initiated and signed if you're lucky over the air manually initiated you're probably used to this with wifi access points or something it does the other updates fine clicky clicky there's manually uploaded binary which is the case with that BMW a USB stick none a lot of them fall into none sometimes devices are end of life and they just don't care sometimes they haven't thought through update processes but not thinking this through it's really hard to fix because how do you put a new update service on something that doesn't have an update service it's a common pitfall anyway and speaking of updates updates can go wrong you've already seen these in your life somewhere in IoT bad updates are bad number one it's non-tech users that get them right it's often people who have no awareness of what's actually happening under the hood so they're not going to fix it if it goes wrong devices are often a brick it's not like windows or something where it goes hey would you like to go into safe mode and fix it mode or whatever else like a brick is a brick and when that's something like a door lock a thermostat, light switches like people don't like brick devices for some reason it also makes agile hard so when people say it takes a long time to fix vulnerabilities often this is why there's like a ton of testing because you have to test the upgrade from every version of firmware to every version of firmware on every version of hardware if you don't things just stop and that's bad and there's also we have to be conscious of the environment so we recycle refurbish and rma rma's returns this also makes it hard because we have to handle the process of being able to get new devices out to customers and not finally almost finally getting finally so there's no friction deployment you will know that like the magic if any of you bought apple especially the airpods for example it's magical you open the box and your phone goes like ah you bought a thing use a thing like that's the end of it that's what like everyone shooting for in this department they don't want people to be like oh and then I have to cable a thing and then plug in this other cable then push this code blah blah blah like people want to just go like oh I got this cool device plug it in works done great so it brings in like this box and you see these triads all over the place in security you can do all these that's great absolutely no chance of getting all three nailed it doesn't happen and so you have to pick which one which one is removed and for many people that is security I would like to think not us but it is often the case but it's not all bad some companies are waking up some companies are getting abysmal press and being forced to wake up and public awareness is moving up which is forcing the hand of people so in summary everything's fine don't worry like it's all good I am like two minutes away from being done so like thanks for your time and if there's any questions I would love to take them otherwise just thank you I'll take it as no