 Hello, everybody. Thank you for hanging around to the end of the day. We hope to give you some light entertainment in a security space with IoT. For those of you who are more experienced in this field, you may not learn too much, but we hope that we can give you at least some insight into some of the wildness that is going on. And with that, hello. My name is Andy. I'm a co-founder of Control Plane, which is continuous build and security for cloud-native stuff. You could say I'm a technical milliner and an advocate of continuous everything. Yes, continuous security practices with a focus on containerized deployments. And this is my dear friend Victor. Hi there. My name is Victor Peterson. I'm an entrepreneur, a geek, tinkerer, built a few companies and considered myself more of a jack of all trades with some security interest. And we will touch a little bit on some of the things we worked in the past in this talk, most notably, Screenly, as well as Watt, which is the latest company that I'm working on right now. And with that, we're going to dive into a little bit of what is, I'm not going to bore you with a vendor pitch, but essentially we tried to solve the DevSec ops space, gave me five security, and help with visibility at large when it comes to security of your workload. So with that in mind, let's dive this back into the IoT space. So IoT security summed up fairly well by this single slide, I think. And it essentially is a matter of time before your IoT device becomes part of a criminalized botnet. That is essentially what IoT security is in this current day and age. And there is another definition of IoT that I really like as well from my good friend, Dom, which is, the Internet of Things is a science project, creating the most complex way possible of turning the lights on. So let's dive a little bit into some of the use case for IoT and the current state of IoT, really. So IoT, as you're all seeing, has a lot of failure points. And here is a Nest Termostat updating, and because it's updated, you can no longer access your Termostat, which is kind of annoying. Here's a real-life scenario of myself. This is outside my building in London, where I was locked outside for 15 minutes because the system was updating. This is not how you should be doing IoT. However, you could argue that's still a lot better than this. I'm not sure how many of you saw this, I think, this leak last week. It was a brand of D-links, D-links routers that they found a massive security mobility, and yet D-link refuses to release security updates for these. So when you start to connect everything to the Internet, all of a sudden there come some drawbacks that people have not really thought about. It dramatically expands the tax surface of all your devices, really. And most of these devices were not even built to be connected to the Internet, so nobody had thought about that. And devices do become obsolete insanely quickly these days. And the problem largely is, and we've got to dive into this a little bit longer in the talk as well, which is there's a misalignment incentive between the vendors and the end-users. So at the very best, security in IoT is at the very best and at the most the best of the best. So instead of having security by default, we tend to today have security or insecurity as a standard. And this is why we see these things. You're finding devices running Telnet, which, why you would do that in 2019 is a big question mark in the first place. But these are becoming fairly common. Again, you have another one where IoT, massive thought of IoT devices being exposed to the Internet, and obviously being vulnerable to the Internet, they were never thought about being secured in the first place. So this is a screenshot here that I really enjoy, which is long day, long before showdown. There was something called VNC roulette, where somebody scanned the entire public Internet for VNC sessions without authentication. And then they threw up a web interface for this, so you could basically browse your way through all the VNC sessions. And in the screenshot above here, you can actually see a power plant that was publicly exposed over VNC without authentication. And that was just one of the many sites that they discovered that were obviously not intended for public exposure, but yet they were exposed because people thought that nobody would be able to figure out their IP address, because I don't know why they would think that would be a good idea. But it's pretty scary nonetheless. So with that, I'm going to hand back to Andy for a little bit of an overview of the talk. Right. We're going to walk through some IoT failures and look at the state of the art, talk about how to build better IoT devices with containers, cover botnets and brickabots for those of you for whom that is a familiar term, and take a glimpse into what the future might hold. So the Internet of shit, just how bad is it right now? Very bad indeed. In 2015, security researchers were able to take total control of a Jeep SUV remotely. They used the vehicles CAN bus, the internal network that allows the bits to talk to each other, and they exploited a firmware update vulnerability. Now, happily, there's a lot of work going on in this space, but at the time and for the amount of time it takes many of these cars to be end of life, this vehicle will still be a problem. So the vehicle was hijacked over the cellular network, and they discovered they could affect the speed, slam on the brakes, and even veer the vehicle off the road. The FDA confirmed that since Jude's medical have implanted cardiac devices with remote vulnerabilities. These devices, like pacemakers and defibrillators, are used to monitor and control patient's heart functions and ideally prevent heart attacks. These vulnerabilities allowed remote access to the devices and once in, attackers can do nefarious things like deplete the battery, and incorrect, well, and administer incorrect pacing shocks. The vulnerability was in the transmitter that reads the device's data and remotely shares it with physicians. Does that sound familiar? It could well be the plot of a thrilling homeland security based television series, but it's not just vehicles and wearables that present this attack surface. There are also devices around our homes and businesses that are now internet connected by default. Locks are another prime element for exploitation. They can be attacked in many ways and should have a secure update mechanism, but we've seen plain text passwords, we've seen signal replay, and plain old man in the middle attacks on these devices. It is probably safe to say smart locks are not a good idea. And webcams are increasingly exposed to the internet with insecure or unapproved passwords. I suspect this guy just saw his webcam move autonomously. I hope he'll take it offline immediately. These things pose both a security and a privacy risk. And of course, while it is always amusing to see the blue screen of death, who knows why this device spews, it could just be a hardware fault, but we shouldn't be displaying this information to our users. So how do we build our IoT devices and how do we expect them to run? Well, we think they're existing in harmony with all their surroundings, the flower power internet's mentality, but what actually happens at runtime? They are in the most hostile of environments with nothing to save them, but their author's resilience and hopefully a patch update cycle. Because these always have to be done, barely secure, hardly updated devices make prime targets. So how do we solve this problem across the internet for the whole of mankind simultaneously? Victor? Thank you, Andy. Well, that's an easy one. Of course, we're going to use the blockchain because obviously the blockchain solves all our problems. So therefore, that's the solution to all our problems for security and everything. That's the end of the talk. No, just kidding. Blockchain will not solve anything related to security whatsoever in this regard. So let's move on to what the real solution is. And let's take a, before we do that, let's take a step back and look why do these devices get compromised. Andy already alluded a little bit to why these devices get compromised by mentioning default credentials, which is a good one. But there's poor to non-existing update cycles usually to these devices. It relies on the user applying updates, which we should really know by now that that's not a good approach because users tend to not care about that. So we need to think about that. And that's one of the biggest reason why a lot of these devices gets compromised. We see a lot of insecure services that really should not be used in 2019 to start with. Telnet, FTP, really. You see, even with these Linux-powered devices, none of what lit up the security features are actually used to confine the problem. They could be actually used, but honestly, a lot of the problem is that vendors of these devices tend to not apply common sense. This is not rocket surgery. A lot of this stuff is well-known and well-documented in most of any respectable engineering framework, right? So let's take a step back again and look at IoT devices for the service. I guess this is a bit controversial because some people, particularly in the hardcore embedded world, would disagree with this. But I would say that we see a lot more IoT devices being more powerful, right? So we see more and more devices running Linux, even if it's a lightweight version of Linux, we see a lot more of Linux running on N devices, which means that they're really general purpose computing at the end of the day, a large degree of the IoT era hardware with, except for perhaps, battery-powered devices where you definitely need to think about the power draw. And that means that we really started to become, the line between servers and IoT has become much more blurry than it used to be. So how do you secure a server, right? So server security is not rocket surgery, either. There's a lot of common sense, a lot of good documentation out there. You should not be running, you should not be, you should know what servers you're running, do you need to run them in the first place? Which of these servers are exposed to the network? Those are things you need to check for. Are those services running as the least privileged user? That's also something you should really be looking for. Don't run everything as routes. There are also plenty of kernel level technologies that help you isolate these processes that can really help you reduce the blast radius if a given service were to be compromised. Then you have obvious things like encryption in transit to remove possibility of sniffing the traffic. So you can actually secure the transit as well. Other things like firewalls, you have CVE scanning, all those things that we need to make sure that are actually taken care of when we have a server up and running. This is common sense for most server operators, but it seems to be lost for, well, even still, there are lots of some users there as well. So how does it transition into the IoT world, right? If we assume the hypothesis that IoT devices are roughly Linux devices, not everything might be true about that, but to a great degree and with Moore's law, they are converging more and more. So the security for IoT is not that different than security for servers. A lot of the same technologies can be applied. So we just need to make sure that we apply the common sense from server security to IoT devices. So just a shameless self-plug here, that's actually one of the things we do in what. So if you want to learn more about that, hit me up afterwards. I'm going to go a little bit now into a little bit more controversial areas, which is containers on IoT. And people would argue that, well, you can't really run containers on IoT, but with Moore's law, we're going more and more close to that. It is actually viable due to a great degree today already. And you start to see more and more push. Not two particular vendors that are driving this a lot, which is Ubuntu and Ubuntu Core and Balena are allowing you to do this really well, not for the lowest power device again, but for slightly more higher power devices. So I'm going to do a really quick intro to these two technologies or these two platforms and show you how they work. So let's start off with Balena. So Balena's tagline is get push Balena, and that encapsulates pretty much what they do. They have a build farm, you have a Git repository, they push to their build farm, their build farm that pushes those Docker images out to the edge devices. So you have your dog edge device paired to their platform. And they've done a lot of work around optimizing Docker or Moby rather for IoT use cases. So they've thought more things, thought along lines of how you reduce the bandwidth as well as how you reduce the failure point when say somebody would pull the power on the device when it's actually running an update. So they've done a lot of work around that and in their estimations, their images that they generate from Moby, sorry, Balena engine that's called these days, is somewhere 10 to 70 percent smaller than the images generated by Docker. So that makes it a lot more IoT friendly. You can run on Raspberry Pi or even more low power devices along those lines. And here's a really quick demo how it works. You make your changes, I think it started from the start, yeah, you make your changes, you push them to their repository, they perform a complete build of that process, all using Docker images and Docker compose files. And then at the end of that process, they are then pulled back to the actual devices. It doesn't matter if you have one or a thousand devices, the workflow is the same. Now, we're running a really fast speed here, but the next one is Ubuntu Core. Ubuntu Core is a complete separate version from the regular Ubuntu classic, as it's now referred to. Some of you may have heard of it as snappy before, but it's now called Ubuntu Core. It's a completely different version than a regular Ubuntu. And it relies on something called Snaps. And Snaps are a package format for containers really, or a transactional item. So it's replaced the Dev packaging. And it allows you to do a lot more things that you couldn't do in a regular system. So even the operating system itself is running as Snaps. And it is completely self-updating. It has a lot of mechanism for diving into how you want to configure the workloads. So if we go in here, we actually see how it works. They use a tool called Snapcraft that generates a Snap file for you. You then have an app store that then allows you to pull this down to the actual devices. So they can be self-updating both for the operating system level as well as the application layer. And if we go in here a little bit more and zooming in on the details, everything in the entire system is a Snap. So they have transactional updates. So if you were to abort it somewhere mid, it will just roll back. If you reboot the system, it will take care of all that. All the updates are cryptographically signed, so you can't temporary them in transit. The default permission for a Snap is nil. So you need to explicitly grant everything to a given Snap. So you have to say, oh, I want this Snap to have network access, or I want to be exposed to port, or talk to interface A, B, or Z. So that's how the Snap system works in really quick terms. And with that, I'm going to dial in the hand of the dandy for a little bit more zooming on how this works behind the scenes. So in this example, the foo Snap plugs into the Quax interface and passes North service, the Quax slot. I hope that's clear from the diagram, if not from my description. After connecting the Snaps, the security policies are configured with Setcomp to allow Bar to connect to North's D-Bus service by the APARMA policy. APARMA standard, well-tested, deployed across Canonicals and now backported into a lot of Debian as well. So this complexity is managed for us because those of us who have hand-authored APARMA profiles may be aware, while there is a useful process to go through and a useful series of steps to go through and do that, it requires intrinsic knowledge of Linux in general and the subsystems in particular. So we don't have to deal with that with Ubuntu Core. When we submit apps to the Ubuntu Store, then the complexity is managed for us, the security team will review these things and help to tighten policies around our applications. So now that we have the Snaps, which are synonymous and essentially just containers, running on our IoT devices, what does our security model look like? Can we prevent all attacks by default and move to Hawaii? Well, not quite yet, sadly, but containers can help with these attacks. So a compromised control server or anything that is pushing to the device, man in the middle, those are application tier problems. We can't fix that with an isolation technology around the process. We need to work on our layer seven concerns. Corrupt firmware, well there should be no access by the process to the underlying firmware, or hardware in fact, unless the applications are running as routes. And even if they were, standard practice tells us to drop our capabilities and provide some form of whitelisting on system calls or with Setcomp or on file system accesses and that slightly high level with AppArmor. We can fix this with cryptographically signed updates. There are many things that will do this. The obtain system is an implementation of TuF, which is what Docker content trust uses. And that is applicable for automotive updates. So things like this, these are solved problems based on cryptography that we have a reasonable bar of confidence in. The assumption that the boot loader has been compromised, well, depending on what we're using, we need to sign everything. Secure boot is the only way to actually instill any confidence that those cryptographically signed bits of hardware and software are doing what they say they're doing as defined by a standard web of trust based on the signer. What about the default password attack? Well, a prompt update mechanism, again, login via a side channel, for example, using OAuth with a third party UI, and ensuring that these can be rotated quickly, standard flavoured security practices. And that's a sage and teller. Well, remote access to any device requires some sort of network connection. It may require that socket to be forwarded through to a publicly-routable IP. None of those things are a particularly good day. Hidden backdoors, we shouldn't really have any remote access to a device. And if we do, then ideally it should be through some centralised management rather than leaving open ports with vulnerable services. But, of course, at the end of the day, if we have physical access to any device, it is probably in less sunken epoxy game over. It should be noted that if your IoT devices dropped offline, never return, you may have an anonymous Samaritan to thank. This guy is called Mr. Brickabot. This guy tried to solve the huge industry-wide problem of manufacturer-abandoned IoT devices in an interestingly benevolent but nefarious manner. He hacked into them, and he bricked them. So this would be an IoT device happily going about its day job, and then suddenly bang, drops offline, never to be restored. You might be able to read into what he's actually doing there, writing trash to devices, blowing away the disks, deleting all the network routes, so the thing is in its own black hole, setting kernel threads to one and rebooting the device, maybe some systems could survive this sort of misconfiguration, but most Linux devices are probably at this point pretty much bricked unless they can be reflashed. So, I mean, what does this mean? If this guy has had to take his own action to backfill for the lack of security across the industry in the space, well, who's accountable? Should it be vendors? And if you know this guy, Bruce Shania, he says, absolutely, vendors should be held accountable. Defconn is not Congress, in case you were wondering. With so many of these devices coming from overseas, especially China and Southeast Asia, Congress viewed his statement that not only should manufacturers and vendors be held accountable, but also financially punished for failing to keep their devices updated or viably secure at all times. This was viewed by Congress as a trade tariff and the idea rejected. This was just over two years ago, hilariously. So, the UK has had some ideas about this too, but until we see consumer or legislation driven action, there is no incentives for manufacturers to change their ways. Maybe GDPR will save us all. But again, without case law, this may have no bearing at all and we need some definitive legal movement. There are multiple cases still pending. And we can't talk about IoT without talking about botnets, huge botnets that have caused internet scale outages with humongous distributed denials of service. Mirai, Japanese for the future and not pronounced like that, I'm told. It's a malware that brute forces networked devices running Linux to join their botnet overlords. It primarily targets online consumer based devices, such as IP cameras and home routers. Then we had a terabit per second attack on OVH, Brian Krebs and Dyn DNS, which, as you may remember, took down all of the internet that only used a single DNS provider, including most of US East One. And we saw how many sites had a single point of failure. So Mirai was bad because devices were internet accessible and therefore brute forceable. And not everything is publicly accessible today thanks to network address translation. And of course, this will change. We're going to see the same problems because of the eventual proliferation of IPv6. By giving every device a public IP, IPv6 exposes devices that were safely hidden away with no public facing attack surface. We have to start doing better and soon. Where soon is the eventual unknown adoption date of IPv6, telecoms providers have made some steps in that direction. But as we've seen, the ever dwindling pool of IPv4 ciders keeps on giving. So we have to build better IoT devices. How do we do that, Victor? Thank you. So building better IoT devices is obviously not an easy one. We know it's a tricky problem. And it's extremely challenging because the problem largely is that there is a race at the bottom across most vendors. Consumers tend to just look at the price tag more so than anything else, which makes it very difficult for somebody who do factor in the cost of security to compete with somebody who doesn't care about it. So how can we change this? And one, and in my opinion, one of the most viable options for this is actually what Andy just mentioned, which is around legislation. So there is a code of conduct in the UK right now that is in process of being potentially gratified at some point in the future, where it actually forces vendors to, in particular, run consumer products to solve this problem and actually be accountable. And the problem really is that we need to make sure that the engagement between the vendor and the consumer is actually more long to more oriented, rather than just transactional. So this will mean either more expensive devices or it will mean subscription services. And I know a lot of people are not very keen on subscription services, but this is a fairly well-known playbook in the enterprise device management space for since forever, more or less. So we really need to think about the device cycle here. And the problem, as we have seen in the past, is that the life cycle really ends for a lot of vendors when they hand off the device to the user. That's when they think the relationship ends. But really, that's just the beginning of the relationship, right, that you want to have this for a longer time. And that pose a really interesting question, which is, what is a reasonable time frame? So what's the end of life for a smart device? Is it one year, five years, 10 years? I think we were discussing just before the talk. I think on Google's Chromebooks, I think they have a two-year life cycle. After that, they just say, we're not going to support anymore. That's at least of being upfront about that, as opposed to a lot of other vendors, but they don't actually give you an end-of-life cycle or end-of-life timeline. But it also pose a really interesting other question, which is something I alluded to before as well, which is, who is responsible for updating these devices? Is it the vendor or is it the end-user? If it's the end-user, the probability of them updating the devices, in particular, if they have jumped through multiple hoops, is pretty small. However, with more and more smart devices connected to smart phones, you start to see more intelligent devices. IKEA is doing a pretty good job there with their infrastructure, where they actually update everything through their app. So it's a seamless process for the end-user. So that's a big problem. I don't think this is a clear-cut solution. There's not an easy way of solving this, right? So let's move into some common failure points. And one of the biggest failure points, really, is that manufacturers don't really plan ahead. Like, they think of security as a cost, and they don't really budget for it. And obviously, that needs to be budgeted for. Otherwise, we'll keep seeing this botnet that Andy mentioned before. They're not going to go away. There's going to be more and more of them. In particular, you start to see the growth and how many of these devices are getting insanely insecure, how many of them coming online every day. So how do we design better IoT devices? Well, again, like we talked about before, servers and IoT devices are not that different. They are not converging more and more. And we really need to start getting back to the basics and applying common sense to this. And that means you don't hard-code passwords. You make sure you draw prodigious whenever you can. You need to have a solid update infrastructure. Having the user go to a web page and upload a bin file is not a viable update mechanism. You've got to lose 90-plus percent of the users if you do that. And another problem that we, in particular, I'm not coming from the IoT space, but I've been exposed to the IoT space more and more. And when I hear tales from the trenches and how firmware is built on a lot of IoT devices, it's really scary. Like people are building their firmware on somebody's desktop at the office. That's really not how you do best practice security. That needs to be done on a CI-CD pipeline, cryptographically verified, all that stuff. But it's not being done today in a large degree of these manufacturers, in particular, the low-budget ones. And people try to also build their own OTA infrastructure, which is kind of silly. I mean, they probably do it for cost free. It's about building an entire OTA infrastructure over there, or update infrastructure. It's really complicated. Most companies are not capable of doing that. So that's some of the things we really need to think about as well, signing updates. And we have to assume from day zero that whatever you're building, the probability of that device is being exposed to the public internet is very high. So if you assume that from day one, well, then, you have to take it into consideration when you design your service. Don't assume that you operate in this flower power internet environment where some vendors appears to think that their device are going to go into. Assume a hostile environment. So with that, I'm going to hand off, actually, to myself. Sorry. This is what I mentioned earlier before, which is this is my experience. So my previous company we're doing digital signage and we're using Raspberry Pis for building the entire infrastructure for those. So right now it powers about 20,000 devices around the world. And we're all using Raspberry Pis. And well, since I'm not coming from a traditional IoT background, we took a more Linux-oriented approach to this from the start. And this is related to the journey that I'm going to tell you about how the lessons we learned from doing that. And we screwed up in the early days, but we got, we learned from those mistakes and I'm going to share with you the various phases. So we started back in 2013 building this. And none of these operating systems that we talked about back then existed. So Ubuntu Core didn't exist. Balena didn't exist back then. So we had to basically build our own system. And me coming from DevOps environment, we decided that using Puppet and self-hardened Raspbian was a good idea. It was given the tools of the time. It was probably not horrible, but it was obviously not a great idea in retrospect. So we built out this self-hardened Raspbian images. We built out this Puppet for orchestration. There were lots of glue there with a lot of back scripting and then a secret sauce was actually the player itself. And what we discovered was a lot of pain points. And one of the biggest pain one of the big pain points we had was actually create disk images. We had to do that by hand because I'm not sure how familiar you are with the Raspbian Raspberry Pi space. But Raspbian back then, at least, was almost impossible to build by yourself. So you have to rely on their upstream images and modify them. That was one of the many pain points. We had to build out scaling. Scaling Puppet, as people may know, is not trivial, doable, but not trivial. That's something we spent a lot of time on. And it was slow at large. And we didn't have any transactional updates. So back in 2015, we had this long list of pain points we wanted to address. And we started to go back to a list that we had then derived and see how we could solve this. And these were the criterions that we had. So we wanted to have disk images built on CI. That was important to us. We want to have cross-isolation using some kind of kernel technology, containers, or just app armor, sec comp, whatever. But make sure that if some piece of the component got compromised, it could not actually affect the rest of it. We want to make sure you have transactional update. That was for us very important because transactional update means that we could not end up in this limbo state that we had ended up in the past, like we had a half puppet run and then we had corrupted files. So we want to make sure we have an entire transactional update system, both on the operating system and also in the application layer. We want to make sure we didn't actually have to reinvent the wheel ourselves. We actually wanted to use something existing that we can just modify with some support contract. So that's what we ended up with. And our bonus point was cryptographically signed updates. So what we ended up with was Ubuntu Core. We were one of the first commercial customers for Ubuntu using Ubuntu Core, actually. So it was really new when we started. And that allowed us to throw out a lot of the tools that we had written ourselves. We were able to scrap a lot of that and rely on the upstream tooling instead because at the end of the day, we're not an operating system company. We don't want to build all that and maintain an entire operating system ourselves. So we ended up doing that. And that allowed us to focus a lot more on the secret source and a lot less on maintaining the system. And we've gained a lot more security features. We got a lot of these solved that we mentioned earlier. And we're running a little bit short and late on time here. And we're going to end up with a recap from Andy here. Right. Thank you, Victor. To briefly summarize, most vendors today basically ignore security. There is, however, some hope with the new generation of innovative IoT platforms, which greatly reduces the lifecycle, well, it improves the lifecycle and lengthens, and also hardens the security elements, meaning that we can deploy things that we then have trust and confidence in for an extended period of time. We believe the fundament of the problem is a lack of legislation in the space. This could probably be extended to computer science, engineering in general, when we are building the tools and the infrastructure that keep people alive, ultimately from power distribution through the medical devices. Then some thought has to be given as to how we regulate, as other people would say, the civil engineering firms. So I'm not a lawyer. Please don't take that as any sort of advice. And finally, go forth and patch your devices. It is the only way that we can maintain security. Thank you very much. I'm not sure if we have time for questions. Yes? No? Yes, we do have. Any questions? The Yachter project? Yes. We looked at that as well, but there's a lot of work that needs to be done on your own side as well there. It's a very interesting project, but the whole update mechanism is still things you have to run for yourself. I know the landscape has changed a lot since back then. I know the tools for, I think it's Mender that does update systems for a lot of those tooling in these days. So that landscape has changed a lot since we did this market research. So I think Yachter is probably in a lot better shape today as far as tooling goes and ecosystem as well than it was back in those days. But yes, it was unfortunately not one of the ones we seriously considered back then. Yes. So we played both with both. Yes, so we have an open source. This is a side note here, but we have an open source version of Screenly as well. So there's a commercial and an open source version. The open source version runs with Balena as well. So we have a fair amount of exposure to Balena as well. There are ups and downs with both of them. I think Balena is a lot easier to get up and running with than with the core. But when you operate at scale, my opinion is that with the core has some better, bigger advantages. Also has a higher level of security baked in by default, which is largely why it's more complicated to get up and running with. But yes, we did consider both. Yes. Well, thank you for coming out all day, everybody, and have a good evening. Thank you.