 and it's also a driving force behind the legal board, the legal board, office legal and legal assistance. I wanted to be a founding member of the legal board on all its foundations and they have a big enthusiast of attracting bakers and learners of all sorts to the open-source world through their legal platform. This is going to be a good opportunity for Q&A, which is a senior engineer who doesn't have a lot of life, so have a lot of good questions. I won't go into my personal life horror stories and stuff, but I spent all my time preparing the slides for the talk this morning on the Pocket Beagle walkthrough, if anybody was in that but it didn't look like it had me familiar faces. But next door they're giving the embedded apprentice Linux engineer training where they're using Pocket Beagles and these add-on boards to teach people about developing with Linux. And I am spending my time there. This presentation was one that I was originally... I'm going to give it this next week at Embedded Linux conference, but unfortunately I was explained already how the slides are done by now but I was just going to reuse it, so forgive me, I don't actually have any slides. But if you want to walk out I'm not going to be offended, but I will try to show you some of the things about Net Console. I don't have it working in this setup, but I can show you where some of the gaps are, some of the things that allow you to get there when I have all the directors. Anyway, I've got a couple of nights of work ahead of me still. But if you've never heard of Net Console it's generally a way to use a network interface to do serial type of tasks. All it is doing is sending UDP packets, pretty much raw checks in UDP packets for the data and that could be all sorts of different. But the Linux kernel, it's the console messages for the kernel log so everything is printed out, gets sent out across UDP ports that's preconfigured for Uboot which I think is more interesting for the early boot phases. The somewhat more challenging part of Net Console is that interactive portion of the boot loader. You can specify to do TFDP, you can specify to do all those things using UDP packets. And you would just use a tool like NetCat, if you've heard of it, seen NetCat. I mean people have heard NetCat somewhere, okay. You showed up to a Net Console because you're actually ready to learn about like see Net Console and things working. So yeah, you can use NetCat and you can just, you know, the port 6666 and you can start interacting with the Uboot console directly from Net Console to see the commands coming from Uboot, right. You can type the commands to do things. The extra little bit that I'm really trying to add here is doing that all over USB, right. So if you, you know, we've got on this one, so this is the Pocket Beagle. So it's a little Linux computer running off of the powered off of the USB cable. And that USB cable also, you know, so it's connected to the Gadget controller on the ROG controller on the processor. So I can actually, and the ROM has a boot loader. So I can, let me just take my disk out, right. So normally it boots off of this microSD card. But the interesting thing is when you start booting up off of the USB. So I've got a little Node.js app that essentially implements, it uses the Node USB, or it essentially uses LibUSB, if you're familiar with LibUSB, that gives you an API to directly initiate packet transfers. So you can say I want to do an int transfer, out transfer, you know, a config, sort of transfer. So you can do all the individual transfers from the API rather than writing a driver for it. You're doing everything in user space. So with Node USB, it's just doing that exact same stuff in JavaScript. It may not be your choice, but the idea, we're trying to put this stuff together so we actually do it with the new web USB stuff. So the overall objective is to be able to boot your computer off of your web browser, and then even to go to the point of seeing the U-boot traffic in your web browser, right, and being able to interact with it that way directly from just the ROM on the processor. That's where we're headed, type my password in, right. So this is that little daemon, and you can kind of do the same thing all with services, right. You don't necessarily, like, because what I'm doing is instead I'm grabbing that USB device here, I could let it get enumerated by the computer and look at it as like a network device and do things that way, but instead I'm grabbing it here to be able to put it all into that sort of web USB framework. But it's essentially a TFTP server, right. So the ROM code on this processor actually comes up and says it is a network device, and then will allow you to TFTP code to it. So we TFTP SPL to it, then once SPL is loaded, SPL TFTPs, or it loads another USB driver and it TFTPs U-boots, and then from there it, you know, you can interact with it via Net Console, or you can boot to the kernel. SPL is the secondary program loader. So it is U-boot. It's a different compilation of U-boot. So it goes through a different set of if-diff and different linking. Such as it trims it down to a much smaller version. So it's a U-boot meant to load U-boot. It doesn't have things like the command line, you know, that you could run different, you know, it's more fixed purpose. The reason that it exists is because for some of these processors, including this one, you need to configure external memory with some initial code before you can load into that memory. So the boot loader itself doesn't, the ROM boot loader doesn't configure that external memory. So in our case, that's exactly what it is. So it all fits, the SPL will fit entirely inside the internal RAM of the processor, configure the DDR that's sitting external to the processor, and then that SPL can then load U-boot that's bigger than what the internal RAM is on the processor. MLO and SPL are the same thing. MLO historically comes as a TI name. It came from a proprietary approach which was to, so the TI developers started with U-boot and they stripped it down into an internal version that's loaded U-boot, but then it got pushed upstream and the MLO, I don't even know what MLO stands for, but the ROM code actually looks, on an SD card, if you're booting from FAT, it'll look for a file called MLO. So it looks for a file with that particular name, but the file is SPL. And did I say that SPL is a secondary program loader? I did, but I don't know what MLO stands for at all. Yeah, right off the, well, and it's just meant to load, it's just meant to load, but the primary one here is U-boot. So the re-of the ROM is first. That's Treasury Area. Yeah, okay. Yeah, I can, we need Tom, Tom Rini around to answer that one. Oh, is it? Oh, but that's the hands-on U-boot. So I'll just show you this. So that's doing the boot P and, you know, it'll just, it'll be sitting there in U-boot right now. So let me, it's not very interesting. U-boot is trying to boot, and so like some of the default configurations it has is to try the TFDP, this PXE config file, that's just part of what Mainline U-boot does. I don't know why it's trying to do that. My little program doesn't have any idea how to handle that. So normally I would stop it before handling it. Let me go ahead and bring up the serial port, which is the whole thing that I'm trying to get rid of. But the way you would normally interact with U-boot is you would have a serial connection to be able to work with. And for people doing development, kind of new, I don't want them to have to go and buy a whole bunch of other hardware and hook up a bunch of, figure out how to configure things just to, you know, just to be able to read boot logs they should almost never have to do. But in the case they do, I wanted to work all over the USB. Pseudo screen. Oh, no, it's a, you have to put the bot right after it so it happens to be 115.2. I hope that's the right location where this guy enumerates its log first. So if I, I mean, I'm going to just pull the power here. It's about to start rebooting again. My little Damon is going to sit there running in the background now. My little Damon is going to sit there and run in the background now. But, oh, it's kept the power up. The serial port has power now. So I'm going to just hit the reset button. And again, so the ROM, the ROM grabbed the SPL. Okay, and that's how I stopped it. So this is, at this point we're inside of the Uboot command line. So what's run here is, if we go back to the very beginning of the boot, on the serial output, we see this, your first line. Here we go. That Uboot SPL line. So that has, that's, that's, so SPL will print something out that says I'm going to try to boot from USB Ethernet. It tried to look for environment on the FAT, of course I have no SD card attached so it's not going to find any environment variables saved off in the FAT. And then it ends up getting the Uboot load. Question? Yes, yes. And the boot ROM itself on this particular device includes that it looks like an R-indist, USB R-indist device. And it tries to do an ARP and a boot P. So you get the, you'll get a boot P request and an ARP request. And then it will start doing a TFTP transfer. So this, that first part was satisfying those requests by the ROM code. And then the second part was satisfying those similar requests from SPL. You couldn't then go from here, if we wanted to TFTP the kernel, we could use the same statement to TFTP the kernel or TFTP a root file system. So, I mean, that part is pretty well handled. What's not handled is the debug part, which is why the net console part is really important. So as long as all that works, you're great, but if you're trying to develop a new kernel, you're trying to develop a new, you know, making changes to Uboot, without having a serial port, you're not getting any real feedback. You're just not getting the packet request and you have no idea why. Or you're not getting your kernel to boot up. And so when you want to do, when you want to do net console, essentially it's pretty straightforward. You have to compile a few variables in the configuration that you say enable net console. But from a practical standpoint in Uboot, you also want to probably have the thing that allows you to muck the output. So it's possible to have multiple consoles, multiple consoles in Uboot. And maybe eventually I'll just switch over to all net console, but typically it's serial, so few prints. So like you'll see these, I don't know how many people, how many of you have played with Uboot's command line very much, right? For the most part, people would say, okay, well Uboot needs, you know, loads of my kernel, I'm happy, everything's good, well I'm good. But it's actually a very, very flexible programming system that it's got in here. So you can print, you can help get a list of all the different commands. There's a lot of different commands. You can find out different version information, so you can query quite a bit. If I wanted to be able to turn this into a USB dongle, I actually run UMS right from here, which means USB mass storage. So from this I wouldn't even have to have an SD card writer that's separate. I could actually just straight from Uboot, I could run UMS. I don't have UMS compiled into this version. Yeah, so if you run the USB mass storage thing, you can actually just use it to serve up the card and write it directly from the card. That's another aspect of what we're trying to do here, so we can use the web USB and it'll automatically down the images and program it. Is there a question or comment? So the device tree, this is super, super useful stuff if your system's not booting because you tried to load the wrong driver or other things, so Uboot has the ability to patch the device tree. So the device tree is the thing that describes all the hardware to the kernel. So for each of the different target boards, you'll have a device tree that gets loaded at the same time as the kernel. The boot loader will load that. For Beagle, we also have an overlay scheme where we'll look at a bunch of different variables that you set up in a text file and use that to load different overlays and Uboot will actually apply those overlays for us. So in the kernel boot, it's all just set up and running. There's also ways to dynamically do it in the kernel, but for things like, oh, I'm adding on an LCD. I would like for just the LCD to actually show the boot messages or something like that. You know, I want it to have it very early on. So you can do that in Uboot before the kernel ever loads. So you can modify the status structure that describes the hardware to the kernel. All things that would be really nice to do without having to go get a serial adapter and things that I found that I can't do in the kernel that I can do in Uboot. So even if you have like a fleshed out device tree, I don't think there's a way to easily, I see remove nodes, but honestly I really, really struggled in any sort of order to really remove nodes practically, but it's pretty easy to go and add a status equal disabled to an individual node so that when the kernel loads, it doesn't load that module. There's a number of hack modules essentially in the world out there that you might want to load that don't unload themselves properly. So any time if you try to do it dynamically in the kernel, it needs to have the remove module support done properly. And the config at Fest Pro is how I'm doing it now on Pocket Beagle. So as far as live patches, we just really discourage people from doing live patches instead of doing the patches as Uboot. But there's a few places where I'm showing people how to do the live patches with ConfigFS. But the Cate Manager, the Cate Manager is not part of Pocket Beagle. You're pretty quiet. I know the rest of the room, I'm sure the rest of the room can't hear you. The advantages of ConfigFS over what? For the most part, people using it really aren't doing editing. They're just wanting to use it. So that way they can have it at boot up that it's all just running. It's persistent when you edit the config for Uboot. I'd say it is a challenge to try to do certain things. We now have in our default device tree for Pocket Beagle, we load a bunch of things called Pinmots Helpers, and as well as we actually export a bunch of the GPIOs to CISFS, so we allocate those GPIOs to CISFS. So they're ready for users to do user space-y stuff to the GPIOs. But just trying to disable those, like the helpers for that is all like one big thing, and it doesn't unload cleanly from the ConfigFS. But if we patch it in Uboot, it never loads. So we just set status equal to disabled as part of our overlays, and then we export whatever GPIO pins we do want to export. There's just a few cases where you really needed it to be in Uboot, and choosing one or the other, like the LCDs, like that was one of the biggest ones was the LCDs. Well, but we have ConfigFS. It's already in the kernel. When you talk about the ConfigFS, the Cate Manager, so one of the things that we do in Uboot now is we actually scan the EEPROM. So the bone Cate Manager, so if you've heard of Cate Manager, it goes and looks at EEPROM values, looks for the device descriptor string in the EEPROM of add-on boards for BeagleBoard, for BeagleBones. And if it sees that it loads the device tree overlays that is associated with that board automatically, so it just dynamically configures and goes at boot. We were doing that in Cate Manager at the kernel level, but now we're doing it at Uboot because Cate Manager, they didn't want to accept upstream. They've got dynamic overlays, but because I think a lot of it has to do with the terminology of calling the capes a bus, that's not a bus, right now it works. By default, it's going to run this Uboot, it's going to run boot command, which goes through this list of different boot sources and tries to look for some valid image to boot. I don't know, we want to spend a whole lot of time going through Uboots, but I did want it like so. The Enverma variables, standard in, standard out, right, they all say serial. If I put serial comma in C in those variables, that starts in that console. Usually, when I do that, I've always created a variable that sets all three of them at once. I don't know if it really matters or not, but if there's a right order you can do it in, but I've always set all three of them at once into serial comma in C, standard out, serial comma in C. Now it's going to complain because I haven't set the server IP or my IP address. This is where I need to kind of think about a few things, but if I do run, start, see that it brought the network up, but it says there's no IP address set, so I need to be able to set IP address before. Does that quite make sense to me? So we need to try to do something to fetch an address before that. There's a couple of steps. I've had this running before like six months ago. I had this running, but I was actually using this to do board testing. I was actually booting the boards over USB and using the net console to issue commands. So I'm going to pull that code up and try to understand everything that I did to get that again. But I abandoned it at the time because it was too slow compared to booting off of the microSD card. So it took a couple seconds to load up everything over USB, whereas I could be done well under a second with microSD cards. So I ended up doing all the board tests. So for the pocket beagle testing, I was all done booting over microSD and then using USB to send the serial number across. So we still make a USB connection and test the USB connection when we build the pocket beagles, but we boot them off of the microSD which tests the microSD card, and then we send the serial numbers to U-boot but over the USB cable. We don't boot it over USB. So the serial number is going, I have to try to look at the code. The tester has the serial number and it's sending it to the board. The test code is still on a gist. I don't think it ever made its own repo. So the test code, this is the DTS file that I was using. A lot of this stuff in this repo is also about bring up. So this is the tester code so it says JavaScript code. So it's actually using NetConsul to send, oh yeah for sure. So we just increase the font. So this JavaScript file is using NetConsul over U-boot. The U-boot boots up over the microSD, but then it exports the NetConsul. So I just listened for the UDP ports here. I saved off my magic foo. I probably should have just copied and pasted it right. But this is the magic foo for starting a NetConsul where I've got the, I'm setting a server IP and a personal IP address and I'm setting all the other items. But the commands that I'm issuing, this is what it actually issues over during the test. So it turns on an LED. This is the write protect signal. The GPO is the write protect signal for the eProm. It's tied to a line. Then I've got this script that sets up the eProm with a base header that's coming across all the pocket beagles. And then I've got this serial text. It's actually the serial number for the board. And then I'm doing, what's some, what are these memories? Oh, I'm putting it into memory chunks at a time. And then I'm going to do a write of the whole memory dump, right? So here I'm doing the eProm write. I've received a reading the data that I put into this location in memory on the, what is that, does that have the link? I have to look at the eProm write parameters exactly. That's the I2C address for the eProm address. It's reading out of memory. It started off at zero and I think 1C is the link. And then it sets another GPIO if the write succeeded. I read it back. I set another GPIO to say that the write succeeded. I do a memory compare to actually see that the eProm value that I wrote is the eProm value that I read back. And then if that works, I set another GPIO. And we've got all four LEDs on board good shipping. So there's the testing here. You've got the USB for all the communications. You've got the micro SD card for doing that. Plus you've got the eProm writing. So it ends up doing a number of tests on the hardware in order to get here. But this is what's being used to test all the pocket vehicles before they ship out. It's really not very well documented at all. There's a reasonable amount of stuff for doing that console from the kernel. But when you start doing that console in Uboot, it's largely undocumented. My hope is to kind of bootstrap some of that and to kind of give a really good example. Unfortunately, I wasn't prepared today. You can definitely miss some of the early stuff. I think you need to set a reasonable boot delay in order to allow time for the interface to come up before you start doing interactions. Yes, I will recommend that people doing this actually just do either both if they still want to be able to access the serial or as a default. There's a way to test if there's a network connection or not, essentially by doing the ping. If there's no network, if the if net console, so pre-doing the boot, we do a ping of the server IP. And if we can't ping it, then we don't start net console. I think for your individual target, it may not hurdle. So you've been trying to ask for a while. That's what we're trying to get to. Where the usage is right now is everybody depends on serial. A lot of good examples, a lot of people coming to talk here are really prepared showing people how they can do everything with net console. Until that happens, and lots of people are using it on a regular basis, I don't think the documentation and the examples are going to be clear enough for people to follow it enough that people migrate away from using serial as their primary debug mechanism. That's where I want to try to get to. Let's not pay for the USB to serial adapter in order to have people be able to bring up Linux. And especially with something like what you've got with Pocket Beagle, you've got that SIP on there that's pretty easy to lay out and make a board for. I think you can actually probably do it with a two-layer board, which currently are four-layer board. If Michael is in the room, we'll have that debate. All you need is power and a crystal, and you still need to do some strapping of the boot config lines. You put resistors on there to kind of tell it what boot mode it should try, or what boot mode, because it tries four boot modes depending on what the boot config is. So you need that much hardware, but really nothing else. So you can get power and you can get the USB connection just from the USB cable. And so if you're trying to build your own board, you don't have a lot of other system dependencies. It doesn't matter what's on the other pin. I can boot up and run Linux, and then from within Linux I can go and try to turn on other things and turn on a subsystem at a time and test it, see how it works on the board, and go about that approach to bring up a new system, which is pretty different from the way people do it today. And although things have evolved a lot, and when I started doing these and everyone would have this, you start with the JTAG connection. First you get JTAG, and then from JTAG you start loading, okay, I'm going to do some memory tests to make sure my memory is all right. Okay, memory is good. Now I'm going to go to the serial port. I'm going to get my serial port connection, and I'm going to get that data going. And you build all these things up from these individual test code items that are personally hard to maintain and move forward on the different platforms. That's the way you traditionally would bring up a board. It's evolved somewhat now because there's a lot of open source support. There's still the initial processor bring up, but if somebody's already brought the processor, you can probably get to that serial port and do some other stuff there. But I think this takes it to a whole other level of simplicity. You can essentially do your board bring up with just the USB connection, providing your power, providing your communications, and everything else. Then you can use the Linux to actually running within a system, booting into RAM, to start going out and doing anything else, sitting outside of that package. I try to look at it both ways. What does the professional need, and what does somebody need as a kid or a hobbyist or something, just trying to get into it to understand what's going on in bootloaders and get that visibility. And if they can get it in their web browser, they didn't have to install any other tools, and it was less dependent on what that system is. That's to me sort of the magic. There are some tools to give the USB, that means a few, browser to serial, but they still depend on external hardware and connections that are just trivial. This is the one connection to the board. Anyway, that's what I'm trying to get to, and we've got the Node.js project, but I think the big thing we have to kind of make a change is get people using Net Console, because it doesn't get used enough to get really debugged and fleshed out. And maybe I'm wrong, but I haven't heard anybody here actually using Net Console on a regular basis. Anybody use it on a regular basis? You've used it on a regular basis, or at the viewboost? Both. Both. Oh, wow. I need to... Why haven't you spread this presentation? If you had some documentation of something right up on that, it would be extremely interesting. The gentleman with the necessary. He looks familiar, but I don't... So what do you use Net Console for? You can't go wire up serial cables to every single board, but you've already run the Ethernet. Did you boot? Were you using Uboot? But you didn't do Net Console. Doing it in the kernel is not that uncommon. Doing it in Uboot, it's the support there, but it's... I mean, in terms of its function. But you can tell, I mean, I've done this a few times and I still struggle with getting it set up right. For the kernel, it's just at the kernel command line, you provide all the parameters for... Where do you want the IP address of where you want it to be sent? And it goes. So it's just kernel command line. Of course, are you telling me you're asking me? You still have to get it up in the kernel, but there's a bunch of how-to's that are pretty good on just setting up the kernel command line. Especially if you can get into EFI, where you can actually run... You can run some nice scripts from the bootloader to get things... If you can get into EFI, you can do the configuration of the kernel command line stuff from there. I think it makes it really easy. Well, this doesn't have either. So that's why we're doing it over USB, which makes it a little bit trickier because you've got to get the drivers to come up first and then you've got to try to get the services. If you're sitting on a static network, all you have to do is you have to have somebody running and listening for the ARP request and the BootP request for when the device shows up on the network. To get it as IP address, to get it as base code, it's a little bit different when you're doing it over USB. I don't think it's that much more complicated. There's a couple twists to it. In this case, we have a ROM bootloader that notes how to boot over a USB. You're going to still need something that knows how to talk to the Bluetooth stack. If you really wanted to add something like that, there are some Bluetooth serial dongles that I think could be an interesting solution, but you're still depending on firmware within that dongle to support the serial connection and doing it that way. At least you're not depending on software in your target. You're just depending on software in some external little module that you can control and replicate. I don't know anything that's got a solved boot interface on Bluetooth. Maybe some six low-pan experts will speak up. Thanks for coming by. I apologize for the lack of slides. If you have any other questions, feel free to come up. If you didn't want to shout them out, feel free to come up to me. If you have any pocket beagle questions, I can certainly entertain those. One more? Yeah, for ELC. I can absolutely upload it on the scale slides as well, so I'll make my slides over the next couple of days. So give it until after ELC and then look, and the link should be live. My apologies. Yes, again, feel free to come up to me afterwards and I appreciate your understanding. Thanks. It's not going inside the pocket. I have a giant felt thing back here. No. Please don't. What? Put it inside the pocket. Why? Because everybody I know who tells me don't put it on the inside of the pocket. Here's what they do. And the next thing I know is the problem. What do you mean? It's not a problem. You guys can all hear me, right? See? They claim it does this work. No. No, it's not my headset. I don't own nice things like this. What happens if I do this? It sort of works, but I don't get those. And this is why everybody hates doing presentations from one expert. Well, some of us do. I still have like 15 minutes. I'm just going to like be ridiculous and silly and tell them. So if you're welcome to leave if I'm being too silly. Yeah, Inkscape is excellent. That is true. I did not do my slides in that because, no. No, I'm not putting anything nice up there. Why would I do that? They're the ones who have to look at it. I know it's on my slides. Oh, I mashed something wrong. I screwed it up. I know it looks fine, but it's obnoxious. I was hoping to get presenter view to work. Presenter view? Oh, maybe if I do this. Can I full screen that? Just click to present. If I do that, and then I hit, what is it? F12? Oh, that's not as good. I think I figured it out. I was going to say all of you guys are going to be able to hear me no matter what. It's whether you're going to be able to record this or not. Blah, blah, blah, blah, blah. Blah, blah, blah. That sounds like I'm more coming out of the, maybe this is a bad idea. Oh, wow. That actually works a lot better. Okay. I was wrong, Tom. The other mic was not as good. No, no. I was going to say the sound is fine for the last part. I don't think I'm too loud on your recording probably. Blah, blah, blah, blah, blah. IOT is broken. It's all broken. We're all going to die. I mean, life is a delightfully 100% fatal sexually transmitted disease. My wife likes to put it since we just had a small one. Her maternity leave was technically an STD. For those of you who are not American, STD being short-term disability. Not the other way that should go. My wife's Canadian, so she found this hilarious. Oh, that's going to be really obnoxious. Yeah, but I never stay put. If anybody has ever been to one of my talks, they all know that I never stand still. And this is why that little camera is going to hate me. Oh, it's super, oh. You knew I was going to talk in here. I don't stay. Yeah, pretty much. I mean, Pycon's in Cleveland. Totally not sneaking into Pycon again this year. And hint, I'm totally sneaking into Pycon this year, because that's what I do every year. Because I'm a pearl developer. I'm not a Python developer. And they think I'm funny. So I have seven minutes to tell jokes before you all start laughing up there. You try to laugh at me or... That would make this topic really, really boring and depressing. And this topic's already really, really depressing. I can neither confirm nor deny that there was a Warthog 1 through 8, and the bodies are totally not in places that you will never, ever, ever find them, because they're next to Jimmy Hoffa. So the entire reason that the Raleigh Durham photo was ticked was when I did a Google image search with a label for reuse, so that hypothetically should my employer decide that they actually want me to talk about this subject internally, that I could still use these slides. That was the first one that came up that was pretty and had an appropriate amount of barbed wire on it. The rest of them got much more depressing, as it looked like they were military members actually stringing the barbed wire in much less friendly territory. And I didn't think that was quite what I wanted to go for. And my laptop decided it wanted to go to sleep. That's going to be interesting. Let's see if we can fix. Do you remember where a known bat setting is? It's fully charged in 47 minutes. Power saving. Blank screen after never. Empirical evidence in greater than 15 minutes, since the 15 minutes was the longest setting that no one would give me. So if it doesn't blank after 16 minutes, I'm going to empirically suggest that that is as close to never as I care about. Because never doesn't exist, only the heat depth of the inverse. Doop, doop, doop, doop. I bet you this is Jason Kripner's water. Always find it fascinating how awesome their recording gear is. If you're ever bored, they built their own hardware to do video recording for everything. So come sneak a peek behind the podium if you're ever interested. Or bored. Pretty solidly full room for three minutes out still. Clearly I budgeted way too much time to make Linux work on the screen again. I remember when I first started doing presentations and presenting on Linux, this is early 2000s, trying to get the second monitor to work at all with a bloody miracle. We all knew magic incantations for X-Rander. How many people actually have heard of X-Rander? I'm sorry to all of you. You're all much too old. To the rest of you, feel sorry for the people who just raised their hands. Because the fact that we even know about X-Rander is, that's right, I have fabulous prizes for this talk. And I remember the last time I actually did, I think, right, I have fabulous prizes that somebody may or may not want to win. Which will become apparent as I actually give my talk on why I'm giving this away. So somebody will either leave very, very happy or very, very sad, and I'm not sure which. Well, it depends on potentially how much you trust me. I'm a very trustworthy human being, right? My wife has a PhD in computer web security. So yeah, I'm probably not going to be the most trustworthy person in this room. Yeah, she is much more educated than I am. That is true. She argues that her PhD is nothing more than her being far more stubborn than me, which is kind of impressive. Because you don't get a higher education degree because you're smart. You get it because you're so bloody stubborn that you actually make it through. As the people with higher education degrees all agree. I had a teacher at one point, since I have 20 seconds, trying to convince him to go for my masters. And I said, you buy me hardware to break loading machines and I will come back and I will be my masters and possibly my PhD. He never bought me hardware. And I went back. If you're doing my intro, you are welcome to do my intro. And that is a lovely machine. I have one in my backpack. Are you running Windows or Ubuntu? I'm running a software site. I wanted to show you earlier, open source contributions for probably everyone who was here. I want to thank you for this. I'm really grateful that I have only a year or two of leading the admin. But it turns out that we as a man with many hands, we have a lot of hardware in the embedded field as well. And I'll be putting on some drones on fire. Are there some things like that? I will set the record straight. The drones never caught on fire, but they did crash because that's what drones do. And it was the robotic dog that caught on fire at scale three years ago. I made a replica of canine from Dr. Hill. Both sides. Anyway, but that is not what this talk is about entirely. I'm here to talk about IoT and how good fences make good neighbors in this magical land of IoT. I am required to start out by pointing out that one of these people in the picture is a snowman and one of them is me. And either way, this entire talk is, my opinion, my commentary, all of my blathering and nonsense, not any of the views of my employer. Who happens to be VMware who was very kind enough to send me to this conference to talk to you guys about all of this. But let's start by taking a quick look at what the IoT world looks like today. I gave a talk last year where I made some interesting commentary about where is this IoT bus taking us. If you came to that talk, a lot of these pictures look familiar. If you didn't, this is a random smattering of all of the IoT devices that I have, not necessarily in my house, but that I have at least looked at from a perspective of how they work. And they're such delightful devices up there like your oven. It connects to the Internet. That's not a bad idea at all. Your coffee maker, if you have one of these, make sure it is connected to the Internet so that we can turn it off so you don't wake up in the morning. Pace makers, that's a totally good idea. Door locks. Door locks have actually had a very interesting change in them last year because a very large online retailer is now asking you to give them access to your electronic door. Is this not amazing? How could this possibly go wrong? I know. I actually get really excited because this is so ridiculous. And to be fair, the very large online retailer, they are actually well played because a lot of criminals are figured out, wait a minute, if I just wait for that weird white van that's not marked in any way, with the person who has no indication that they're a delivery person, goes up to the door and I wait 10 minutes after they leave. I can go steal all the things. So a lot of criminals have figured this out. So what they're trying to do is actually put your packages inside the door and they're there for the criminals. Great. Except now that you're giving random strangers access to your house. Good job. It's not entirely ill thought out, but it does make some very interesting commentary in an internet connected world. I'm going to come back to that one. Your refrigerator? That's no more dangerous than your oven. Thermostat? Again, really no more dangerous than your oven. Your TVs? Oh, this is always a really nice one. How many people have a TV? Okay, keep your hands up. How many people have a smart TV? How many of you believe that it's not spying on you? I have some news for you guys. And how many people have one of these magical smart speakers? How many of you have had a cackle at you in the last few months? How many of you have had a cackle at you in the last couple of days? Because apparently they're all cackling masks. Because that's not super creepy at all. Dark out. It's raining. I'm from Portland, so it rains all the time, so this is really easy. You know, I also have kiosks that come in my backyard, so you hear a kiosk, and then the Alexa device just starts not creepy at all. Let's see. What else we got here? Oh, and cars. Internet connected cars. Some friends of mine presented to DEFCON this last year and pointed out that several luxury branded cars, down to and including some Nissan Leafs, which was the car that they were looking at, are actually vulnerable to the haze modem attack that affected the iPhone 6, or iPhone many, many years ago, like four or five years ago. These cars were still susceptible to this attack. It's great. It means you can take over the entire car. You can turn the windshield wiper into the brakes and the accelerator into the horn and all kinds of stuff. And you know how this got fixed? The carrier that was providing the GSM connectivity to these cars decided to block the port, the incoming port that would have connected to these cars. Do you like to know what port that was? 6667. And for those of you who are on IRC, you'll notice that that's the IRC point. So no more running IRC on your car. I'm sorry, guys. Or at least you have to do it on port 7000 now and not 6667. And then we skew way, way, way off into the weeds and then we look at things like the Bluetooth enabled pregnancy tester. And you wouldn't think that a pregnancy tester is the thing that's going to pee on once and then throw away would make sense as an IoT device. But it does. Because it just so happens that this is a marketing ploy. Absolute marketing ploy. Because you know what the most profitable demographic is for marketers? People who are having a kid. Disclaimer, I just had one. The marketing material is insane. If you have not had one, be prepared. Just junk mail apparently. So yes, you install their app and once it determines that you're pregnant you just start getting coupons and spam constantly. For the record, my wife and I did not use that. We found out because we were going to India to give a pair of keynotes and they found out when we were getting vaccinated. And then the last device on here which I'm going to talk about quite a bit are these things. You plug into the socket. You plug it in your device. And it's got a giant relay because click, clack, and turn your lights on and off. Or your oven, your coffee maker, your crock pot and anything else you can look at here. So these are a random spattering of all these IoT devices. And these are all coming to market as quickly as possible, as fast as they possibly can. And they're great. They're cheap. Everybody's putting them everywhere. How many people do not have an IoT device in this room? In their house running right now? I'm actually really impressed. I'm almost shocked and I'm probably lying, but I'll trust you. So here are some really neat sacks about all these devices that we're cramming into our houses. And these are news articles that I have pulled from within the last year. I decided that anything older than that, so like the original or I thought that and everything, was just a little too old to care about. The most recent of these was reported yesterday that shows that most IoT devices can be hacked into bot nets. This is great. Look at all that cheap computational power that you're all buying for all the people who want bot nets. It's great. The Bitcoin mining is going to get even slower because trying to compute Bitcoin on this doesn't work very well. And it's not exactly profitable. But there's everything from in 2017 distributed denial of service attacks increased 91% mainly due to the rise of bot nets on IoT devices to IoT bot nets bypassing firewalls to get at the actual firewalls themselves so that once they've infected your network they stay infected forever until you take that router, take it outside, put it under some thermite and just nuke it into the ground and start over. And then hopefully you don't buy that brand of router again. IPv6 the problem is only going to get more complicated and harder. If for no other reason then IPv6 flattened the entire IP space across the internet. Right now we have these really broken things called NAT. NAT and devices. Don't get me started on how terrible they are that's an entire talk in and of itself. But once you flatten the entire topology of the internet where I can now talk to this light switch directly what's going to protect this light switch? This gets complicated very quickly. And one of the most interesting things and one of the early things was again an out a piece that came out of the morai bot net which was the IP cameras participated in one of the largest distributed denial of service attacks at the time in recorded history. GitHub last week eclipses that rather soundly so to those of you who perpetrated that particular distributed denial of service attack in graphs may you rot in the eternal fires of whatever circles of bad places that you would like to go to. But this is only going to get worse and in the upper left hand corner there's this ever present now spectrum meltdown that is now hanging over every device whether it's your plugs or your laptop all the way up through the cloud servers where we are now actively finding hardware bugs that are fundamentally impossible to patch in some useful way. And this should scare the crap out of all of us. Because whereas there are large companies who will try and ship fixes for my laptop nobody's going to give a flying rat patootie to try and fix the security bug when it's in my light switch. So there's some other interesting things going on here. So there's all these vendors out there, there's all these devices and from a consumer perspective I really want to control this device from anywhere. I don't know why you want to control this from anywhere. It doesn't seem to make a lot of sense to me personally but I am told that consumers really want to control the light switches from anywhere. So okay you as a product development company figure out there are these stupid things called firewalls and they prevent you from talking to everything. That's obnoxious. How do I fix that? I'll reverse proxy the internet connection which means that this will connect out to some cloud light device and create a tunnel which can then connect you from my phone to turn it on and off. It's great. Except when you start thinking about security or the fact that these companies may not actually be telling you the truth about things. So on the left there is a rather unfortunately heavily edited photo of a display from one of the IP cameras in my house. I believe I have 9 or 10 cameras around my house right now mostly because I have two and a half acres of land and trying to keep an eye on that big of a property as a bit of an issue. But one of the things that I specifically don't want these cameras to do is talk to the internet because the internet is full of scary, scary things. And so there's this magical line that says platform access and there's this magical thing that says enable. You'll note that it says it is disabled. This thing should not try to talk to the internet at all and as soon as I move my big head you'll notice the dump from IP tables that specifically says on my IP camera network trying to access the internet that device was attempting to talk to this IP address on port 85555 or 8555. That IP address just happens to go to Amazon AWS. For some strange reason I was not particularly impressed with that device. It is that every IP camera I own regardless of manufacturer does this. And some of them don't even give you the nice fiction of being able to turn off that. Some of them don't even give you it. This is great. We've now put all these neat little devices in our houses. They are lying to us a lot. If anybody is familiar with their update strategy it's great because they never update them. And there's all kinds of neat other just problems. I mean how many people have something wrong on their laptop right now? Okay. Think about that when you have like instead of like 10 devices on your network when you have 100 devices on your network how many of those devices probably have something wrong with them? That you misconfigured it, you just haven't figured out how Lua works on that particular platform. The vendor stopped supporting it so it only sort of works now. Okay. There's some neat things we can do to fix this problem or at least maybe not us exactly but there are some things to fix this. Companies can build better devices and provide long term updates. That's a pretty solid thing. I paid good money for this thing it should be updated every so often. Maybe once a year. It's a switch. It's got a relay. Maybe update it once a year. I'm not asking for much here. Open. You know, hardware and software. In a previous life I worked for INDO as their open source hardware evangelist and so I have actually a pretty strong bent on trying to convince everybody that they should be using open source software which I've been trying to help facilitate for the last 20 some years and hardware because well frankly hardware's gotten cheap enough we can all build hardware. There's probably a few people in this room who have literally laid out their own PCB case in point but you know be more open so that we as a greater community can at least come in and maybe help. You know, maybe we won't run your firmware but how many people have a rooted phone in this room? How many of you are running non-stock for your particular hardware manufacturers? Oh that's on your phone, tablet, whatever. Okay maybe a quarter of the room. Smaller than I expected I would admit. But you know clearly in the phone space we've all proven that we can build some maybe not quite necessarily always quite as good as the manufacturer but at least an interesting remix for your phones. So there's that. Devices can be less reliant on the cloud to spread out the attack surface. This makes sense. There's a lot of good and bad things and devices can be better or possibly even automatic updates and alerting users about updates. Most Linux boxes if you're slightly brave will auto-update. Unfortunately while I'm wishing for somewhat impossible things out of the universe I'd also like a unicorn preferably in green and purple paisley. Because none of this is going to happen let's be honest. When I said the first one you all started and you've mostly been chuckling under your breath on each of these that I've said. And that's depressing because we all have a ridiculous amount of cheap computational power that we're literally connecting to everything. That could turn on and off our refrigerators and our TVs and listen to us and cattle at us and record our entire movement as the IP camera in the back of the room is doing. This is genuinely kind of scary. And it gets slightly worse when you take a look at what a normal home router looks like. And unfortunately this is even more complicated than what most people have because most people don't have the wireless guest network. This is a more recent thing. This is advanced that we now have a wireless guest network that can't normally talk to the main network. It can only talk to the internet. And this is maybe a little bit of an over simplification but trying to describe networks is really hard. So let me give you an example of what this mostly implies. That we think the internet looks like this wonderful high-key party. Everybody's all being rules or all being polite. Why yes, I would like to talk to that port, sir. Would you like to call it? Why yes, the key is lovely. Can I have a thing of sugar? This is what most everybody right now and maybe not necessarily in this room because if you're here, you either want to hear me song and dance for an hour which is possible. Or you're genuinely kind of scared about how the state of IoT is and you're kind of in the right place. But most people who are consumers of straight up electronic devices think that this is the end all be all for how the internet should look. At least from their home perspective. Except that there's things. They bump into the firewall and they go away. It's great. That works most of the time. Except that's not the way these things work anymore. You'll remember when I pointed out that the IP camera was continually trying to talk to the internet. Oh, that's a problem. Because oh, yes, okay. I changed that point. Because this is the way the internet actually looks. It's a mess. It's a studio and it's the Wild West and everybody's shooting everybody. And it's really quite deadly. It's for no other reason that this device now connects to a random cloud provider of some sort. It then pulls down whatever it's going to pull down. I have no idea how this is authenticating anything. And I have no idea what's running in the cloud. It's for no other reason that let's say that flybynight.com built this device. It did not. I have no idea what flybynight.com is. And they built this. And then they go out of business tomorrow. Well, somebody comes in and buys up their domain name. This is connecting to a domain name not necessarily that Amazon AWS IP address. Because well, IP addresses in Amazon AWS are relatively ephemeral. Great. So now, because I've just bought up the domain name for flybynight.com I can do anything I want to this device. That's great. Right? What happens when I don't even have to buy flybynight.com? What if I just poison your DNS to the point where it connects to my copy of flybynight.com instead of the actual flybynight.com? ISPs don't, you know, muck with your DNS queries at all, do they? That's never been done before. And for those of you who don't get the joke I do this all the time. Usually to make money on ads. Neat, huh? Great that we killed that neutrality. Yeah, that's so then we infect this magical device that's inside my network with the Ebola virus which is arguably the most deadly virus known to the human body. And that's great because now we've just infected our entire network and everything's doomed and we're back to the every IoT device is hackable and part of the botnet now. So great. So all of you who have IoT devices in your house congrats. You're probably participating in some botnet somewhere. You know, say hi to the nice people who are controlling it. So yeah, this is to say the least. And this is where kind of the crux of this entire talk comes from because a number of people have known over the years my network has gotten more and more ridiculously complicated. And a lot of my friends would argue that I'm just an overzealous systems architect who likes things complicated. They're probably not wrong. But it also means that I've done a lot of things to my network that are weird. This is what my home network looks like right now. You'll notice I have a lot more VLAN than your average consumer. I'm also buying much nicer network here unfortunately as a result but my network is expensive. This also means that I can do a lot of things that we frankly should be doing by default already. This IoT device there is no there is absolutely no good reason other than the belief that consumers want to be able to control this while they're in Australia. When the device is in America they want to control this. There's so good reason. Seriously. It's just a bad idea. If your oven is on and you're in Australia call someone. They'll go over to your house and they'll turn it off because most people are kind of nice like that and I'd rather tell people where I hid the key to my house that I know and trust than putting this on the internet so that John Q. Stranger who just happens to find it on the internet can turn it off for me or on when I'm not there and they figured out that I'm not there and they put my house down. Of course, that's never happened before. My IP cameras I'm going to kind of blather through this a little bit but there's some really neat things that we can do here. And yeah, that this is beyond overkill. So I made network has access into all of these things. This is a somewhat dangerous policy, but if you're on my main network I kind of trust that you're pretty much me, my wife or somebody who actually literally runs the internet because I think those are the three people who have access into my main network who are not me or who generally have access into my main network which means that from my main network I can still get my IOT devices and my IP cameras and my multimedia stuff and my wireless. So I can get all of these things and it's kind of nice to have access to these devices. But when you're this this magical little device that turns things on and off I don't want you to have access to anything including the internet. I don't care if you get updates from the internet. If for no other reason then if this can't talk to the internet and it can't talk to anything else it's probably going to stay in roughly the same state that it started in. And if every device is in that same state you know, yes, it may not have the latest and greatest firmware it may not be connected to Alexa anymore but this device can no longer do any damage or the damage it can do is very limited. It can't get at by laptop, which is good. You know, because things like light switches these are random devices now that we buy we connect to the internet it can turn lights on and off they can calculate us they can do all kinds of things. Give me one good reason can anybody in this room give me one good reason why that IOT light switch should be connected directly to the internet. Okay, that may be the only reason sir. And I would argue that you win the prize but we haven't gotten to the prize yet. So, yes, the best reason that this entire room full of I don't even know how many people are in this room but there is standing room only in the back which is kind of impressive. The only reason we can come up with why that light switch which controls the lights in my living room should be connected to the internet is to screw with my kid who's five and a half months old who's way easier to screw with than the light switch. But I will remember this for a couple years from now. So, you know, good on you sir. Okay, now I'm not wanting my IOT devices to have access to the internet and yet I have this VLAN that says IOT with internet access. That seems really odd because IOT's things scare me in a lot of ways. So, there's some reason for this and I go back to the generally consumers want access to things. There are a lot of gateway type devices for Z-Wave and ZigBee and all these kinds of things and the open source ones are all great except they are about as user friendly as bashing your head on a text console with a brick. Which, you know, isn't entirely fair to them. Some of these things are actually genuinely complicated and hard to set up and make good UX for and some of them are eventually going to try and solve this problem. Great. You know, hand editing text files to add a new device to my home automation network is not something that I'm going to be able to convince a normal consumer to be able to do. And so, most people buy up these devices from various companies. There's a whole slew of them. I'm not going to name and shame all of them. But if you just go to, you know, some large online retailer and type in, you know, IOT gateway you'll find plenty of these things. And most are almost universally these all connect back out to the Internet because, again, people want to be able to pick up their phone which currently is showing a lovely clock and be able to turn their lights on and off when they're not at home. Which, again, you know, if you have consumer friends who believe this, please, you know, sit them down and ask them why they want to turn their lights on and off from Australia. Because I'm still not convinced that any of them have any better excuse than they want to screw with their kids. Because, you know, oh, we have a new idea from that front. Okay, correct. If you're home, frankly, you're already on your home network and if you're not there, you probably don't know they're there anyway. But your point of using it to scare off an intruder is a potentially valid use case. Although, you know, in some cases, if your home automation network is complicated or sophisticated enough it would be able to detect movement when you're in an away state and then automatically just turn the lights on and then maybe, you know, in an hour, you know, be able to email you or text you or something. It may make slightly more sense. But, oh yeah, automated tool Ram and Time also works. So yeah, so there are some ideas here but realistically, you know, at best, the best one I've been able to find that does the least amount of communication with the Internet is a device I have in my house which is from a company called Vera. They've had various names over the lifespan. And it's a pretty decent little device except when the EMMC on it sales and then they don't tell you how to, you know, re-flash the device over TFTP or anything. It's kind of a notch. But it does everything locally except for one thing. All authentication to the device happens at the cloud. So I can still go to the IP address for my device and I can turn my lights on and off and I can check the temperature. But if I want to clear out an alert, you know, that somebody moved in my pantry or something, I have to go and authenticate to their website that then kicks me back to my device with a token shoved into my browser. It is the most depressing thing I can think of if for no other reason everything else runs locally on the device. It doesn't need any other Internet access. And yet that is the best device I've found. The worst is, strangely enough, things like this! And you'll note at the bottom it says prize time. This is a switch that I bought off of Amazon or, you know, your favorite large online retailer that claims to connect to Wi-Fi. I'm like, great! There's a few more things in my house I would like to connect to things. Z-waves obnoxious. Great, I know how to deal with an IP. Great, so I bought this. Guess how you control it? It only works through the cloud! Awesome! There's this entire instruction manual which I did bring with me that explains how you install an app on your phone which, by the way, hasn't been updated since 2015. So it's totally secure. And the only way to get this relay in this device that is not actually connected to my Wi-Fi network because I figured this out quickly enough that the only way to connect this to anything is to let it log into Amazon AWS and you click a button. So the person at the end of this talk who comes up with either the best question or the worst question I haven't decided yet wins a prize. And you may take this home and you may do whatever you want with it because it wasn't worth my time to ship it back. And that should tell you how cheap these things are getting that I think I spent like $10 on this thing and it wasn't worth my time to put this back in a box and ship it back. And yet here I am I'm getting a good solid laugh out of it that I'm going to give this away today to some either very happy person or very sad person depending on their perspective. But yes, these are the devices we're putting in there. And this is why my network is as complicated as it is, is that the interactions that these devices have is insane. Why does my light switch need direct access to my IP cameras? What do my IP cameras need access to anything? And yet these are full blown Linux computers that have build root and Yachto based distributions installed on them. These are non-trivial kinds of devices at this point. They're almost as powerful as your phone. And that should scare you that these things have gotten so cheap so powerful and yet we're just putting them on the internet and letting them talk to random websites they've got a gig of a local storage that they could do things with because IP cameras they sometimes store things locally at least for a set amount of time. This is insanity. And so yes, I take my IP camera then they get shoved off onto their own network. And part of that reason is is that, again, since I have a little one why would I want to give anyone else access to the ability to look at my son while he's sleeping? That is a genuinely creepy thought that, you know, just because I want to make sure that my son is still breathing at three in the morning and I really don't want to leave my bed because it finally just got warm. How many of you are parents in here? My son is five and a half months old. I sympathize with all of you on those late night feedings now. I grok. I did not grok before. Those of you who do not have kids, you do not grok until you have them. It's... I love my son dearly. Gosh darn it! Could he sleep through the night once? My wife's going to kill me. I know. Oh, soon! Soon! Yes. Well, hopefully sooner than later. But yeah. My... Sadly, my wife has stopped dealing with it is overnight feedings right now by herself. So, hopefully, I don't return home and be killed. But yeah, I mean, this is the kind of stuff that we're doing and this is one of the specific reasons why people are putting cameras in their homes. People are putting these voice assistants into their homes. These voice assistants do not stop listening to you. If you have a voice assistant on your TV, it does not stop listening to you. It is listening 100% of the time. Because at some point, you might say the magic word. And for some strange reason, the universe doesn't work that it can trigger and interrupt that says, you should wake up now. They said the magic word. No, the way that works is it just listens forever. So, you know, if you have one of these devices that you can talk to and say things to and convince them to go into an infinite loop if you have enough of them. You know, these things don't stop listening to you. And if you have a smart TV in it, have a camera in it, there is nothing that you can really do to prove whether somebody is actually, you know, looking on the other end of it, whether you're watching TV or not. Because there's no, video cameras on like laptops and all used to have the LED that was attached to them and so they would only be on when the power was applied to the camera. It's not the case anymore. It's just a soft LED. Which means, you know, if you want, you can come take a look at my laptop. You'll notice that the camera has a cover over it. And if you don't have one of these, I highly recommend them. They're not that expensive. They're usually cheap plastic. Or you get them free at conferences. I'm sure when the exhibit hall opens tomorrow somebody will be giving them away. And yeah. Yes. So for no other reason I wanted to point this out and try and get people to think about this. And that's the reason this talk when Scale asked me to do this talk is they wanted me to kind of explain why I was insane. And the real answer is I'm not exactly insane, I'm just paranoid. And when you run kernel.org or a large internet infrastructure for a decade, you get paranoid. Because, you know, the target on your back is much larger than the target on most people. But it also means that you're more you're more aware of what that target means than your average person. And so they asked me to kind of come and explain why does this make sense? Why do we need this? And this is frankly why. And there are some neat tricks you can do depending on how you set your network up. The wireless network is not necessarily directly connected to my main network. It is on its own VLAN. And that is because I bridge the two VLANs back at the router. Because for two reasons. One, you can filter IP traffic on a bridge network. Which means that you can filter traffic as it crosses the bridge. This is really neat for a number of reasons. Because you can, you know, I don't want somebody on my wireless network regardless of who they are to have access to the IoT network. But I do want them to have access to my Chromecast so that, you know, they can throw things onto the TV and make, you know, pretty cat pictures or something. Well, great. You know, my wireless network now routes back through my router. I can block the traffic appropriately. And everything's still on the same broadcast. Because again, most home systems and a lot of systems even in business all assume that they are on the same broadcast domain. What this basically means is when I, you know, scream out into the void. You know, I'm on this network. Everything else goes, we know. But when you're on different network segments when I scream out, I'm on this network. The Chromecast, if it's on a different network doesn't tell me to shut up. It just goes, I didn't hear you at all. And that becomes a complication. So this is a neat way that you can get around segmenting things you know, domain groups to help with your security. So you know, if you want to follow in my insanity, which you may or may not want to do, there are some tricks that you can play here. You do have to have pretty much full control over your router, which means you're not running you know, stock routers anymore. I'm sorry, the open WRT stuff's not going to cut it anymore. But there are some very lovely small board computers that can do all this and you know, FreeBSD and all those kinds of things. They all work great. But there's also some interesting problems when you get into the wireless guest networks. Because I still want the random people who I actually allow into my house and then on to my network the wireless, the guest network specifically to have access to my Chromecast. Because they may want to show me pictures of their cats. The direct that I'm definitely allergic to cats I don't know why I want to stare at them, but everybody tells me that I should look at cats. How do you solve this? Because you don't want them on the same broadcast domain because that's bad. Because then they can guess all kinds of things about your internal network. But you also want to be able to give them access to your Chromecast. Well, there are a lot of neat ways to solve this and Avahi, if you're all familiar with it does have some ways of repeating broadcast traffic and doing it in a filtered way. Effectively, you're doing mDNS at your router. This works great. If you have devices that don't play with mDNS correctly you can always set up tunnels you know, little software tunnels across the networks at your router. And this all works. Works great. I'm living proof. If for no other reason then I haven't had any complaints in the last six months of things not working on my network from the users who come and have very strong views on this. This is all a mess. I've got a little bit of time for questions. But before I give you the opportunity to win this fabulous prize there are some things here that I want everybody to kind of take as a take-home. I may not have explained them nearly as well as I would have liked to in this talk but I only have an hour and this is an insane topic. Mostly because I'm explaining my own insanity. And yes, my network is likely to be overly complicated for what we actually need. There is in some respects no reason for me not to just lump my IP cameras in with my IoT devices and just accept that they're all one type of mess. I bought high-end network gear I can have 512 some VLANs VLANs are free I'll just do that. The equipment we were putting into our houses was nicer and we had better infrastructure available to people we could do things in even more clever ways. Things with like WPA2 Enterprise when you authenticate to a WPA2 Enterprise device you can actually set VLANs based on that authentication which means you can expose one SSID instead of the six I'm exposing at my house and then you can route things into different VLANs. But that means that this type of device needs to be able to talk WPA2 Enterprise and most of them don't. That is a thought. We need to do a better job of educating users and by users I mean probably not the people in this room because you're all very smart you're sitting in this room. We need to explain to users that their network topologies as simplistic as they are don't work anymore. They're fundamentally not safe particularly if you're adding IoT devices to it because the IoT device will be the infection point for your entire network at this point and if you have them I guarantee it. And it may not just be because you may go to a website the website will want a bunch of JavaScript it will go and probe your internal network find the IoT device and then infect it. And this is a known attack. So please run Adblocker. If you're not already please. Home networks are no longer simplistic things. They can't be. The belief that they are is just silly. Most users don't understand what they're actually getting into. Frankly most of us don't know what we're getting into with networks but that's a different problem. And we have to be better at every level about the defaults that we give to users. So if you're working in projects like openwrt or router stuff or you're building products take a look at what you're doing and seriously ask does this make sense? Why is there not a default IoT network exposed by every router on the planet now so that I can connect this stupid device I mean stupid in every sense of the word to the internet and then it can't infect things that don't make sense to infect. Yes it may still participate in that distributed denial of service attack but at least it can't do any more damage to the things I actually care about like the baby photos of my son. Something to think about. And while a lot of the things when I showed you my network they're very complex, it's not something a lot of people would even think about trying to set up because VLANs are almost mystic knowledge depending on what switch you've gotten, how you set it up and where are the configurations and it's not simple. We all have to think about how we can take these incredibly complicated things that we know this stuff can do and we have to get it to the point where we have to understand that any of us can use it at least at the most basic level. This is my fair mongering talk of the year and at this point I thank you for sitting through me being silly. And if you've got questions I think we've got a mic that I may run around or something. Actually if you have questions line up behind the projector. We do this at work so and I will badly answer your questions in the order in which they are received. De-solder it I don't have a good answer because laptops are designed in such messy ways. Purism is creating devices that have physical hardware on off switches. Take a look at their stuff. An earplug or something just shoved into the mic. I don't have a good answer for you because it's hard. You could but what does that mean? It means if you want to use it you've got to go faster. But just because software tells you it's disabled is it? That's the gotcha. It's that you can't It could be connected in any number of ways. And so the answer is valid or the question is valid I don't have a good answer for it. De-solder it is the best answer I can give you but I'm not a security person and I might be overly paranoid. Yes One of them is is there some internet place where something is posted as to what would constitute diligence for a ordinary person to have reasonable security for some daddy reasonable security at home? Number one Number two Number three How many man hours per year would it take for the person to invest to actually make that work? Because you're learning how things are changing. Oh, okay. So the answer to your first part of your question is I do not believe that anything exists but frankly something should and now I think I'm stuck being the last one touching that idea which means that I'm probably going to have to do that. Because this is the way the universe works the last one touches it at their head. And two, I don't have a good answer on that. Rough needs Well my knee jerk reaction would be if somebody goes and takes a look at some sort of updated document maybe once every six months and just sees even if there's just a diff of what changed since the last time I was here that I should be doing differently or better or something. That might be good enough. But the upkeep should be I mean once my B-lands have been set up it's not like I've had to do a whole lot to continue that security model. That beyond section. I don't know. It's an excellent question. I genuinely do not know and I'll have to go think about that. So. So I don't know if it's a comment or a question but seeing you express ideas that you might have been paranoid about what you've been to view that you are safe with a B-land in that with an embedded device I don't see what stops them from crossing the B-land barrier or crossing the Wi-Fi barrier either for that matter if they want a Wi-Fi transmit receiver how do you know that's right? So the simple you know how paranoid is John I think the simplest way to put that question is the point you have to draw a line on how far down the rabbit hole you want to go because if you keep going you go all the way down to I need to make my own computer chips and by I have to make my own computer chips I mean me I have to build my own fab from literally dirt to 29 millimeter fab so that I can build my own chips so yes there are attacks on B-land hopping and all that kind of stuff I'm mostly expecting the people who are doing these attacks they want to go after low hanging fruit I want to make myself so on low hanging fruit that they either can't figure out what I'm doing and go away anyway or that I don't fit into their their logical model and I just I become too much of an annoyance and they go they literally just go away so yes if I convince everybody to do this I've literally lowered my own safety threshold because now I've made everyone I've floated all the boats which means now the low hanging fruit is much closer to me yeah yes I don't need to be the fastest person I just need to be faster than the last person to outrun the bear we have that bite on you there I was wondering if there's got to be other companies that are producing I can see more responsible thinking do you have to write those things oh ok do you like brand life there are some ok the best answer I can give you to that is go and take a look at the devices that home assistant specifically it's a home assistant for an open source project that does gateway kind of stuff if they support it the company is at least giving you either giving you direct access to the device via your local network and that should be mentioned probably somewhere in the description or they're giving you an API through their cloud if they're giving you an API through the cloud you'll probably need some sort of API key when you would set that up so that should be pretty obvious in the documentation that is not the device you want to buy unless you trust the magic of the cloud what companies should we be supporting I mean the two switches I have mostly in my house right now that are wifi based TP links have an excellent one that you can completely control directly on your own network no cloud access necessary and there's Sonoff Sonoff but you still have to set up something else to control them you still need something like home assistant or if you're willing to give Vera a little bit of internet access and a little bit of access to your home those kinds of things work otherwise we should probably be looking inside the open source community and trying to help projects like home assistant get better the other thing you mentioned was the WCA2 enterprise which is that have that so the most wireless access points do support WPA2 enterprise the things you're going to want to look for if you're looking at access points are going to be whether they support VLAN tagging based on WPA2 enterprise I know that devices like Unify do support this which is from a company called Ubiquity but the gotcha is that you need to be able to support the authentication scheme that WPA2 enterprise needs and that requires radius and radius is kind of a pain in the butt to set up so it's a really good idea that's like even for me is relatively obnoxious to try and deal with and I haven't quite gotten my setup working yet so you're thinking of having a free of the march for anything just wait for the next generation I'm not saying wait for the next generation I'm saying go buy the stuff that's there now and help us fix it wait come help with IPv6 things like people don't really run NAT anymore NAT fundamentally should die with IPv6 and devices are a lot more uniquely addressable to the internet so the question is do you see this helping or hurting the problem so I have very mixed views on this because having run large internet infrastructure having a flat topology makes everything easier and better however also having devices sitting on my home network with IPv6 scares the Jesus out of me you are fundamentally losing a layer of firewall but NAT is fundamentally a lie anyway so all we're really doing is we're getting back to what we should have had in the first place more or less what that means is you have to be more diligent about your own firewalling and this isn't to say that having this pingable on the internet is a great idea that may not be and you should be able to stop that at your firewall or you just assign it there are private IP ranges that are specifically non-routable I believe they're supposed to not be NAT-able so local link layer IP addresses that you could assign to this so that it doesn't talk to the internet Do you think home routers should be designed by default? Yes, if your router is not blocking all incoming connections by default go home throw it in the trash because it's not even remotely doing its job They need it like that but also get the original designers of the internet so the internet would be like that Yes So I believe the real approach or one other approach to solving this problem is to be designing how networking is done I won't disagree with you on that the gotcha is IPv6 is already effectively deprecated and it's taken us 20 years to get to this point and how many people in this room have IPv6 at their house Yeah so I don't disagree with you this is something we could fundamentally fix by taking everything we know about IP throwing it out and starting over but that will be completed sometime around 2100 What's the project? I will talk about it the talk is about let's fix the internet I hope you will be here I will try I will try so there you go pitch for his talk, let's fix the internet by burning it all down let's fix the internet is the talk, it's on Saturday 3 o'clock open yeah okay so from a hardware perspective it gets really complicated probably the best thing I can suggest to you that is off the shelf that it can do everything that I kind of described that has a user interface that is not like please edit this text file please edit these 20 text files which is what I do PF Sense which is an open source project there's actually commercial support and you can buy specific hardware that does all of this I think it's stored up pfsense.com or something like that the boards are sold from a company called Nightgate and so open sense apparently PF Sense doesn't build properly thank you I will go bug some people about that actually I probably know what they're doing but yes, open sense or pfsense I mean just go take a look for those those are the things I would probably recommend if you're much more comfortable with the command line and you like hacking on things my firewall is entirely based on IP tables with a wrapper called shorewall and I just run it on a just a random x86 box that I basically pulled out of the trash and a couple of 10-gig mix that I threw in up yep I did mention ubiquity their firewall does not do what I want it to do but their access points and their switches are pretty decent okay for your Wi-Fi connection so the cameras so what happens if somebody can show up with a open Wi-Fi connection to the IP do they even if you lock them down do they start checking in with the letter shit so let me rephrase your question and see if I rephrase it correctly if someone knows my SSID is being broadcast everybody knows what my SSID is and if somebody was to know what the password was for that particular network segment which one would hope that they don't but you know let's say the WPA2 can't stand up and open them if you get two and a half acres they've got plenty of space to do it and I wish to see if she was working with a cold area and people had to go up by the hill that she couldn't find but they came back with their net so yes if somebody was to fake enough of my network to get those devices to actually reconnect out to the internet yes there's not a lot of ways that we can stop that particular problem other than one would have to hope that the device once it's connected to the network would try and attempt to forcibly only connect to the way it's expecting so you know if you stand up an open network one would hope that it doesn't actually just you know fail over that's not to say that that wouldn't happen but now I'm curious so I'm probably going to go home and try this and then be like oh god it's all broken and this will probably turn into a pocket desk con or something I have no idea but um yeah correct well it's fundamentally the same problem of if you know if you own a large house or whatever and you pulled the ethernet to the outside like I don't know they connect to your fiber there are problems of well what happens if you don't pull that connection and plug the different one in more or less the same problem there are ways around this 802 1x WPA2 Enterprise should fix that in the general sense because of how it authenticates because it's basically 802 1x but WPA2 Enterprise if you're flipped over to it it won't connect unless it authenticates correctly so but that's Enterprise not PSK and I don't know what happens with PSK so yeah now I'm curious it would have to be at the device layer and the problem is that getting access to these things is almost impossible these I effectively have to treat as a black box that we will never be able to fix at least that's the way I look at it well yeah it's it's a pain in the butt and it's trivial to spoof MAC addresses in fact most of your phones probably are randomly assigning their MAC addresses they connect to the network yeah that was it was a connected Barbie was it Matthew Garrett who did the teardown and they actually did genuinely do a bang up job on that so if you're ever curious that's actually a really good read if you're building IoT devices that's a way to go and look so there's a couple of IoT devices that are running around right now that effectively provide their own access points my Vera for instance has my Vera actually runs open WRT which is why I'm slightly annoyed that I can't convince them to let me have the bits to reflash it if I needed to but I haven't really tried very hard on that so that's a different problem but there are things like the fire TV the remote for the fire TV these days it actually does what's called Wi-Fi Direct which means it's actually broadcasting it's on Wi-Fi SSID to do connection with the remote and there's a bunch of things like that it's probably bad in the the grand unified universe view it's probably bad because now it's poking holes in a security layer that you don't control necessarily so like the Vera I've actually turned off all the access point abilities on those devices in my house mainly because I have much nicer access points than what they'll ever provide but things like the Amazon Fire TV with its Wi-Fi Direct I would like the remote the remote makes this usable for me but how do you is Bluetooth really better is Wi-Fi better I don't know and there's just the devices that bring their own access point for whatever reason whether it's Wi-Fi Direct or they believe that they should be the one true gateway for everything I think they're probably misguided at least in some sense now I do have some faith that Amazon if they do get hacked for whatever reason or their Wi-Fi chip has an issue that they'll push an update because frankly I think they have slightly better hardware engineers involved but I can't say that necessarily about all the companies involved that do these kinds of things because who knows maybe the device that I bought today I don't think I bought any devices today but if they ship an access point in their device how do I know that they'll get updates how do I update it if it's a problem how do I even disable it which goes back to the question about how do I disable my mic how do I know that my Wi-Fi chip on my vera is disabled well I don't see any SSID broadcasts that would match anything to it so I'm reasonably sure it's off but that's not that's absolutely guaranteed unless I go in there and I rip the Wi-Fi chip out that's why I'm doing that yeah it does come down to a well how do you reasonably authenticate stuff and yes, if they provide their own access point and you've got to go over and push a button so that it sets itself up and then pushes some data I don't know that may or may not be a good thing and some of that's just going to be a if you look at it and your risk assessment says that that is a scary thing don't buy the device again some of that does come back to a how paranoid do you want to be yes then I got to decide which one are coming with GSM in the lead module inside of them yeah and the intramed autobox what do you think about that yeah there's a lot of neat devices these days because GSM has gotten so cheap in some cases that in fact I'm backing something on Kickstarter right now which is a little device that you slap it in the back of your mailbox, the ones that have the key up the street from you and then tells you when your mail came they're doing light sensors and scanning and all this other kind of stuff which is neat and you can buy up the credits for the GSM for a year in the like five bucks or something like that for the GSM credits but yes, how do you protect against that effectively if it's on the GSM network or if it's on the public internet or on the Wi-Fi here even you basically have to treat it as a public device that you know it is just out there if you want to trust it with anything you're probably insane now if you want to just believe that you know it told me that my mail came you know yeah and everybody else on the block oh that goes back to the Amazon door thing hahaha um yeah it's neat I like the idea there's no problem of you know the connected car the the mailbox generally will not kill me unless somebody else has done something else malicious but don't do those things that's bad I think I'm going to give the prize to the answer that I could answer or to the question that I could answer the least so congratulations sir you want a prize you may do with that as you wish up to and including thermiting it which I would probably recommend anyway um let me get to the end here again my name is John Holly that's my contact information these slides I will get up to scale and they are CC by SA except for the pictures those are all copyright respective whoever they did it if it does not have a copyright notice it's probably my picture which is probably CC by CC by SA hey thank you we'll be doing more embedded tomorrow in this room starting at 10 thank you we knew that there that the cameras were insecure to begin with um and so