 Welcome everybody Hi, so yeah, thanks for the nice introduction Yeah, so today we're gonna talk about PVFD, which is a project. I started to make it easier to run BCC which is a powerful tracing tools toolset on an embedded remote target I'm gonna start off with a demo and then I'm going to show I'm gonna build up towards BCC talking about Linux tracing in general and then how that builds up to BCC and what BPFD is and Hopefully the demos will make it clear how how I'm using this stuff So a little bit about me first I work in the Android kernel team at Google We ship all the kernels on all the Android devices out there and my two areas are Linux scheduler which is the piece of code in the column that placing tasks on different CPUs and Load balancing the task between CPUs and scaling the CPU frequency and stuff like that. The other thing I work on is tracing Which is frays and perf and all these powerful tracing Technologies available and in the Linux kernel Yeah, and BCC is the is the latest thing that I mean so Yeah, so I'll start off with a demo of a tool that lets you look at all the File reads and writes that are happening across the system. Just show you how powerful this thing is how powerful BCC is And because we're in the kernel we can't miss anything Anything that's you know involving the kernel itself across all Processes we will see that in the kernel I'll just show you an example tool file top So just showing my setup. I have Embedded target high-key 964 Which is which has the high silicon 960 SOC? and it's running Android and It's connected. It's connected over Wi-Fi to my laptop And so everything I'm going to be doing is over the Wi-Fi network I'm going to like be tracing the Linux kernel remotely over Wi-Fi So we had to before I run five top. I had to set up some environment variables Where my kernel sources are how I'm communicating with the target what the target architecture is And so I run five top now you can see there's some some warnings that happen when the tool runs I want to go into that now, but right off the top you can see the Wi-Fi service in Android is Continuously reading from I fight that format to get information about the Wi-Fi interface So you can see that you see this you have this global view of what's happening on on the Android system And so I'm going to like open an app called the contacts the contacts app You can see it did a lot of reads and you know, it's reading the contact database there And I'm going to create a new contact just like how you would create a contact in a regular Android device and I just created a contact and you can see that There is you know, this entry here that shows that the contacts database was written into 53 rights in total So it's kind of nice to be able to do this over Wi-Fi To get that kind of observability you can see that There's not a lot of data transfer happening because All the data collection is happening in kernel As these like Thousands of events are firing the data has been collected in kernel aggregated there and periodically sent to user space in this case over Wi-Fi, so So all this builds upon a lot of technology. That's not visible. I just ran files out But there's so much going on under the hood that you do not need to know that's the beauty of this So probably speaking there are these six different Signals that you have in the kernel that you can monitor This is not related to BCC or BPPPF or anything just tracing in general. These are the signals you have the static trace points first of all, which are Points in the kernel that are inserted Place points that are inserted at compile time So at compile time you you can like have a static trace point that emits some information if it's when it's enabled So they call static because done at compile time These are the S trace events that you have and then you also have dynamic trace points which are which are Trace point probes you can insert at runtime That you don't know in advance That's how file top works actually it Instruments VFS read and VFS write in the in the VFS layer Dynamically so these are not these trace points are not decided ahead of time. That's what I call dynamic That's the K probes infrastructure In the kernel. That's what allows you to do that And then you have the user space equivalent of that you probes. Let's you do that for user space applications You probes is dynamic Again, you don't need to do it ahead of time. Then you have something called USDT which is the static Trace points for user space and then last to our first event So you have your performance counters. Those are also signals that you can attach hooks on to like cycles and cache misses and stuff Finally you have the sampling signal Which which lets you run some profiling code or some code Periodically a certain number of times a second. That's how perf profiling works You have an interrupt that fires 100 or 500 times a second and you can run some code that can observe the system So these so these are the various signals that we can build on top of to get to get the observability that we need So now let's come to BPS. So BPS lets you Write programs easily that can run when any of those signals are triggered Any of this any of the signals happen? the way it works is you have a BPS program that's compiled to BPS by code and That's loaded and verified by the kernel and so once it's loaded it's sitting in the kernel ready to run and Then you can attach that program to any of those signals. I just mentioned in this slide and program runs in kernel collect some data That the signal might have emitted or you might just want to read some kernel data structure or something like that to get that information the information that you're trying to observe and And Finally the BPS program in the kernel can output that in different ways using Maps there are masses a Data structure that lets you aggregate information as you get it like so for example histograms is a kind of a map And there are various other ways to to image information from the BPS program So BPS is a technology that lets you Let's you do that So now let's come to BCC. So what is BCC? So BCC actually puts all of all of that together. So it lets you set up all of these signals I was talking about the dynamic Signals especially activates them and all of that and it also does all of this stuff takes the program that That does the observability like files off this in the demo I just showed you and builds a BPS program out of that and Finally it attaches it does all that automatically for you attach it to the signals that are required by the tracing tool and It also has a set off Libraries written in Python and stuff that can read those read the maps and present the information to the user periodically BCC kind of it makes it easy the beauty of it is it does all this powerful stuff at the at the Behind the scenes that you don't even know like when I ran file top You didn't see any of that happening. Just had to say file top and it was showing you You know all this information there are like a hundred plus tools that you don't even know need to know how they work And you can just run them That is opposed to running tools like first or something like that where the running those commands might be more complicated and Interpreting the information so BCC does all the powerful stuff for you, but Really is very easy for easy to run So yeah, feel free to interrupt if you have any questions so far, please Okay so traditionally BCC has been Has has been typically typically run on the same device where you have your kernel sources and your clang stack and all of that And the kernel is running on the same machine where you're running the tracing tool and That would have been fine But unfortunately you also need kernel sources to be able to run the tools as I showed you you needed to set an environment variable To run file top that pointed to the kernel sources so all that is That doesn't work very well for cross development kind of Development model where your target is remote remote and your your development tools are on a different machine So That was so that so that makes it very difficult to run these powerful tools So what I did is I separated out the whole stack Into two into two parts one is the stuff that doesn't need to run on the on the Device you're tracing and then the stuff that needs to run So the stuff that needs to run on the target is just the stuff that that does a low-level Things that are not compiler related like Loading a BPS program once the bytecode is available or creating You know attaching the BPS program that was loaded to a trace point for example these things are low-level and are not like Not related to compiling the tool or Compiling the program that Produces the BPS bytecode and stuff like that. So, you know, that's and the way I got that to to work is I I started this project BP FD which Kind of encapsulates all the low-level things that are needed that BCC needs and have that have BP FD run on the target so in the in the demo, I just showed you BP FD is running on the arm target and BCC is running on my laptop Just to summarize why we want to do this You know, we don't want to sync kernel sources to our remote target We don't want to download them onto the target. We might not have space for that Or the sources might get out of sync with the target all these things. It's really unnecessary With this kind of a model you don't need to cook across compile Python Clang All of that stuff it can run on your host quite easy quite a very well run any host and This fits the cross-development flow where as I was saying you build on your host and run on your target You don't do it the other way and also your host machines are more powerful They have more processing power typically than battery power devices So it makes sense to do the compilation of the tool on your host And also many embedded targets don't have symbols in their executables things like that On the host you you have a bigger chance of having that information and the tools need that information to Need the symbol information in many cases to produce meaningful results So that's why these are all the reasons why we want to do it Any questions so far? Not I'll move on to the demos and show you some more demos Okay, so I Guess I'll just move on to the demos then So this is a tool, so I just showed you five top the other tools like hard irqs dot py Which summarizes the amount of time That I spent in interrupt handlers Over the period of time that he specified in this case ten seconds every ten seconds It's going to show me the total time interrupt interrupt handlers spend on CPU and That's useful because You might run into real-time issues or latencies and stuff like that if interrupts are spending too much time You need to as a designer you need to make sure that your interests are not taking up too much time These are all the interrupts that Spend time in the last ten seconds The second column is in microseconds So I'm just going to do some UI activity and show you that the GPU interrupts are now going to take a lot of time You can see that Mali here just like 141 microseconds milliseconds Ten seconds, so that's hard irq dot py just to show you the tool itself What it looks like? Tool itself has an embedded seep it has a C program embedded in it that is compiled into BPS Should yeah, no problem. So where this tool works is it uses K probes to instrument Handle IRQ events per CPU function in the kernel And it attaches the count on the Function and then when that runs it There are two modes actually you can either count or you can time I think the default is time so For the for the mode in which it times let me show you how that works because that's what I demo so for that trace start is is executed whenever handle IRQ per CPU runs and and it makes note of the starting timestamp and Trace completion Runs whenever handle IRQ per CPU finishes and It you it uses the the time it stored in the start takes a difference between that and Adds that to the to the histogram So some other tools that I wanted to show you This one I won't be demoing, but it's useful to see How much time was spent doing block IO operations that summarizes? Shows you a summary of every every so often of all the processes that we're doing block Ios how much time They took to run here by your top and biosnope there in the slide the Difference between biotop and biosnope is that biosnope shows you an entry per it shows you it doesn't refresh every every so often it It shows you like a tracing type of output It doesn't clear the screen but by top clears the screen and both these tools show you the the latencies that the bio operations took So this can give you an idea about whether the disk I was slow or not for example So another tool is cash that Which is useful to monitor what's happening in the page cache in the kernel whether you're having page cache hits or misses Stuff like that if you have too many misses in the page cache that can indicate that You know that is what is called causing a performance issue. I'll just show you a demo of cash that so I'm going to remote into my target board over adb And I'm going to run a while loop that reads a hundred MB file So there's a hundred MB file I'm gonna I'm gonna write a loop that that reads the file and then in another window I'm going to run cash tax every two seconds you can see in the third column here a cash status showing me that all the Cash accesses were hit Because the first time the hundred MB file was What's ready? It was cashed into the page cash Then I'm going to I'm just going to drop all pages in the page cash And so the next occurrence of cash that you can see that there were a ton of misses That's because everything the cat the page cast has had to be three prop populated with the file So the fourth and fifth column are not working because it's reading Files that are on my laptop instead of the remote target to populate them But the first four columns are valid again to run this tool all that you know, it was very trivial I just have to run the tool. I didn't have to specify any complex arguments or anything like that It's very easy to use if you want to do more complex stuff and and don't want to write your own program There's there's the set of tools called the multi-tools. It's one of my favorites the truth trace multi-tools and The multi-tool Automatically will generate a ebpf program for you and compile it with bcc based on the arguments you give it You can do a lot of powerful things with it related to k-probes and stuff like that in this This slide here. I want to see all the open The file opens that are happening on the system so this function in the kernel called do this open of which receives arguments from user space to open files and this you this uses k-probes to instrument do this open behind the scenes So I'm telling the trace multi-tool. I want to instrument do this open and print the Second argument that is faster do this open second argument is the file name I can just do that It's already showing me. I think that's the the while loop that I wrote in the other window that's continuously running So I'll stop that and so if I do some activity in Android It shows me all the opens that are happening surface finger and all that Yeah, some some files are being opened a lot of times, you know, it might be better to open them just once And that might give you an idea of maybe something is unnecessarily done And this tool will tell you that yeah, you're doing this and it's not necessary It's very useful for that kind of performance debugging And I wouldn't be running a demo of this so but still called run queue length Which uses the first sampling mode and every time per sampling Event occurs the BPS program runs and it looks at the run queues on all the CPUs and as information to histogram about what The run queue length like how many times a certain run queue length occur The histogram has that information in the end This will give you an idea about whether there are too many tasks running on the system causing performance issues Sometimes you might have so you can either have too many tasks running on system or you might have too few CPUs Like if you have too many tasks in the run queue, then it's you know The run queue is overloaded and you might have performance issues. So The show this will let you let you look at get that information So many things are not working right now, but they are very close to working I call these the boring issues because there's a clear path to getting them work But just need some work to be done so that involves user space dynamic and static tracing symbol lookups new tools tools that need to Read stuff on the remote target, but they read stuff on the on the On the local machine That is my laptop in this case There's a full set of issues. I've created in the issue list I would really welcome any contributions there Maybe have some more interesting issues That is not clear how to solve like One of the issues is that BPFT itself the reads and writes it does can cause or the other system cause it makes can cause a lot of Tracing activity itself. So sometimes that interferes with the tracing to a lot for something. Maybe we can blacklist the BPFT process from being created somehow The some tools can generate a lot of output. So We have to Reduce the amount of data that is being sent somehow maybe Batch the data together stuff like that Perf polling is another issue that we're looking at and also Logging is not working. So there are there are a couple of interesting issues that it's not clear How do how we solve but we have to solve them So that's all I had, you know This project is completely open source. So you can go to GitHub and download it and contribute to it I wrote an LW an article that goes in a much more detail about how it works and A Brandon Greg has set up nice resources on how to run these tools So I would encourage you to Go check it out. I guess we have a lot of time So we can just do a Q&A and any discussion So usually You usually as an embedded developer you have the kernel sources available to you because you build them on your host right So you just find BCC to that using the work that I did now it's possible for you to set an environment variable pointing your kernel sources in that and BCC will look at the kernel sources on your host not on the target on the host and use that to build your BCC program the box symbols are not needed in the general case To run a lot of these tools. They're needed to interpret the output sometimes like there are tools that Do profiling of stacks and stuff like that and so then the output will just have Have instruction pointers if you don't have symbols So for things like that you need Symbols on the host But otherwise you don't really need symbols Something Yes, so the first question About the other ways you can use this the way it's been written the architecture right now It's very plug-and-play kind of thing. So for Android. I have an adb module that does Android specific stuff But PPFT itself is written with a standard IO interface kind of thing So every command is like one line and you send that to be you can you can run it locally and just give it a command String and it'll respond to it. So to load a program. We give it the bytecode in basic base 64 encoding so I somebody else was telling they wanted to use it on servers and Over SSH so that's you know, that's definitely possible to add very easy to add. I would say The other question was failure mode. So right now it's very early stage. So we haven't really thought about that So any kind of cleanup? Will have to be done you know for Android what happens is the adb connection dies the Processes that was started remotely will will also be killed automatically So and then all the resources will be freed that way but yeah But clean up will have to be thought about this There's a lot of those kind of low hanging fruit issues that have to be worked on So yeah, feel free to pull the code on github and ping me if you want to work on this stuff and contribute Yeah, yeah, right now. It's just a single device but Potentially that could be done. Yeah, I don't see It's just a matter of somebody taking it and applying it to their use case. I don't see a reason why that cannot be done Testing Testing one two Mike check test test. Hello. Hello, everyone. Can you hear me? Can everyone in the back hear me? Okay Hello everyone, I'm Robert call. I'm the developer of the livery CMC project I was the co-founder of the livery wrt project. It's the project that essentially came before this livery CMC It was a fully free And very guinea-lack distribution had some different goals I'm also I've worked for a few nonprofits that worked for providing free livery education resources one of those nonprofits worked to make a Place for students to share their notes when they're done from class with classes and share them under a free Creative Commons license I'm in my free time. I do advocate for the usage of free software both in my personal and in my professional career I also break stuff for other people Sometimes for the better or worse So on before we get to talking about livery CMC we need to talk about livery wrt. This is pretty much where it all started Five of us at livery planet 2010 got together and wanted to create free embedded distribution a fully free embedded distribution for The focus is around about PDAs and music players. So long. There's this Fully free PDA device called the banana. Has anyone ever heard of this? Does it ring a bell? Awesome so It was essentially probably I would say it was one of the freest Computers that we've had to date at this point Considering that all the schematics are free the CPU was free It didn't need any microcode or initialization blobs and the like So we set out to with livery wrt to create a fully free distribution hope with the hopes of Getting the banana know our wife by the fsf the fsf's our wife program hadn't existed as of yet But there were internal talks about starting such a program and we wanted to banana note to Meet all those goals. So we need a fully free operating system that didn't have any non-free blobs or any Non-free packages if you will Sadly, I Ran that I was limited on time. So I kind of had to walk away from the project in 2011 I also had some design Issues that I wanted to fix but a lot of the people using livery wrt Really didn't want to Didn't feel that was really necessary. They wanted to keep it mostly a clean open to VRT With the Lenox livery kernel and we've all most of the packages in So what's livery CMCA it stands for the livery Concurrent machine cluster the name at this point. I'll get you in a minute. I doesn't really reflect what it does now, but It is a fork of open to VRT then we now open the VRT again And is anyone familiar with the open to VRT the for the forking situation? So I believe it was back in 2015 Seven of the open to VRT developers weren't really happy with the way that the open to VRT project was being managed So many of these main contributors went off and forked it into the lead project They forked it with the intent of making it a more community focused endeavor they wanted to Make anyone who's a contributor have voting rights and they wanted to focus on certain things one of the notable things that they did want to focus on was Focusing on the IPv6 stack in open to VRT and then about Was it three months ago four months ago? they finally Re-merged into open to VRT. It's essentially they took lead and Through the open to VRT branding on it and just now it's open to VRT LibreCMC uses the Linux Libre kernel. Has anyone heard of the Linux Libre kernel? So the idea behind the links Libre kernel and that effort is to remove all the non-free blobs and components from the Linux kernel So we use that instead because we don't want to advertise non-free software and we Don't we don't use want to use non-free drivers because part of the FSS Requirements for being a GNU free mean the GNU free software distribution guidelines We remove targets which have a hard dependency on blobs these targets can be Might require a non-free driver mean the source code for the driver is available We're dependent on the OEM to what the chips that manufacture to build the driver for it or it requires a loadable firmware blob which again the community doesn't have the sources for it or they're under a restrictive license So we remove those but we do leave in some targets which do might have a Soft dependency on these blobs, but still would function as intended or might function in a limited fashion So on some of our targets With the 802 of an AC stuff, we don't really have any drivers however the first generation of AC routers shipped with a removable mini PCIe or a Double mini PCIe type card, which could be replaced. So those are where we make the exception We don't officially support announce support for those devices, but we do leave it in So if people actually do want to use liver CMC with with those devices, they actually can We also don't include support for devices which require initialization blobs. So The rabbit's very pie is one example that people are familiar with however there has been Work progress being made in creating a fully free initialization blob for Free There isn't work Okay, so the Raspberry Pi requires a non free initialization blob There's been progress made in making a free software initialization blob as a drop in replacement So the liver liver CMC's original delivery CMC project as itself started in October of 2012 its original goal was not to target routers or some of these devices, but I wanted to create a essentially a dropping component that would allow for distributed compilation of Because at the time I really didn't have means to go by expensive hardware for building library WRT I Needed to use the hardware that I had just lying around and this was an endeavor to do that It failed because I just lacked the time not to mention dependency issues and other complications I'm there's a lot of other pack of stuff there's a lot of drop-in replacements that already do this, but I wanted something that was a little bit leaner and Then don't fell off the face of the earth at this point So the delivery CMC name doesn't really reflect what its initial goal was I was approached in 2013 by an OEM to They wanted a fully free embedded distribution because they wanted to Be part of the SSF RYF program. So We I sent you I couldn't think of a better name. So we're CMC stuck and we created a Maintained version maintain embedded distribution Liberty WRT had kind of Been erratic with Releases sometimes happening every six months sometimes a year. So we need something that was a little bit more up-to-date So the Liberty CMC project right now does target routers and NAS devices at this point in time We release every three months Six to three months and then usually will of course have some security updates depending on the severity of the security update So things that allow us to allow someone outside of the device at running stock Be able to get route or access to the router that would be really bad. So we always patch those Usually within 24 to 48 hours of being made aware of that Right now. We also we want to try to start targeting Divide having a 10-year lifespan for the distribution for long-term LPS releases and possibly longer the problem that we have with the embedded space right now is the fact that we While we have some we don't really have long-term support in terms of being able to build it 10 years from now Which it can be a huge problem So um Why would we so why do all of this work right now? We have open to VRT We have a whole bunch of other projects and we have a lot of stuff in this open space But some of it doesn't quite fit what we're trying to achieve And then I don't want So How many people in here? I'm sure a lot of people in here are working the embedded space right in some shape or form Right raise your hand if you do actually okay A lot of times many of these embedded devices they run non-free software or they're closed in some fashion They restrict users Freedoms in some ways or they don't they're not compliant with the sources so Many of the solutions that exist for providing a open source of free Distributions for these devices some of them require Provide blobs or non-free packages or packages which are encumbered by patent issues or The light that makes them non-free Many of these devices can tend to be black boxes, which is a huge problem Has anyone recently bought a car within the last 42 or three years Okay, how many of them included those infotainment systems Okay, how many of them have stopped working because of a forced update so as um So if it has anyone heard recently is anywhere familiar with fiat Chrysler and their you connect Consoles so recently they just force pushed a an update which causes it their Center console unit to reboot every 40 seconds So we get this with no mechanism to revert back Which of course if you're going to be have work with embedded devices and you're going to be doing updates You might want to have some sort of mechanism for reverting back So a lot of these people are going to have to hit their vehicles in and they're non-free So anyone in the community can't go and actually Fix these problems The other problem we have in the embedded world is forced obsolescence Who here has smart light bulbs? anyone How many have you have used that Philips hue light bulbs so With their first generation light bulbs they forced an update to brick these devices so they just stopped working This is a huge problem. It's not just this Just smart light bulbs a lot of these smart devices Where companies can see well, they're not going to be paying more. So they're not good They're just going to make it dead which I guess some that is like one way to go about your security model If you just don't want people to use it anymore once it goes in stops working You don't have to worry about updates anymore, but the problem is that The user in a lot of cases doesn't have control over this and a lot of these devices are running free software But you're kept from that. They don't distribute. They don't comply with the licenses And or they they do a lot of design They design it in such a way that it makes it really really difficult for you to go and run your own software on these devices Another mechanism is secure boot I spun arm for instance, so some of these devices have Signed firmware and they have signature checking which the keys are few is in the CPU die at the factory So you can't you yourself can't actually go and change these keys Without a lot of work and a lot of times. It's actually just impossible sometimes Manufacturers love to cut serial lines Or make them really really difficult to break out. So if you wanted to at all start playing around with it, you can't A lot of times if you change the software on the device The vendor might be say, okay, you can put whatever software you want on it. However You might get they might When you unlock it, they might make the bootloader show a scary screen saying this device may cause death if it's unlocked essentially usually says that they're not responsible for injuries as caused as a result of Changing the software on these devices that for a lot of people that probably would be scary They go they go pretty far with that one This isn't the case if does anyone have Motorola phones? Has anyone seen the scary message? Yeah, I've been wanting to change that I just haven't had the time to actually go forward with doing that and then some devices they'll go and They'll just brick those if it detects that soft unauthorized software is being used. They might actually brick it this was made famous with Release of the first Nintendo 3ds. I don't know. I don't get their naming scheme at this point But it was the first and 10 of the 3ds system that Nintendo released back in 2010 The SSF did a campaign about this and on the box of the intent of 3ds the first time it says that If you running off or I software on this device may result in it not being non-functional so there are many options in in this space in terms of being freer than What normally would ship on a lot of devices that these projects talk about many of these projects listed are great They they just are not great for some of the focus of some individuals Open WRT it does have some non-free packages Some non-free drivers the driver situation has gotten better. There are more free software drivers But some of these chipsets require loadable from our blobs and some of these can be really small like maybe a few dozen kilobytes or whatever Or it might be that they Sometimes it's a half a meg of stuff that has to be loaded to some of these chipsets They also include some patent encumbered packages which have a gray area, but And those can cause some issues and some Devices have blobs have restrictions where you can't reverse engineer these blobs. It violates the license that the firmware blob is under Also, they sometimes have restrictions on how they can be distributed. So does anyone here with anyone here running? any like Nvidia cards back in the Late the early the early the early to mid 2000s to late 2000s Does anyone remember the restriction on Nvidia's? Distribution of their firmware blob their portable blobs So so it's not just graphics cards, but there are a lot of devices which require these types of blobs Video Has anyone here heard of the DDWRT? I know it's very very popular And it does provide lots of functionality However, it is a little bit restrictive Some features on certain platforms do require activation. I told can fully sympathize with them It's their business model But some of the sources that they provide don't always build or Sometimes they build with some tinkering if you know which path to go down or Just not at all And I can understand sometimes it doesn't because it requires maybe a specific distribution which this is Really a huge problem, especially if you're wanting to build it to Five or ten years down the road. Sometimes it's not really practical to do that sometimes though You might need it because maybe it supported a particular platform But newer versions of what available is not there So you might want to look down and see what exactly they did to get support working Open them bedding in Yachto. I Thought they're really really great platforms. I mean they do distribute the vanilla likes kernel which FFF does have a problem with but The other thing is blow it in my view it's For what it provides. It's a great tool and it makes it really easy to build for embedded platforms It's just a little bit too blooded for our focus The bunch of cores available is anyone using it in anything right now It's essentially they're targeting for Embedded in internet of thing devices and sometimes some networking appliances This has changed a little bit. I probably shouldn't have that but a bunch of cores a little bit bigger than our target platform at this point We definitely our target devices are Half of our devices that we support we only have four megaflash and 16 to 32 max of RAM so That's a huge problem and a lot of Conautical and a bunch who have been working to reduce some of these footprint issues, but um This is the way they're going. I for what they for what a bunch of core provides it is definitely a great way to To ensure that you can easily roll back images. So there's like a There's like a last time I left there is a one gigabyte fallback for each system running on it So you essentially have two installs one to fall back if in case something breaks No, what no, you don't you don't want this Wait Wait, no one's speaking in here, right? Because I mean Okay, well Did I come in okay, I'm still speaking I just Yep, okay Okay testing, okay, I think I think we're good again. I mean um, so no so um again Within the last seven months. I have not followed it went to core. I will admit that they are getting better in terms of flow Okay, well Okay, I think I'll continue now So they are making they are making they are trying to work on Okay, I don't know Testing testing Every time Wouldn't it be easier to Okay, I don't know So I don't know if they want anyone changing that because I Think Really really high Okay Yeah, they're gonna change the frequency it's not gonna I Don't have the handheld one Testing am I good to go? Okay, awesome Okay, so So again, my information on a bunch of core might be out of date again, they they are doing things to rain in the bloat however We want to make we want to make our core a lot smaller than what their vote with their goals I mean we sort of I'll talk about that later There are those also Embedded Debian, but again, it's a little bit too large to be running on routers And if you're looking for package management, well the kind of all bets are off at that point considering again How most of our targets are only have four megs of flash and might only have 16 to 32 megs of RAM And we definitely don't want to make those targets obsolete. We should definitely be able to still Provide a router distribution that will fit in that space given and with the minimal amount of features that people are looking for So again, package management comes at a cost. It does add just a little bit of overhead in terms of resources and a lot of these embedded devices, it's Not really packed practical to have full package management support But we are trying to we this is one of the areas that we do want to work work on as the Liberty CMC project Many people Complain that manufacturers don't update their platforms often enough, but the real problem is a lot of these manufacturers Don't really focus much on really focused on their security model you should be able to have these devices working for long periods of time and Prepare to be able to mitigate for some security issues that might pop up Properly configure device definitely goes a really long way in terms of preventing a lot of Holes that might pop up Not everything needs to be connected should need to be connected to the internet I mean how many people in here can help to own a smart fridge or smart device or smart light bulbs. I Mean, it's a convenience thing, but I'm not I personally don't want all my light bulbs I mean, I can understand having a separate network Completely isolated, but a lot of these people are connecting them directly to the internet in some way of fashion and so it Pretty easy to you know, turn on someone's light or possibly do something even more dangerous. I Mean, there's no toaster ovens that now run Android. Has anyone seen one of these? I Mean there is definitely a market these things definitely go a long way And they do make our lives more convenient, but we should definitely think about what networks that they're being connected to So also a lot of these devices should all fail gracefully have Has anyone seen message board or signs that have a Windows blue screen of death or a kernel panic? Yep, and a lot of times That you really shouldn't be happening It's just showing Information sometimes it's because the device overheats or sometimes it's because some resources aren't available or there's some other weird bug A lot of these devices really need if they're being used in critical pieces sign. I would say as a critical device, but They should definitely fall back gracefully. They shouldn't have to They should be the manufacturer should prepare them device in the event that some component of it will fail and have a plan for dealing with that So these are all the things that I've talked about at this point that are wrong with the embedded space we would I like to address some of these issues with using Libre CMC as a platform for moving that work forward so I'm going to talk more about Libre CMC, but I get a lot of people saying well Libre CMC doesn't support X Y and Z so I'm not going to use it. Okay. Well, Libre CMC doesn't support Everything in the Sun given our constraints and the fact that we want to be a fully free software distribution We can't support everything under the Sun because of that. We can't support some non free project packages or device drivers and There are people out there that that really that do want care about that and want that focus So that's what the Libre CMC project is here to do We want We also want to focus more on security and bloat as well There are more platforms that we would love to target, but we need to get our base down to in the region ballpark of two megs of flash and eight megs of using only eight megs of RAM This may not be practical today given what people really want in their devices But we're certainly going to try and it might involve a lot of work We're not competing with projects. We are standing on our own right for these reasons that I've laid out So if it works great if it doesn't well Maybe it might be we might be able to make it work in the future and maybe a manufacturer might see oh there might be some benefit in releasing the firmware blobs for Particular chips that they do that. That's great. We can then support that device, but at this point in time we can't so right now the Libre CMC project is essentially two projects one is Focused at consumers and people who actually want to use this in on their routers and other devices and the other is research which of course is unstable and Some of it doesn't see the light some of it does but Our consumer-facing project helps fund the research side of the project so We work with a few different OEMs We encourage them to make a small donation if they're going to ship with Libre CMC. We encourage them to make a small donation for the hosting space possibly of custom images and source code distribution disks and the preparation of those so one of the What we push OEMs to do that are shipping our devices devices with Libre CMC is We want them to provide a source code disk with the device and it must build on a recent GNULEx distribution We work with them to prepare these disks to make sure that they build all the sources are available in the respective image and That's pretty much it and that's a huge thing because a lot of These devices running GNULEx Or Linux the Linux kernel They don't provide source code disks and half the time They you might be able to write them and they're you're lucky if they're going to respond and sometimes if they do respond They might provide you a disk, but the what you get isn't always the source code that they distributed and a lot of kit companies just feel oh they can just They don't have to worry about it because the litigation aspect of a lot of the licenses is pretty well the enforcement is Constantly and there's not no enough people doing enforcement in that area Also, it's a nice thing to give your customers the source code disk that actually builds an image that can be flashed on it Considering that they can then expand the capabilities of the device later whether it be Now when the device releases or maybe five years from now when the when the device is on a site is on the second-hand market Why these many people don't realize that not everyone has the resources to go buy a brand new router every two years or something ridiculous like that And in the case of Liberty MC sometimes some research Some stuff that we do doesn't always pan out. We just move on it's expensive work And it does take a lot of effort So that's a great question. Thanks for bringing that up. So a lot of work has gone into Open to VRT and lead and open to VRT again for the idea of reproducible builds. So Some aspects of Liberty MC are some aspects and packages are reproducible. So not so much some kernel stuff A lot of the kernel stuff is reproducible On some platforms, but not all platforms lead I Think it might have gone on the back burner, but I'm not sure what the progress. They've made lately But again, we've received see since we're basing it on Leeds stable 1701 We do benefit from some of those The stuff that we does Did I answer your question? Oh No So we're gonna get to that in a second it's um I didn't okay Hold the questions for a minute as we're still gonna move along here. I guess There's it. So Where's it? I think wait So a lot of our focus again in terms of research is into to blow Our what our build system right now. It's currently using we're pretty much just a fork of lead Lengthly recurrent or removed. So we're using essentially modified version of build route To put it simply We'd look to look into We want what we would like to focus on is to have a build system that can build on systems for the 10-year span and the reason that's difficult is because of Dependency issues so a newer version of certain components Might break compatibility down the road. So that's a huge problem Especially in terms of build systems for embedded devices Build route does a really great job at this Reducing the breakage, but it's it's still quite It still does it probably once every two to four years so if I take a version of OpenWRT or any arbitrary version of OpenWRT or LiveryCMC and I want to build it Four years from now. There's a large chance that some of the things will break. The other problem is Some packages might drop off the face of the earth So if you don't actually have a copy of that package then it might fail if the blink is dead So we want to reduce all those things anytime. We have a dependency. We always make sure all those All those dependencies are available for each and every version of LiveryCMC We started doing this early and we plan on keeping all those packages For long spans of time as long as the LiveryCMC project is around It's something that we need to address I guess in the free software world where Projects don't Projects some projects disappear. Sometimes they're hosting changes. Does anyone remember Getorius at all? Anyone anyone you actually use Getorius? Anyone remember the breakage some of the break. Oh wait, Getorius and Google code Both of those were depreciated and there's a lot of projects that were dependent on those platforms for hosting and when those went away a Lot of the links broke Google code they did keep some of links on for quite a while I think they're getting about to the point in time where they're going to start Making them completely dead But that's a huge change and a lot of time if you have this old version of the project you then have to go find Where they're hosting the code now and they might not even have that old version of the package And this is an issue that we need to address And that's what I would like to address with the LiveryCMC not just by us hosting some of those older packages but also Coming up with ideas to better manage these dependencies 15 years long extended long-term support is probably going to be sound ridiculous It probably is ridiculous in terms of effort needed to actually make that happen It requires probably giving up a lot of Shaving off a lot of different dependencies of a build route and a lot of the tools that meet packages depend upon for building but If it might be worth it it might not be that's probably going to be up to Me and the community to decide on that it may not be a great option So I mean right now we're targeting routers and controllers and PDAs are out of fashion at this point. Well, they're kind of out of fashion in 2010 But I still like the idea of having a small Organizer like device that I could control all the software on and communication applications We would like to Get LiveryCMC Some of the extended packages. We'd like to get support ready for some federated and self-hosted services like on cloud or and Like the new social or pump. I owe is anyone you are diaspora. Does anyone use any of those? Okay, so we would like to make it so that we have a nice wing core that you can install some of or we Actually package up some of these things and make them available to people to use because right now a lot of these services are really really difficult to install and deploy especially for people who have no background in GNUX or The like so we'd like to meet be able to make it so you can just download this image and Maybe answer a few questions and it would configure it for you But this is a long way around down the road I know there's other projects that do focus on this like anyone heard the freedom box project So that but they're targeting raspberry pies and they're using embedded Debian But we we kind of don't want to go down that road So our first Area, I think So our first area is research design We want to not just limit ourselves to the Linux the vanilla X kernel There are other Linux variations Linux kernel variations that we could possibly use You see Has anyone heard of UC Linux? So those do run on some even more limited devices Again, I'd in some Area divide areas. I don't think that this is still the right route to go I could be wrong on that but um any other paradigm is going to be really expensive in terms of research There's a lot of cool projects going on and we'd like to probably take some of those pieces and work run with them if resources allow for it We already talked about dependencies So the build system again, we do we use Essentially, it's build route. It's my modified by open of air T And that's pretty much what we're sticking with everything is based around the fact that we use build route So yeah in terms of bloat though one of the recommendations from One of the Routers is the However, it doesn't really fit well if you want Flash plus have a full Sadly right now we're going to removing the package management and While I Now a lot of people might not have an IPv4 connection provide to them right there ISP, so they have to use So again Liberty seems he's not for everyone, but if you care about software freedom and pushing that idea forward in the embedded space That's what it's that's what we're here for we want to make long-term support much easier and guaranteed You should be able to take the sources that we provide you now and build them Hopefully five and ten years from now if you follow the dependencies As long as we make sure that the dependencies are cut down in such a way that they'll be guaranteed to be supported Down that far down the road That also helps in terms of compliance. So a lot of times Has anyone actually got requested sources and then they give you build instructions that are very big And at the end it says that you need for Dora course three or something like that. Has anyone run into that? Yeah, so A lot of times you might not be able to find that version of Fedora course three. Luckily we have the internet archive But years ago there have been versions that I just you couldn't find anywhere. They weren't uploaded yet They weren't the right person. You didn't get a hold of the right person. So You're just out of luck you can't you couldn't build it and I've actually have run into that recently because Version of you, but I couldn't get to build on any modern System and I need to I couldn't I was hoping it would build on this particular version of the door if I couldn't find it so We are open to incorporating any new features as long as they are free software and they don't add too much Blow for additional dependencies that are required. I guess I'm going to leave it up to Q&A Anyone has any questions? I That's not we given the the routers that we build for it's not entirely That could be done in a lot in a better fashion On our AR 71 xx targets. I don't think that's the case. There are I mean everything else is a module But that adds more bloat to the to these images More static linking could possibly be done, but we can we could probably modify it slightly so that we could do that That would probably save a lot of space It's just a matter of time constraints. So just maintaining what we have now Does take some some effort in terms of the rebranding Making sure links we read kernel work sometimes some patches their conflicts. So we have to do other patch those out or rewrite certain patches to actually make the thing built so Hopefully down the road we can approach blow from that perspective, but I don't We could be doing that better. Yes, that's a great point. So any other questions? I have a comment and a question Okay So the thing is that no matter how small you make it somebody gonna come with a smaller device and Where that goes is that remember you mentioned Yuclunux or Yeah, I think that the point behind that is it doesn't require an MMU So my cork on something like an M-series Arm machine They're really really small things typically around M-series arms Do you have Do you require an MMU to run? Sadly at this point. Yes, we do require that. I don't think I don't know if there are there any I don't think there are any platforms that that don't require one that's open to beer cheese, of course I could be wrong on that one. We Could possibly do that it just be a prior matter of using the Yuclunux kernel, but I haven't built it in quite a while So I don't know what I like Is the real problem The problem with this is Is time because it's all this stuff does take time to like play with and actually see how well that would pan out Right. Um, I mean The thing is is we are getting more and more these resources in this space. The problem though is those are not guaranteed Has anyone looked at RAM prices lately? I don't know They've recently in the last year or so or two years. They've gone through the roof So those types of fluctuations can happen and what the manufacturer decides to use Might change and what the manufacturer has the option to use so Many projects don't really see this as We need to shave off every single light ten kilobits here ten kilobytes here. It's Yeah kilobytes, sorry But some people don't see the justification it but it can be a heck of real profound effects when someone's going to try to build some of these devices Is there any more questions? Yes, that so that does come wait so Can I address that or do you have more? So um, yes, I do realize that that is a concern in this space A lot of these blogs are getting pretty huge and if anyone's familiar with the it's in the last year There was a huge security vulnerability with some broad con chipset that Were Revolved around some of these blogs and we as a community couldn't fix them. We'd have to wait on Broadcom to actually go and push up a new push out a new update I understand that we're not going to be liver seems he's not going to be popular with a lot of people for that particular reason We I didn't start this project because to take easy way out That a lot some of us you care about software free. I want to be able to See what that hard what what the software on our device our embedded devices is doing and be able to fix issues that come up So Okay, so we do support some other targets and not officially though for some reasons we do have some We do support some of the all-winner boards that use the a20 So see The eagle bone The lack is not officially supported. I'm If anyone can answer this they I'm still having issues with the license around one of the the blogs that are initialization blocks Not really a blob in the first is available for it, but under some Texas Instruments license anyone to clarify that? Okay, cool Yes, but what about I thought there was a there's a component that you have to build drop in to Get it to run on the beagle bone black Mainline you would does have some initialization blocks sometimes around memory initialization and some other CPUs set up stuff. So we should Is also the other people are Right Right, so I'm not an expert in this we're not gonna initial We're gonna for some things if it looks good or clean like if the art we can build a male image with it using only free software Then we'll support it But we only want to announce support for things where we're guaranteed that all parts of the stack are free software Yeah, we're still in question I have the Okay, thank you. Thank you for supporting the project that that really does help Run run what exactly So our bottleneck right now we could and the problem is that again wife 802 open AC chipsets There's no one right now that we know of that is releasing Versus for the loadable firmware and there's a lot of things that we're actually restricted from doing because of that as well In terms of making mesh networking that's slightly more difficult And in terms of initial bugs Not at 10k had quite a few bugs initially so our bottleneck right now is mostly Blobs in that case, so that's the reason why we're not supporting any slightly newer targets We also have been looking at possibly Getting a more modular router out the door that we might be able to support faster Have faster CPUs and things so we can do Some cooler things and actually have some resources on some of these devices, but as of right now I Mean if it's free software will be more than glad to support it So yes, in some cases There isn't there's not really so it only applies to five gigahertz devices and the approach That TP-Link made was not really the right way to go But that's kind of what we feared initially that companies would do because they Wanted because that's if you required that you couldn't make the Wi-Fi chipset operate and a An approved way so TP-Link went throughout of just walking down all the firmware You pretty much had to TFTP flash it you couldn't flash it from the web UI for a lot, oh, and they went to further and cut the Serial line, so you have to solder to solder for solder pads on the board to get a serial interface Because of those so the serial is there they just cut the line and put to solder to pads there four pads there, so Yeah, oh Okay, so So there is someone else who was Did a lot of work around that whole thing What's his name Eric what yeah Eric Schultz? He's given a few different talks which are online. I believe he talked actually last year about this very same thing It hasn't affected yet But it could definitely affect us At some point actually I would like to add one thing some so some vendors have actually gone the route of having Doing signature checking on the particular loadable firmware to get around that but I mean The only real way to do it with the F9k stuff is to just lock down the whole firmware People from doing that It's not really it's not really. We're really You would just go that route and they they tried really really push that What we said wouldn't happen when TP like actually went and did the exact same thing that we said would happen But can you Speak up a little bit louder like, you know So the other thing is on our targets do we do Try to focus on targets that allow you to flash an image through the web UI And we made the exception for some TP link routers where you have to use TFTP But you did not actually have to open the router unless you need to debug something that had failed But they start locking it down So in TP link, it's not it's kind of probably wrong to actually say it's a full lock down it's more or less a Restriction on the web the web UI is that you would load the firmware to and so they made it so you could still TFTP flash it you just had to name it to a certain strength So what you do is run your TFTP server and see what file name it requested and then rename the firmware you're trying to flash it to to that name and then it would Magically flash it it didn't do any signature checking on it Yeah But it's still more difficult for regular users When you try to get regular people to use some of this stuff, it's it's challenge especially in terms of the steps The hopes that you have to go through Is there any more comments or questions? Okay, I believe we're We've got two minutes Well, thanks, I'm glad that I was able to make it and my flight was We luckily were able to bump our flight up otherwise would have missed this because of Thank you everyone Testing testing just call my pocket. Oh shut the screen off Okay, yeah, and if I want to toss the presentation over there All right Conference Being Thank you, so yeah, I had a slide about this actually So my talk is called Linux I scored C and me and I'll take care of it and me part first He did go through it I'm an electrical engineer by schooling But I have a lot of science experience as well I do pretty circuit board design and some of these boards up here Are boards that I helped on or I actually all three of the ones that are shown there are boards myself and He said that I worked on the pocket Beagle I actually worked on the pocket bone, which was the predecessor to the pocket Beagle But it was kind of the I do a lot of the Linux board support Colonel you boot I do a little bit of real-time stuff and Some user space but mostly low-level I've worked on several community initiatives including the big board org GSOC 96 boards the EALE training that's going on next door and key cat Training is coming up in Reno next month. So anybody's interested I'll have some information about that on social media Okay So we're first we're going to talk about What I squared C is Now we're going to run through some example devices Talk a little bit about the protocol and we're going to skip to the I squared C subsystem Give a little history there some links back to different resources Talk a little bit about each type of driver. Well, not all of them. I'll mention them But there's a there's a few other ones other pieces, but I figure there might not be enough time to go through it all Then I'm going to talk about instantiating devices and there's quite a few questions about by tree in the last talk over in the other side of the wall So I'll go over a little bit of how to instantiate devices using different interfaces Talk a little bit about the user space tools Character device a generic character device for I squared C and then I'll run little short demo and Pocket beagle with Baking bits cape so I scored C Stands for inter integrated circuit. So it's kind of a mouthful But it was built in 1982 by Phillips So it's been around for a long time Synchronous multi master multi slave interface Not typical, but it's it's possible because of the type of signaling it uses Half duplex meaning it can't communicate in both directions Simultaneously, that's where it differs from SPI He is open open drain signaling so If two devices are trying to be masters at the same time it won't cause electrical issues It'll just follow the transaction The arbitration to determine which masters which but for the majority of cases you're a single master multiple slaves You only have two wires for signaling Sda and Sdl they are Usually a hundred killers signaling speed not super fast There is a link down there for Wikipedia She goes on and on about the protocol. I really recommend taking a look at that. They did a really good job on that article But here's some of the highlights so it's a seven bit Bus addressing protocol At least originally and the original back also called out a hundred killers and then Progressively got faster and faster and added different features so version one added foreign fast mode with 10-bit addressing which is part of the protocol and Then they kept boosting it up The really fast ultra fast is actually unidirectional being that it is push-pull instead of open drain that means it'll drive both low and high states and and so it It's just one way and then Intel came up with this thing called SM bus which is a subset of I squared C which is used on motherboards particularly for monitoring devices temperature band control stuff like that they constrained the protocol electrically and they added some optional software adjusting address resolution so I squirt see you it doesn't really have a way of Broadcasting back What's on the bus so that that's kind of the limitation of the I squirt C in SM bus Optionally takes care of that So here's some example devices that would use it I squirt C We got the real-time clocks a problems analog converters, which I use quite a bit Sensors temperature pressure accelerometer, etc a microcontrollers can actually be Master or slave and we'll talk a little bit about that What the difference is There's touch screen controllers Usually four wire resistive if they're I squared C GPIO controllers and the monitors and TVs have a these things called DCD, which is kind of I squirt C for Detecting the the parameters of the monitor when you plug it in so that it automatically works so here's here's a few sample circuits that I actually I Granted from a couple of my designs This here up top is an accelerometer That's actually what was on This guy. There's an accelerometer on here, and I will show you guys Accessing that's accelerometer towards the end This guy down here is a PWM controller this I squirt C and It was on this board called Robo Mezzi and it's also on some data for boards different breakout boards So you can drive up to 16 servos from the two I squirt C lines The one of the important thing about I squirt C is you have you have to have at least you have to have the Pull-up resistors on there. You'll see that there's pull-up resistors there They're up to 3.3 and the signaling voltage is dependent on the controller you're interfacing with and If you don't put those resistors on there, it just won't work. It won't pull up so the open collector could pull it down and If you don't put the right values Then you also have problems because of the rise times There's a really good talk by Dave Anders The talks it goes into detail about how the mechanism works. I'm not going to get too far into the hardware stuff If you're interested I can point you to that. I was going to add a link, but I forgot Okay, so here's a little Picture of the protocol This is just the the addressing in seven bit mode It's a basic transaction so what happens is the lines will fall Low in a specific sequence that to signal the start of transmission That's called a start bit really just a transition sequence and then the address bid is shifted out Along with a read write signal Which will be one for read zero for write and then an act bid will be either zero one based on the acknowledgement and then The data comes out either if it's a if it's a read The slave interface will send the data back on the on the lines If it's right that the master will send the data out on the same line So it goes it's bite. It's not a bidirectional Communication but it goes either way so it goes from the from the host to the slave during a write and Then you'll have a little bit from the host and then the rest comes back from the slave with a read And then there's another transmission Signaling the end now typically a device will have another this data is actually per device. There's a definition of the internal chip addressing which will be you know what register internally and Then it'll have the values you want to send to that register or what is read back from it So there there is a bit more to the protocol, but it wouldn't fit on the slide with you know being able to see it all So here we'll get to the to the Linux portion of it the early implementations were pretty early on in the two the two point X era and But they were there's one out of tree and one was kind of tied I think the out of tree one was the LM sensors but either way eventually Greg Carhardtman came along and kind of pulled it towards the new Developing device model which came out in the two five and the two five early two six era and then David Brahmo and John Belver did the standard device model port and two six and a lot of things happen since then I can't really go through everything, but Then Wolfram saying is the is the current maintainer We'll talk about how you would get to a hold of the maintainer in a minute here so Here's an overview of what the subsystem looks like There's a couple of pieces that are required for the overall system You have the the the core peripheral inside of the SOC or Linux capable processor We'll have a peripheral controller internally and this thing called the adapter will allow the the the the core portion of the I squad C subsystem to talk to the host controllers peripheral adapter or whatever and then there's an algorithm that's usually built for each Host they'll have an adapter which will It's pretty much pass along a function that allows the core to talk to the hardware essentially And then on the on the top side, which is where most people are interested you have the drivers the client and The driver driver it's kind of a silly name, but It's we'll go into that a little bit so The the the slave device has it in a instantiation talks the ice core C core, which abstracts the the SOC's peripheral interface So that it's the same across different platforms. So your your ice core C driver shall work across the whole ice core C subsystem and that's part of the device model and Then yes, you can map it and go There is mention of ice core C dev, which is a user space interface. So we'll talk about it towards the end okay The Linux ice core C subsystem mailing list and if you're a kernel developer You want to get to know how to use the mailing list? So I get this is actually straight from the page I cut pasted it in You can subscribe to the mailing list for the link there and there's the the web interface for the mailing list and The mailing list is is your entry point into the subsystem For mainline. So if you wanted to do mainline development, you'll have to go through the subsystem which the subsystem maintainer will Monitor and then he has his own separate git repository Which we he'll he'll apply the the patches that you send to him And then he sends them up with a merge request to Greg Hartman, which eventually goes up the stream to the mainline kernel. So This is going to talk a bit more about the algorithm and adapter so The algorithm driver contains general code that can be used for a whole class of ice core adapters This is actually directly from the documentation, which I have a link down here. I Can actually show you that it's towards the end We have time the actual documentation and each Specific adapter driver depends on one over them or includes its own implementation Which I've stated previously okay, and then for device drivers, they call they have split to driver and client and It actually says that right in the documentation a driver driver It's kind of silly, but That's what they call it This is essentially the iSquad C device driver where You'll contribute if you wanted to add like a new touchscreen controller You'd create one of these type of device driver for iSquad C and iSquad C subsystems The protocol Drivers are spread throughout the drivers subsystems. They're not contained to any one specific subsystem So if you have an analog converter You'd be in the IIO subsystem or HW mon hardware monitor So the the the host controllers are actually in a specific driver directory drivers iSquad C in the kernel Source where these these are everywhere they're across the board is any kind of device you can imagine That has iSquad C will be spread across the kernel drivers I'm gonna talk a little bit about this, but I don't have a whole lot of time to go into it Recently, I think 2014 iSquads or yeah iSquad C maintainer added slave support what that means is if you're if your controller supports it Your Linux SLC can actually act as a slave instead of a host controller that means you could actually talk from one from a Linux SLC to another SLC and It'll emulate of a slave device and there's some bindings that are required This is a SysFS interface that you use to kind of poke in hey, I want this particular type of slave The slave Interface documentation is down here at the bottom and there's a nice presentation from the iSquad C maintainer down there at the bottom It he goes through in pretty much pretty pretty good detail okay, so I Haven't really covered like the anatomy of of the driver. I didn't think I'd have enough time, but maybe I Can show you at the end But here's how we how we instantiate devices so one method is the device tree and this is preferred method for you know arm and Other you know device tree compatible devices, so the device tree Binding will say hey. I have this particular device Which is specified by the the register and The compatible string so the register is iSquad C address and The compatible string is what binds the driver to the registration here I can show you inside of one of the drivers where that is and Then say you have multiple devices on the on the bus Then they'll use head head further Registrations this one is special because it's a iSquad C controller for GPIO So you have to specify that it's a GPI controller There's actually a GPIO controller on on the bacon bits, but it's SPI controlled Okay, okay now back in the day before the transition to Device tree and when I first started There's thing called platform devices own platform Devices that are a platform bus is still used But it's abstracted a little bit. This was actually a physical C file in arc arm Mach whatever architecture and then a board file and board file contained the registration for your different devices And it was a C code I don't really use this so much anymore There are maybe a few cases for this still around but they kind of transitioned away from it over time Okay, and there's a way to do Instance eating from user space, which is a neat feature as long as your driver supports it and You could you echo the driver the driver name address And they have a sys bus iSquad C devices and then you have for each bus that you have on your system You'll have a different iSquad C dash whatever Edit new device I didn't want to break it down too much if you're interested in how to do that. There's an actual there's a whole For all the instantiation methods. There's a documentation in the kernel source directly okay, so We have some user space tools they're used Kind of abused at times A character device which is kind of an abstracted device which can access pretty much anything on a iSquad C bus The device note will come up and dev iSquad C dash Whatever the bus number is and then you have an iO control that sets a slave address Which was the number that the address in the slave protocol and then you have you can use simple read and write Mechanisms so you can say hey write this register a read from this register There's in a capsule a version called iSquad C SM bus rewrite If you don't understand that notation I can break it down for you. Yeah. Yeah Well the protocol depends on the device you're accessing but typically You send the address of the of the device which is the iSquad C address right then you have an internal register address Which is set as a set of data, so you'll send the address and Sometimes there's other bits involved depends on the control or the controller on the slave So you'll just send a sequence of bytes to it and the sequence of bytes will determine what address Register internally is addressed so that's part of the protocol essentially What's that? No the recommand is kind of raw whereas this will actually break it down into separate like it I'll actually do a little bit more for you and the dev the dev interface is also linked down here at the bottom. I know I've kind of pushed everything off to the links, but this is more of a higher level kind of thing and You can you can go and kind of dig in if you're really interested These these instructions are pretty important for the iSquad C user space implementation there's a whole slew of defines and and I didn't want to have to put them on the slides. It'd be slide after slide And then there's some tools that use that interface one is called I2C detect which it'll just kind of They'll go out on each address and kind of see if anything acknowledges and it'll print it out on the screen a nice little tape tabular form It will ask you if you really want to do this when you use it. It's kind of real kind of tricky about it. This is This is used for mostly for early testing and I can I'll actually show a demo of that as well as a demo of a driver running Here at the end. Okay. So there here's the demo Of course, this is always where the problems start right? Okay, let me sure first. We got to get the count flow up there, huh? Get out of that probably not large enough is it I'm going to plug in the bacon bits keep We use the serial debug header because it's just what I'm used to I'm actually in an it currently in a mini-com instance I believe it's the wrong header Should know this There we go So for the first the demo amazing the standard Debbie an image Debbie an image takes a little while It's a full-blown OS like All the Gadgets and what have you you'll notice I have a build root File system that I'm going to boot up next for the next portion of the demo That's what's a lot faster. That's why we don't do live demos All right Debbie is okay There were prompt here or not. Can you see the prompt? There it is. Okay? so I'm going to show an example of Oh, I know it's disappeared Oh, I'm guessing it's down further. Hold on. I'm going to use I support see Utils to access the accelerometer on the board directly This is good for when you're first trying your hardware. You want to say hey, this was hard work working This will give you a real real good idea real real fast So first, let's do an I score C detect. See what comes up if I take that till the front there And then I think it's on bus to it tells you what protocols are supported by the bus Tells you the different adapters that you have Of course, I don't remember the command off the top of my head again Q either way That's our the last one you try of course I Can tell I don't use this often But it's very useful when you first start you see here. It says one Charlie or one C That's the address of the device The accelerometer that's on the board So now I'm going to try and I'm going to poke at the registers internally on that on that device to set it up for sampling on the on the Converter this is going to poke a register internal So this to a is the internal register on the I squared C device and this is the value that I want to write to that register That one turns the it's shutting off the conversion actually and this one here is configuring the conversion to 2g or 2g Okay, and then we got I'll do that right And the last piece is enabling the conversion Okay Yes All right, so I've written out some registers and it's doing the conversion internally and what I'm going to do is I'm going to use a the I squared C double command Which I It's really just a read across a bunch of registers a watch Or why I want kind of loop to so that it shows the values as it kind of moves around Okay, so you see how those you see all those registers are kind of wiggling around Those are like the raw hex counts of the different Excesses of the accelerometer. So if I move around the device You'll change the values that are coming up down there Not really readable in this form at least not easily readable It's a signed value. So you'll see that you'll have F E F Whatever and then the number it's a for this one. It's a 10-bit conversion so but if you're going to Really use an escort C device you should make a real driver So what I'm going to demonstrate next is The use of an I an industrial IO or IIO driver to access the same device So Well, I got to get the other I Was going to put it on there directly, but I kind of last minute did this one This is a Builder file system. Okay, you see that? Blinding fast. All right, so Let's log into this guy All right, so tried to access the same device on this with this build because it has a the registration It won't allow you to access it with the other tool. So If I were to do this I squirts you set on here, it'll probably walk at me That's advice or resource busy. I mean, that's the driver is bound to the I squirts he bus at that address and then Let's see. Let's go back into the word of our registration comes through I scored C and we have a device this directory So anything that's that's registered will show up here and you see there's the the second bus 001 Charlie, that's our device that means the platform Register there was a registration for this which is in the device tree And I can I can show you guys that if you'd like to see it So yeah tells you that a bit about the device so you see that it's an industrial IO device so that Think about the the device model is it's kind of recursive and so if you register something on I squirts you buzz and It comes up as a different type of bus. It can kind of go down from there. So I I Is in Now bus the whole there's a whole other talk about how I works But we're just going to show you this that the device isn't attached here Okay, so you see there up at the top you'll have The accelerometer scale and then accelerometer readings are all broken down into virtual files So if you can't those files, I'll give you the raw account. That's a signed account. So If we do a Y reading I guess why would you do this to? And that shows you that it's oriented either right side up And then you have one for each axis. I've created this little special script that accesses the Accelerometer So let's take a look at it first Okay, and what it's doing is it's getting those files into variables and then Rendering them out to the screen that I'm not really complicated So then you got Just and there you go. So you can watch it kind of live If they move it around you can see the different accesses change so The protocol driver in this case isn't is the industrial IO driver The adapter driver is an old map Controller driver that's based on the so see on the bottom here So I guess I could show you some of the various pieces if we have some time Do we have some time? Yeah, it looks like we have plenty of time So let's take a look at Some of the Different pieces here So a lot of those links I showed were hyperlinks But they're also all the documentation is actually in the in the source tree directly. This is the kernel tree here and You'll see I have them all full Different Sub-repose origin is the mainline kernel here and then Documentation And a lot of the text that I kind of just kind of scammed out some of these files If you want to get into more detail on anything Particular it's going here and say oh, I want to learn more about how the slave interface works. There's a separate file for that One thing that I didn't mention was multiplexers or muxes This allows multiple devices of the same type to be on the same bus, but multiplex This is useful because the address space of I squids is quite limited and there are going to be times when They overlap or you want to have more of the same device Sometimes we just use pins change the address slightly, but when you're on that option you use a multiplexer So anything that that you're interested in no should be here Make this a little bigger Okay, and then let's go to the Registration I guess that would be for the platform registration For this system You use the device tree This is the controller registration. I'll be in here I scored C2 and there's I scored C2 and What this does is it maps a compatible string which will be in the driver and It gives you the registers the platform registers our memory map registers inside of the so see and This is used to pass the driver These are kind of fixed Usually have to worry about this if you have a system that has Good Linux support. This will already be taken care of for you But then which one Hw mod Yeah, I'm not real sure on that one. It's a TI specific see how it has PI in the front So you'd have to look at the driver and see how it parses that what it does with it So the driver for the ice court C court C the drivers and then anything that's a Bus or a an adapter Would be in the bus's directory so every so see or implementation is going to be for the for the Adapter side We'll be in here and the TI one would be I think it's High-square C All map you'll have a compatible string here All right, I guess it would be Townthorne's in So it's a platform driver. So there's there's gonna be a lot of boiler flake code here And then you have the open from our match table Here and you see you'll have the compatible strings and And when it when it probes what what it sees a match a compatible match It'll probe the driver and this data will be passed. So this data will give you a little bit more information this isn't necessary for all different controllers, but they like to combine the Controllers for different like series of processors into the same driver to eliminate redone redundancy in the code and then you have That gives you that tells you what's going on with that particular device and then There's quite a bit involved in making a Pay adapter driver check really couldn't cover in one presentation. Maybe it could be a separate session but There's gonna be callback functions that are called from the iSquad C core driver to Accomplish what you need to There's a lot involved so let's Quit out of that one, and then let's look at the protocol or a driver Yeah, the name of the accelerometer a camera it's MMA something now I don't think about it aren't that 84 52 There's a there's a series of them that are going to be supported by it and 84 53 is the one that we're using on the red one of the of the Cape, this is the one that was we usually are used originally, but we leveraged stock from GHI So we we had to go down to this guy, but no big deal and so this this is the the driver that's Quite a lot of code as well Let's go to the end where the important stuff is I guess So here's your typical, of course the driver will have this driver struct and it'll have some callbacks here and Pointers to tables that are useful in matching the device so again, there's a device tree Pointer table and That'll match with the device reentry So we put this free scale underscore 84 53 into the device tree under the I squared C sub node and then off it goes And it'll probe that advice on boot if the driver is loaded and if the driver is loaded dynamically it'll detect and load And then what will happen is the probe? the I squared C probe will be called and it's The I squared C client Struct will pass then a relevant information down to this guy, and then you'll have first per controller Data, so so for each you have to allocate a little bit of memory DevM IO IO device allocates specific to the IO and anything that starts with DevM is nice because When you won't have to do a free call and the remove function It'll just do that magically through the through the course So and then you got you know your basic Some bus read byte data. Who am I and then it figures out who it is and Attaches yeah It's going to tell you a little bit about the device It's a register internal to the to the accelerometer the and then this these are just kind of Defines to say hey what which register reads out this value and it can be slightly different based on the different types of devices But you get the idea Back to the bottom and this table here is is you can use for kind of like dynamic probing and Platform registration outside of device tree Yeah, the platform files if you guys would like to see an example of the platform files This is kind of legacy But they used to end up in Linux Ark Ark arm a lot of them in factor out especially in arm because it was quite a mess and Linus 12 all got really mad and They fixed it. That's a Oma to there's not going to be a whole lot of examples here, but There is still platform file. There was one for each different board And here I got it. It just gets a little out of hand after a while But inside of these they'll have those that platform registration, which I showed in the slides earlier So I guess that's enough for the demo. We can move on to questions now that we have like 10 minutes device Yeah, yeah, it's already it's already instantiated. So I don't know if that doesn't work for all drivers but It well it depends on the probe function. So let's go back into the driver this particular driver I think Jason had problems getting it dynamically allocate or whatever So he had to use a device tree overlay. I think The problem was they didn't have any code for handling that so it was either you had a device tree Registration or just wasn't there and the driver I suppose I could show you that What's that Overlays are kind of new but oh is it I to CD twice. Yeah distracted typing again. There we go 84 And then we go down to the probe function and Then you see this device tree of this is the open firmware device match and you see how it says if not match Unknown device model So it doesn't it doesn't really expect anything but a device tree for this particular driver every more code platform code and I have a little Session about I scored see that we'll cover a bit more of this detail tomorrow in the embedded a If you're registered you can come and see that as well So I guess questions It's pretty different This is this is a busy box image, and it's very small. It could be smaller actually, but on that freedom H is free dot Okay, I think two case There we go. We don't want to megabytes kill bites Gigabytes, okay, so we're only using 13 mags Not bad It's it's can be up there depends on what you have running the Debbie in the stock image does run a lot of web services and stuff So really this this is about as pretty bare bones. It doesn't have a whole lot of features installed So builder it is good for kind of a small footprint type of builds Anybody else sure Yeah, I Would be pretty tricky because these hex addresses on Unique in that they can be designed to multiple devices So you could have two separate completely different types of i-square C device. They'll have the same address but you could perhaps kind of Maybe grep through the driver code. It's really not easy Yeah, it's kind of tricky I guess other way around would be trying to find the schematic for the Board that you're working on and if you're doing boards before you usually have that If it's still something you can't figure out then I guess you kind of go upstream to the the hardware manufacturer And see what's going on. I'll get one back here first It seems like that but what I'm doing is I'm I'm actually pulling their advice continually Yeah Yeah, though if there's multiple devices on the bus They won't they won't talk to the to the master unless their address So that's a way of arbitrating between so if and there's really no streaming involved You have to you have to from the from the The master you have to say hey Read from this specific device. It's not going to stream across the Because otherwise it would cause all sorts of collisions on the bus Yeah, that's something specific to that a kilometer Right Yeah, we'd have to dig into the protocol a little bit there are There probably is something like that the who am I register might be Standard, but I don't know I don't go that far into the spec usually I just try to get the device at hand working The the link for Wikipedia has some pretty Detailed explanation of the protocol and it also has a from their links back to the actual spec Where you might be able to find something like that For I squared C protocol I The protocol is defined and the address of the device has to be specified before you do a read The registers on its device are really dependent on the implementation of the device itself Yeah, there's no no direct standardization I don't think but there there is kind of a Kind of a guidelines so that you can use basic The same kind of algorithm or the same kind of register mapping from one device to the next so if you have like From the same vendor you have like say from TI you have Touchscreen controller and then you have an ABC Accessing the registers the address probably would be in the same place because they're probably going to reuse that internal hardware slave core or whatever, but it's really up to the To the data sheet to determine the protocol beyond the addressing of the device itself So there it's kind of algorithm Let me try to Get a feel for what you're asking No Yeah, I think there are just I don't know them off the top of my head Yeah, it wouldn't be quite like that It would be other types of parameters. I like the the protocol would maybe determine the sequence of bytes or whatever but I Usually do you you the protocol is really just saying hey jam these two lines the adapter is pretty much saying hey Call back to this function and then that function will do what you want it to do There's not a whole bunch to it there If I was an I squirts the expert or a Linux kernel I squirts the expert I'd probably give you a better answer on that but I Just kind of get things working essentially and how that Yeah, there's there's a you mean the name of the device Yeah, there's there's a mapping called at aliases and the in the device tree file, which will determine the order If you don't specify it, they'll kind of pick the order for you I could probably show you that I'm guessing probably pretty long time But let's check it out. Okay. Here's your aliases list and this will determine the order so It's pretty straightforward on this particular device There's one to one mapping but you can change you could say if you wanted the peripheral controller that's mapped it at I squirts you one on this Come up as I squirts see zero It really depends on this alias here and that it's useful for the ordering of the serial ports as well and The USB controllers, etc. Ethernet, so that that's used to kind of determine the order and the device Essentially, it's completely possible Yeah, you if you don't register in the device tree and you enable it it's not going to show up so and the top-level DTS You'll have one and this this is the DTS. I which is kind of like for the so see and You'll notice that I'll see I squirts see You have the status equals disabled right here So as long if the if you're not mapping a device to it then This will stay disabled and they'll it won't show up in the devices if you if you if you have The device tree file for your specific device then you can probably enable it It's just a matter of getting to it another thing to worry about is the pin boxing though, so there may be a five or six controllers But they're all coming out through a limited number of pins So they have to be multiplexed out to the SOC pins So if you want to use them as I would see you got to make sure they don't contend with anything And they're actually broken out to pad somewhere and then you have to set the multiplexer so that it sets that pain for that specific function And then you can enable it and go essentially so it's kind of like there's a few hurdles But you could do something like that. Yes Yeah, so you're kind of piggybacking on to another bus. Yeah, okay Yeah, they probably some kind of maybe for firmware or something that they don't want you messing with Yeah, that that's Either happens earlier on in the boot and then they don't expose the interfaces to the user space or to You know even to the kernel depends on how it's mapped in the in the device tree Yeah, if you if you have mainline support for your device I'm not sure about that particular one, but Yeah, it's one thing I didn't bring up is pin boxing can be kind of One of the first things like if I can't get my device working First thing you do is determine if your pin boxing is correct From there you gotta determine if you have the the pull-ups on the lines and not in multiple places Not too small of values and then from there. It's just kind of Software pretty much went a little over didn't we The The original boards I have a few of those I hadn't populated some of those Yeah, the guys over at Osh Park got you those early on I Brought him to the expose show off. I brought him back and then I actually assembled hand assembled A few units like five units. I have a few that I hand assembled then Before that Jason ordered some from Handful we were gonna try to go with it, you know at group kits we had like a campaign We just kind of It was an economical Because Okay, well, and they I think they were really You It's a different So you have to use a different build, but then anything is all right and you can try to make it as good as you can. Maybe from bike drivers, as long as it is affordable it should be good. Now, I only broke out a few years ago with IEL, so it was just FBI. It is as many as you could. Yeah, there was very much space on the board. Yeah, this one, you can see the ship was a lot smaller than this one, and it makes a lot more matters worse. Sorry, I couldn't answer your question earlier. For which one? For the algorithm. Yeah, the algorithm. I never heard. I had no idea that was something that... Yeah, the algorithm is just a pointer back to the function that makes it happen. They're all doing it kind of the same way. They have a function that's passed in this depth, or in the function, or in a steady structure. It just kind of pushes around this function pointer so that it's natural for this hardware. I would have showed you that on there, but I haven't been flustered. I had never seen that that was something that could be different. Yeah, you say the 10-bit protocol, say you want to do the 10-bit, whatever, you might have a different way of doing that. It's just a matter of working out what's what's what's what. I really haven't done a ton of development on that. It's really just kind of driver stuff. Yeah. A lot of the... I really like the unified thing. Yeah, it's interesting. The new device? Yeah. Is that what you call it when you kind of echo the address in the driver into that file and load it? Yeah. Yeah, that's from user space in the fancyation. Yeah. It's pretty cool. As long as you have the table set up and the probe function set up to handle it properly, it should work fine. A lot of the older drivers will have a platform code, and then they adapted it to the drive street. So those would probably work. Because I.O.9... Well, they just do a device with that more recently, and it's not a hard-to-padded requirement to have every different kind of stampede more than those. So that was an I.O. driver and the probe didn't have it. Are there other I.O. drivers that have that in their probe? There might be something I'm missing. There's a string that's a passion that might be a little different, but I don't think that's the truth. It's one of the best effects of the driver. Yeah. I might spot it off any falsities I'd like people to tell me so that I don't do that in the future. And, you know, I still see maintainers by holding their heads, made foaming, or whatever. You guys got all the terms mixed up. What happens when you push them to do something and then you don't work on until the last minute? Yeah. I think it's good on jager and it's going more in-depth in the workshop. Yeah. I see how my hours are going to go. It's less than 500,000,000. Because it's running on because of the limit. But... Do you change what's between the revisions, right? What's that? The part? The part got changed? The accelerometer? Yeah. Okay. Because of what they had to stop? Okay. Because jagen, he was confused. He didn't know what was going on. Why isn't it working? Why isn't it working? Right. Last night, he was going to get his dimple. Test, test, test, test. All right. I'm live. Jason, president of the authority. In your aesthetic, Texas Instruments was 90 years of experience with doing event processes. And it also is driving force behind the legal form of legal laws, which are oxy-legal legal systems, as one of the founding members of the legal board, that holds foundations. And they have a big influence, though, attracting bakers and learners of all sorts to the open-source world through the legal platform. This is going to be a good opportunity to have a lot of slides. I won't go into my personal life horror stories and stuff, but I spent all my time preparing the slides for the talk this morning on the Pocket Beagle Walkthrough. If anybody was in that, but it didn't look like that made familiar faces. But next door, they're giving the embedded, apprentice Linux engineer training, where they're using Pocket Beagles and these add-on boards to teach people about developing with Linux. And I am spending my time there. This presentation was one that I was originally... I'm going to give it this next week at Embedded Linux Conference. But unfortunately, I was explained already how the slides are done by now, but I was just going to reuse it. So forgive me. I don't actually have any slides. But if you want to walk out, I'm not going to be offended. But I will try to show you some of the things about Net Console. I don't have it working in this setup, but I can show you where some of the gaps are and some of the things that allow you to get there when I have all the rest of the... Anyway, I've got a couple of nights of work ahead of me still. But if you've never heard of Net Console, it's generally a way to use a network interface to do serial type of tasks. All it is doing is sending UDP packets, pretty much raw checks in UDP packets for the data. And that could be all sorts of different. But the Linux kernel, it's the console messages for the kernel log. So everything is printing out, gets sent out across UDP ports that's pre-configured for Uboot, which I think is, in a lot of ways, more interesting for the early boot phases. The somewhat more challenging part of Net Console is that interactive portion of the boot loader. So you can actually... You know, you can specify to do TFTP, you can specify to do all those things using UDP packets. And you would just use a tool like NetCats if you've heard of it, seen NetCats. I mean, people have heard NetCats somewhere. Okay. You showed up to a Net Console talk because you're actually ready to learn about, like, see Net Console and things working. So, yeah, you can use NetCats and you can just, you know, the port 6666 and you just... You can start interacting with the Uboot console directly from Net Console to see the commands coming from Uboot. You can type the commands. The extra little bit that I'm really trying to add here is doing that all over USB. Right? So if you... You know, we've got... So this is the Pocket Beagle. So it's a little Linux computer running off of the powered off the USB cable. And that USB cable also, you know... So it's connected to the Gadget controller on the ROG controller on the processor. So I can actually... And the ROM has a boot loader. So I can... Let me just take my... My disk out, right? So that's... Normally it boots off of this microSD card. But the interesting thing is when you start booting up off of the USB. So I've got a little Node.js app that essentially implements... It uses the Node USB or it essentially uses LibUSB if you're familiar with LibUSB that gives you an API to directly initiate packet transfers. You know, so you can say I want to do an int transfer, out transfer, you know, a config, you know, sort of transfer. So you can do all the individual transfers from the API rather than writing a driver if you're doing everything in user space. So with Node USB, it's just doing that exact same stuff in JavaScript. It may not be your choice, but I'm trying to put this stuff together so we actually do it with the new Web USB stuff. So the overall objective is to be able to boot your computer off of your web browser and then even to go to the point of seeing the U-boot traffic in your web browser, right? And being able to interact with it that way directly from just the ROM on the processor. That's where we're headed. Check my password in, right? So this is that little daemon. You could kind of do the same thing all with services, right? You don't necessarily, like, because what I'm doing is instead I'm grabbing that USB device here, I could let it get enumerated by the computer and look at it as like a network device and do things that way, but instead I'm grabbing it here in that sort of web USB framework. But it's essentially a TFTP server, right? So the ROM code on this processor actually comes up and says it is a network device and then will allow you to TFTP code to it. So we TFTP SPL to it, then once SPL is loaded, SPL TFTPs, or it loads another USB driver and it TFTPs U-boots, and then from there you can interact with it via Net Console or you can boot to the kernel. SPL is the secondary program loader. So it is U-boot. It's a different compilation of U-boot. So it goes through a different set of if-dif and different linking, such that it trims it down to a much smaller version. So it's a U-boot meant to load U-boot. It doesn't have things like the command line that you could run different... It's more fixed purpose. The reason that it exists is because for some of these processors, including this one, you need to configure external memory with some initial code before you can load into that memory. So the bootloader itself doesn't... The ROM bootloader doesn't configure that external memory. So in our case, that's exactly what it is. So the SPL will fit entirely inside the internal RAM of the processor, configure the DDR that's sitting external to the processor, and then that SPL can then load U-boot that's bigger than what the internal RAM is on the processor. MLO and SPL are the same thing. MLO historically comes as a TI name. It came from a proprietary approach, which was to... So the TI developers started with U-boot and they stripped it down into an internal version and then loaded U-boot. But then it got pushed upstream and the MLO... I don't even know what MLO stands for, but the ROM code actually looks... On an SD card, if you're booting from FAT, it'll look for a file called MLO. So it looks for a file with that particular name, but the file is SPL. And did I say that SPL is a secondary program loader? I did, but I don't know what MLO stands for at all. Yeah. Right off the... Well, and it's just meant to load... It's just meant to load the... Well, the primary one here is U-boot. So the re-of-the-ROM is first. That's Treasury Area. Yeah, okay. Yeah, I couldn't... We need Tom Rini around to answer that one. Oh, is it? Oh, but that's the hands-on U-boot. The... So I'll just show you this. So that's doing the boot pee and, you know, it'll just... It'll be sitting there in U-boot right now. So let me... It's not very interesting. U-boot is trying to boot. And so, like some of the default configurations it has is to try to TFDP this PXE config file. That's just part of what Mainline U-boot does. I don't know why it's trying to do that. My little program doesn't have any idea how to handle that. So normally, I would stop it before handling it. I'm going to go ahead and bring up the serial port, which is the whole thing that I'm trying to get rid of. But the way you would normally interact with U-boot is you would have a serial connection to be able to work with. And for people doing developments kind of new, I don't want them to have to go and buy a whole bunch of other hardware and hook up a bunch of... Figure out how to configure things just to be able to read boot logs. They should almost never have to do. But in the case they do, I wanted to work all over the USB pseudo screen. You have to put the bot right after it. So it happens to be 115.2. I hope that's the right location where this guy enumerates. I should have looked at my message log first. So if I... I mean, I'm going to just pull the power here. It's about to start rebooting again. My little daemon is going to sit there running in the background now. But... Oh, it kept the power up at the serial port. Hit the reset button. And again, so the ROM grabbed the SPL. Okay, and that's how I stopped it. So at this point we're inside of the U-boot command line. So what's run here is... So if we go back to the very beginning of the boot on the serial output, we see this very first line. That U-boot SPL line. So that has... So SPL will print something out that says I'm going to try to boot from USB Ethernet. It tried to look for environments on the FAT. Of course I have no SD card attached, so it's not going to find any environment variables saved off in the FAT. And then it ends up getting the U-boot load question. And the boot ROM itself on this particular device includes that it looks like an R-indist USB R-indist device. And it tries to do an ARP and a boot P. So you'll get a boot P request and an ARP request. And then it'll start doing a TFTP transfer. So that first part was satisfying those requests by the ROM code. And then the second part was satisfying those similar requests from SPL. You couldn't then go from here, if we wanted to TFTP the kernel, we could use the same daemon to TFTP the kernel or TFTP a root file system. So I mean that part is pretty well handled. What's not handled is the debug part, which is why the net console part is really important. So as long as all that works, you're great. But if you're trying to develop a new kernel, you're trying to develop a new, you know, making changes to U-boot without having a serial port, you're not getting any real feedback. You're just not getting the packet request and you have no idea why. Or you're not getting your kernel to boot up. And so when you want to do, when you want to do net console, essentially it's pretty straightforward. You have to compile a few variables in the configuration that you say enable net console. But from a practical standpoint in U-boot, you also want to probably have the thing that allows you to muck the output. So it's possible to have multiple consoles in U-boot. And maybe you'll eventually have to switch over to all net console, but typically it's serial. So if you print, like you'll see these, I don't know how many people, how many of you have played with U-boot's command line very much, right? For the most part, people would say, okay, well U-boot needs, you know, it loads my kernel, I'm happy, everything's, you know, good. Well, I'm good. But it's actually a very, very flexible programming system that it's got in here. So you can print, you can help, get a list of all the different commands. There's a lot of different commands. You can find out a different version information. So you can query quite a bit. If I wanted to be able to turn this into a, like a USB dongle, I actually run UMS right from here, which means USB mass storage. So from this, I wouldn't even have to have an SD card writer that's separate. I could actually just, straight from U-boot, I could run UMS, have UMS compiled into this version. Yeah, so if you run the USB mass storage thing, you can actually just use it to serve up the card and write it directly from the card. That's another aspect of what we're trying to do here, is so we can use the web USB and it'll automatically download the images and program it. Is there a question or comment? So the device tree, this is super, super useful stuff if your system's not booting because you tried to load the wrong driver or other things. So U-boot has the ability to patch the device tree. So the device tree is the thing that describes all the hardware to the kernel. And so you, for each of the different target boards, you'll have a device tree that gets loaded at the same time as the kernel. The boot loader will load that. For Beagle, we also have an overlaid scheme where we'll look at a bunch of different variables that you set up in a text file and use that to load different overlays. And U-boot will actually apply those overlays for us. So in the kernel boot, it's all just set up and running. There's also ways to dynamically do it in the kernel, but for things like, oh, I'm adding on an LCD. I would like for the LCD to actually show the boot messages or something like that. You know, I want it to have it very early on. So you can do that in U-boot before the kernel ever loads. So you can modify the status structure that describes the hardware to the kernel. All things that would be really nice to do without having to go get a serial adapter and a number of things that I found that I can't do in the kernel that I can do in U-boot. One is, so even if you have like a fleshed out device tree, I don't think there's a way to easily, I mean, I see remove nodes, but honestly, I really, really struggle in any sort of order to really remove nodes practically, but it's pretty easy to go and add a status equal disabled to an individual node so that when the kernel loads it doesn't load that module. There are a lot of, there's a number of hack modules essentially in the world out there that you might want to load that don't unload themselves properly. So anytime if you try to do it dynamically in the kernel, it needs to have the remove module support done properly. Yeah, and that's, I mean, the config at FES Pro is how I'm doing it now on Pocket Beagle. So as far as like live patches, but we just generally discourage people from doing live patches instead of doing the patches as U-boot. But there's a few places where I'm showing people how to do the live patches with config FES. But the Cate Manager, the Cate Manager is not part of Pocket Beagle. It's pretty quiet, I know the rest of the room, I'm sure the rest of the room can't hear you. The advantages of config FES over what? Well, for the most part people using it really aren't doing editing, they're just wanting to use it, and so that way they can have it at boot up that it's all just running. You know, it's persistent, right, when you edit the config for U-boot. I'd say it is a challenge to try to do certain things like we now have in our default device tree for Pocket Beagle. We load a bunch of things called pin month helpers, and as well as we actually export a bunch of the GPIOs to CISFS, so we allocate those GPIOs to CISFS, so they're ready for users to do user space-y stuff to the GPIOs. But just trying to disable those, like the helpers for that is all like one big thing, it doesn't unload cleanly from the config FES, but if we patch it in U-boot it never loads. So we just set status equal to disabled as part of our overlays, and then we export whatever GPIO pins we do want to export, so there's just a few cases where you really needed it to be in U-boot, and choosing one or the other, like the LCDs, like that was one of the biggest ones was the LCDs. Well, but we have config FES, it's already in the kernel, when you talk about the config FES. The CAPE manager, one of the things that we do in U-boot now is we actually scan the eProm, so the bone CAPE manager, so if you've heard of CAPE manager, it goes and looks at eProm values, looks for the device descriptor string in the eProms of add-on boards for Beagle board, for Beagle bones, and if it sees that it loads the device tree overlays that is associated with that board automatically, so it just dynamically configures and goes at boot. We were doing that in CAPE manager at the kernel level, but now we're doing it at U-boot because CAPE manager, they didn't want to accept upstream. They've got dynamic overlays, but because they think a lot of it has to do with the terminology of calling the CAPE the bus, and say, well, that's not a bus, right now it works. By default, it's going to run this, U-boot's going to run boot commands, which goes through this list of different boot sources and tries to look for some valid image. We want to spend a whole lot of time going through U-boots, but I did want it like so. The Enverma variables standard in, standard out, they all say serial. If I put serial comma in C in those variables, that starts in that console. When I'm doing that, I've always created a variable that sets all three of them at once. I don't know if it really matters or not, but if there's a right order, you can do it in, but I've always set all three of them at once. The set environment, I'm going to standard in to serial, comma in C, standard out, and C. Now it's going to complain because I haven't set the server IP or my IP address. This is where I need to kind of think about a few things, but if I do run, start, it brought the network up, but it says there's no IP address set, so I need to be able to set IP address before that. Does that quite make sense to me? So we need to try to do something to fetch an address before that, and I don't know. There's a couple of steps. I've had this running before like six months ago. I had this running, but I was actually using this to do board testing. I was actually booting the boards over USB and using the net console to issue commands. So I'm going to pull that code up and try to understand everything that I did to get that again. But I abandoned it at the time because it was too slow compared to booting off the microSD card. So it took a couple seconds to load up everything over USB, whereas I could be done well under a second with microSD cards, so I ended up doing all the board tests. For the pocket-beagle testing, I was all done booting over microSD and then using USB to send the serial number across. So we still make a USB connection and test the USB connection when we build the pocket-beagles, but we boot them off of the microSD, which tests the microSD card, and then we send the serial numbers to U-boot but over the USB cable. We don't boot it over USB. So the serial number is going... I have to try to look at the code. The tester has the serial number and it's sending it to the board. The test code is still on a gist. I don't think it ever made its own repo. So the test code... This is the DTS file that I was using. A lot of this stuff in this repo is also about bring up. So this is the tester code, so it says JavaScript code. So it's actually using NetConsul to send... Oh, yeah, for sure. So we just increase the font. So this JavaScript file is using NetConsul over U-boot, so U-boot boots up over the microSD, but then it exports the NetConsul. And so I actually... I just listened for the UDP ports here in... I saved off my magic foo. I probably should have just copy and pasted it, right? But this is the magic foo for starting a NetConsul where I've got the... I'm setting a server IP and a personal IP address. And I'm setting all the other items. But the commands that I'm issuing... This is what it actually issues during the test. So it turns on an LED. This is the write protect signal. The GPO is the write protect signal for the eProm. So it's tied to a line. Then I've got this script that sets up the eProm with a base header that's common across all the pocket vehicles. And then I've got this serial text. It's actually the serial number for the board. And then I'm doing... What's one of these memories? Oh, I'm putting it into memory chunks at a time. And then I'm going to do a write of the whole memory dump. Right? So here I'm doing the eProm write... I've got the reading the data that I've put into this location in memory on the... What is that? Is that the link? I have to look at the eProm write parameters exactly. That's the... That's the iSports theater for the eProm address that's reading out of memory. It started off at zero. And I think 1C is the link. And then it sets another GPIO if the write succeeded. I read it back. I set another GPIO to say that the write succeeded. I do a memory compare to actually see that the eProm value that I wrote is the eProm value that I read back. And then if that works, I set another GPIO. And we've got all four LEDs on board good shipping. So there's the testing here. You've got the USB for all the communications. You've got the micro SD card for doing that. Plus you've got the eProm writing. So it ends up doing a number of tests on the hardware in order to get here. But this is what's being used to test all the pocket vehicles before they ship out. It's really not very well documented at all. There's a reasonable amount of stuff for doing that console from the kernel. But when you start doing that console in UBoot, it's largely undocumented. And my hope is to kind of bootstrap some of that and to kind of give a really good example. Unfortunately, I wasn't prepared today. You can definitely miss some of the early stuff. You need to set a reasonable boot delay to allow time for the interface to come up before you start doing interactions. Yes. I will recommend that people doing this actually do either both if they still want to be able to access the serial or as a default. There's a way to test if there's a network connection or not essentially by doing the ping. And if there's no network, if the if-net console still pre-doing the boot, we do a ping of the server IP, and if we can't ping it, then we don't start net console. I think for your individual target, it may not. That's what we're trying to get to. But where the usage is right now is everybody depends on serial until there's a lot of good examples, a lot of people coming to talk here and really prepared showing people how they can do everything with net console. Until that happens, and lots of people are using it on a regular basis, I don't think the documentation and the examples are going to be clear enough for people to follow it enough that people migrate away from using serial as their primary debug mechanism. And that's where I want to try to get to. Let's not pay for the USB to serial adapter in order to have people be able to bring up Linux. And especially with something like what you've got with Pocket Beagle, you've got that SIP on there that's pretty easy to lay out and make a board for. I think you can actually probably do it with a two-layer board, which currently are a four-layer board. Michael is in the room, we'll have that debate. All you need is power and a crystal, and you still need to do some strapping of the boot config lines. You put resistors on there to kind of tell it what boot mode it should try, or what boot mode, because it tries four boot modes, depending on what the boot config is. So you need that much hardware, but really nothing else. You get power and you can get the USB connection just from the USB cable. And so if you're trying to build your own board, you don't have a lot of other system dependencies. I haven't gone out, it doesn't matter what's on the other pin. I can boot up and run Linux, and then from within Linux, I can go and try to turn on other things and turn on a subsystem at a time and test it, see how it works on the board, and go about that approach to bring up a new system, which is pretty, pretty different from the way people do it today. And although things have evolved a lot, and when I started doing these, and everybody would have this, you start with a JTAG connection. First you get JTAG, and then from JTAG you start loading, okay, I'm going to do some memory tests to make sure my memory is all right. Okay, memory is good. Now I'm going to go to the serial port. I'm going to get my serial port connection, and I'm going to get that data going. Individual test code items that are personally hard to maintain and move forward on the different platforms. That's the way you traditionally would bring up a board. It's evolved somewhat now, because there's a lot of open source support. There's still the initial processor bring up, but if somebody's already brought up the processor, you can probably get to that serial port and do some other stuff there. But I think this takes it to a whole other level of simplicity. You can essentially do your board bring up with just the USB connection, providing your power, providing your communications, and everything else. Then you can use the Linux to actually running within a system, booting into RAM, to start going out and doing anything else, sitting outside of that. But also you want, so I try to look at it both ways, right? What is the professional need and what does somebody need to do as a kid or a hobbyist or something, just trying to get into it to understand what's going on in bootloaders and get that visibility. If they can get it in their web browser, they didn't have to install any other tools and it was less dependent on what that system is. That's to me sort of the magic. There are some tools to give the USB, that means to get to your browser to serial, but they still depend on external hardware and connections that aren't just trivial. This is the one connection to the board. Anyway, that's what I'm trying to get to and we've got the Node.js project. I think the big thing we have to kind of make a change is get people using Net Console because it doesn't get used enough to get really debugged and fleshed out. Maybe I'm wrong, but I haven't heard anybody here actually using Net Console on a regular basis. Anybody use it on a regular basis? Have you used it at the kernel or at the bugoo? Both. Oh, wow. I need to... Why haven't you done this presentation? If you had some documentation, I've been like something right up on that. That would be extremely interesting. The gentleman with the necessary. He looks familiar, but I don't... So what do you use Net Console for? You can't go wire up zero cables to every single board. But you've already run the Ethernet, so... Did you boot? Were you using you boot? But you didn't do Net Console. I mean, doing it in the kernel is not that uncommon. Doing it in you boot, it's the support there, but in terms of its function. But you can tell, I mean, I've done this a few times and I still struggle with getting it set up right. For the kernel, it's just because at the kernel command line, you provide all the parameters for... Where do you want the IP address of where you want it to be sent, and it goes. So it's just kernel command line. Chris, are you telling me you're asking me? You're probably... You still have to get it up in the kernel, but there's a bunch of how-tos that are pretty good on just setting up the kernel command line. Especially if you can get into EFI, where you can actually run, you know, you can run some nice scripts from the bootloader to get things. If you can get into EFI, then I think you can do the configuration of the kernel command line stuff from there. I think it makes it real easy. Well, this doesn't have either, right? So that's why we're doing it over USB, which makes it a little bit trickier because you've got to get the drivers to come up first, and then you've got to try to get the services. Like, if you're sitting on a static network, all you have to do is you have to have somebody running and listening for the R request, you know, and the boot P request for when the device shows up on the network to get it as IP address to get it as base code. It's a little bit different when you're doing it over USB. Yeah. I don't think it's that much more complicated. It's just that there's a couple twists to it. I mean, so in this case, we have a ROM bootloader that notes how to boot over USB. You're going to still need something that knows how to talk to the Bluetooth stack. If you really wanted to add something like that, there are some Bluetooth serial dongles that I think could be an interesting solution, but you're still depending on firmware within that dongle to support the serial connection and doing it that way. Well, at least it's not, you're not depending on software in your target. You're just depending on software in some external little module that you can control and replicate. But, yeah, I don't know anything that's kind of got a solved just boot interface on Bluetooth. Maybe some six low-pan experts will speak up. Thanks for coming by. I apologize for the lack of slides. If you have any other questions, feel free to come up. If you didn't want to shout them out, feel free to come up to me. If you have any pocket-beagle questions, I can certainly entertain those. Oh, one more. Yeah, free LC. I can absolutely upload it on the scale slides as well. So I'll make my slides over the next couple of days. And, yeah, so give it until after you'll see and then look and the link should be live. My apologies. Yes, again, feel free to come up to me afterwards and I appreciate your understanding. Thanks. Inside the pocket. It's not going inside the pocket. I'll try and help him back here. No, please don't. What? Put it inside the pocket. Why? Because everybody I know who tells me don't put it on the inside the pocket. Here's what they do. And the next thing I know is the problem. What do you mean it's a problem? It's not a problem. You guys can all hear me, right? See, they claim that... Does this work? No. No, it's not my headset. I don't own nice things like this. What happens if I do this? It sort of works, but I don't get those. And this is why everybody hates doing presentations from one expert. Well, some of us do. I still have like 15 minutes. I'm just going to like be ridiculous and silly and tell them. So, if you're welcome to leave if I'm being too silly. Yeah, Inkscape is excellent. That is true. I did not do my slides in that because... No, I'm not putting anything nice up there. Why would I do that? Let's see if I can get... They're the ones who have to look at it. I know it's on my slides. Oh, I mashed something wrong. I screwed it up. I know it looks fine, but it's obnoxious. I was hoping to get Presenter View to work. Presenter View? Oh, maybe if I do this. Can I fullscreen that? Just click to present, click present. And then I hit... What is it? F12? I think I figured it out. I was going to say all of you guys are going to be able to hear me no matter what. It's whether they're going to be able to record this or not. Blah, blah, blah, blah. Blah, blah, blah. Blah, blah, blah. It sounds like I'm more coming out of the speakers. Oh, wow. That actually works a lot better. I was wrong, Tom. The other mic was not as good. No, no. I was going to say the sound is fine for the most part. I don't think I'm too loud on your recording probably. Blah, blah, blah. Blah, blah, blah, blah. IoT is broken. It's all broken. We're all going to die. I mean, life is a delightfully 100% fatal sexually transmitted disease, whereas my wife likes to put it since we just had a small one. Her maternity leave was technically an STD. For those of you who are not American, an STD being short-term disability. Not the other way that should go. My wife's Canadian, so she found this hilarious. Oh, that's going to be really obnoxious. Yeah, but I never stay put. If anybody has ever been to one of my talks, they all know that I never stand still, and this is why that little camera is going to hate me. Oh, it's too far. Oh, you knew I was going to talk in here. I don't stay. Yeah, pretty much. I mean, Pycon's in Cleveland, and I'm totally not sneaking into Pycon again this year. And hint, I'm totally sneaking into Pycon this year, because that's what I do every year. Because I'm a pro developer, a model Python developer, and they think I'm funny. So I have seven minutes to tell jokes before you all start laughing up there. You try to laugh at me, or that would make this topic really, really boring and depressing. And this topic's already really, really depressing. I can either confirm or deny that there was a Warthog 1-8, and the bodies are totally not in places that you will never, ever, ever find them, because they're next to Jimmy Hoffa. So the entire reason that the Raleigh term photo was ticked was when I did a Google image search with a label for reuse, so that hypothetically should my employer decide that they actually want me to talk about this subject internally, that I could still use these slides. That was the first one that came up that was pretty and had an appropriate amount of barbed wire on it. The rest of them got much more depressing, as it looked like they were military members actually stringing the barbed wire in much less friendly territory. And I didn't think that was quite what I wanted to go for. And my laptop decided it wanted to go to sleep. That's going to be interesting. Let's see if we can fix. Members were known that setting is fully charged in 47 minutes. Power saving. Blank screen after never. That didn't screw up anything. Empirical evidence in greater than 15 minutes, since the 15 minutes was the longest setting that no one would give me. So if it doesn't blank after 16 minutes, I'm going to empirically suggest that that is as close to never as I care about, because never doesn't exist, only the heat depth of the inverse. Doop, doop, doop, doop. Which of this is Jason Kripner's water? Just find it fascinating how awesome their recording gear is. Whenever bored, they built their own hardware to do video recording for everything. So come sneak a peek behind the podium if you're ever interested. Pretty solidly full room for three minutes out still. I budgeted way too much time to make Linux work on the screen again. When I first started doing presentations and presenting on Linux, this is early 2000s, trying to get the second monitor to work at all with a bloody miracle. We all knew magic incantations for X-Rander. How many people actually have heard of X-Rander? I'm sorry to all of you, you're all much too old. To the rest of you, feel sorry for the people who just raised their hands, because the fact that we even know about X-Rander is, that's right, I have fabulous prizes for this talk. And I remember the last time I actually did, I think, that's right, I have fabulous prizes that somebody may or may not want to win, which will become apparent as I actually give my talk on why I'm giving this away. So somebody will either leave very, very happy or very, very sad, and I'm not sure which. It all depends on potentially how much you trust me, because I'm a very trustworthy human being, right? My wife has a PhD in computer web security, so yeah, I'm probably not even remotely the most trustworthy person in this room. She is much more educated than I am. That is true. She argues that her PhD is nothing more than her being far more stubborn than me, which is kind of impressive. Because you don't get a higher education degree because you're smart, you get it because you're so bloody stubborn that you actually make it through. As the people with higher education degrees all agree, a teacher at one point, since I have 20 seconds, trying to convince her to go for my masters. And I said, you buy me hardware to break voting machines and I will come back and I will do my masters and possibly my PhD. He never bought me hardware. You'll do my intro, you are welcome to do my intro. And that is a lovely machine. I have one in my backpack. Anyone, does there a bunch of it? One of the earlier open source contributions that probably everyone of you has heard of and he's writing about it by the end of the year, he was leading the admin. But it turns out that we as a man, when things happen, we have a lot of hardware and embedded skills as well. I will set the record straight. The drones never caught on fire, but they did crash because that's what drones do. And it was the robotic dog that caught on fire at scale three years ago. I made a replica of canine from Dr. Hill. Both sides. Anyway, but that is not what this talk is about entirely. I'm here to talk about IoT. I'm here to talk about IoT. And how good fences make good neighbors in this magical land of IoT. I am required to start out by pointing out that one of these people in the picture is a snowman and one of them is me. And either way, this entire talk is, my opinion is my commentary, all of my blathering and nonsense, not any of the views of my employer. Who happens to be VMware, who was very kind enough to send me to this conference to talk to you guys about all of this. But, let's start by taking a quick look at what the IoT world looks like today. I gave a talk last year where I made some interesting commentary about where is this IoT about faking us. If you came to that talk, a lot of these pictures look familiar. If you didn't, this is a random talk that I'm going to talk about today. These pictures look familiar. If you didn't, this is a random smattering of all of the IoT devices that I have, not necessarily in my house, that I have at least looked at from a perspective of how they work. And they're such delightful devices up there like your oven. It connects to the Internet. That's not a bad idea at all. Your coffee maker, if you have one of these, make sure it is connected to the Internet and turn it off so you don't wake up in the morning. Aces makers, that's a totally good idea. Croc pots, again, a totally good idea. Door locks. Door locks have actually had a very interesting change in them last year because a very large online retailer is now asking you to give them access to your electronic door. It's not amazing. How could this possibly go wrong? I know. I actually get really excited about all this stuff. This is so ridiculous. And to be fair, the very large online retailer, their motivations are actually well played because a lot of criminals are figured out, wait a minute, if I just wait for that weird white van that's not marked in any way, with the person who has no indication that they're a delivery person, goes up to the door and I wait 10 minutes after they leave. I can go steal all the things. So criminals just figured this out. And so what they're trying to do is actually put your packages inside the door instead of just leaving them there for the criminals. Great. Except now that you're giving random strangers access to your house. Good job. It's not entirely ill thought out, but it does make some very interesting commentary in an internet connected world. I'm going to come back to that one. Your refrigerator, that's no more dangerous than your oven. Thermostat, again, really no more dangerous than your oven. Your TVs. Oh, this is always a really nice one. How many people have a TV? Okay, keep your hands up. How many people have a smart TV? How many of you believe that it's not spying on you? I have some news for you guys. And how many people have one of these magical smart speakers? How many of you have had it cackle at you in the last couple of days? Because apparently they're all cackling now. That's not super creepy at all. Dark out, it's raining. I'm from Portland, so it rains all the time, so this is really easy. I also have coyotes that come in my backyard, so you hear a coyote wail. And the Alexa device just starts cackling. Not creepy at all. Let's see, what else we got here? Yes, oh, and cars. Internet connected cars. Some friends of mine presented at DEF CON this last year and pointed out that several luxury branded cars down to and including some Nissan Leafs, the cars that they were looking at, are actually vulnerable to the haze modem attack that affected the iPhone 6, or iPhone many, many years ago, like four or five years ago. These cars were still susceptible to this attack. It's great. It means you can take over the entire car. You can turn the windshield wiper into the brakes and the accelerator into the horn and all kinds of stuff. And you know how this got fixed? The carrier that was providing the GSM connectivity to these cars decided to block the port, the incoming port that would have connected to these cars. Would you like to know what port that was? 6667. And for those of you who are on IRC, you'll notice that that's the IRC port. There's no more running IRC on your car. I'm sorry, guys. Or at least you have to do it on port 7000 now and not 6667. We skew way, way, way off into the weeds and then we look at things like the Bluetooth-enabled pregnancy tester. And you wouldn't think that a pregnancy tester, the thing that you're literally going to pee on once and then throw away, would make sense as an IoT device. But it does, because it just so happens that this is a marketing ploy. Absolute marketing ploy. Because you know what the most profitable demographic is for marketers? People who are having a kid. Disclaimer, I just had one. So the marketing material is insane. If you have not had one, be prepared to just junk mail constantly. It gets worse apparently. So yes, you install their app and once it determines that you're pregnant, you just start getting coupons and spam constantly. For the record, my wife and I did not use that. We found out because we were going to India to give a pair of keynotes. And they found out when we were getting vaccinated. And then the last device on here, which I'm going to talk about quite a bit, are these things. These wonderful little devices that you plug into the socket, you plug it in your device, and it's got a giant relay, and it goes click, clack, and turns your light on and off. Or your oven, your coffee maker, your crock pot, and anything else you can look at here. So these are a random spattering of all these IoT devices. And these are all coming to market as quickly as possible, as fast as they possibly can. And they're great. They're cheap. Everybody's putting them everywhere. How many people do not have an IoT device in this room, in their house, running right now? I'm actually really impressed. I'm almost shocked and you're probably lying, but I'll trust you. So here are some really neat facts about all these devices that we're cramming into our houses. These are news articles that I have pulled from within the last year. I decided that anything older than that, like the original, or I bought that and everything, was just a little too old to care about. The most recent of these was reported yesterday that shows that most IoT devices can be hacked into bot nets. This is great. Look at all that neat, cheap computational power that you're all buying for all the people who want bot nets. The Bitcoin mining is going to get even slower, because trying to compute Bitcoin on this doesn't work very well, and it's not exactly profitable. But there's everything from, in 2017, distributed denial of service attacks increased 91%, mainly due to the rise of bot nets on IoT devices, to IoT bot nets bypassing firewalls to get at the actual firewalls themselves, so that once they've infected your network, they stay infected forever until you take that router, take it up, I put it under some thermite, and just nuke it into the ground and start over. And then hopefully you don't buy that brand of router again. IPv6, the problem is only going to get more complicated and harder. If for no other reason, then IPv6 flattened the entire IP space across the Internet. Right now we have these really broken things called NAT, NAT and devices. Don't get me started on how terrible they are. That's an entire talk in and of itself. But once you flatten the entire topology of the Internet, where I can now talk to this light switch directly, what's going to protect this light switch? This gets complicated very quickly. And one of the most interesting things, and one of the early things, was again, a piece that came out of the Mariah Botnet, which was that IP cameras participated in one of the largest distributed denial-of-service attacks at the time in recorded history. GitHub last week eclipses that rather soundly, so to those of you who perpetrated that particular distributed denial-of-service attack in graphs, may you rot in the eternal fires of whatever circles of bad places that you would like to go to. But yeah, this is only going to get worse. And in the upper left-hand, or I guess in the upper right-hand corner for you guys, there's this ever-present now Spectre and Meltdown that is now hanging over every device, whether it's your plug or your laptop all the way up through the cloud servers, where we are now actively finding hardware bugs that are fundamentally impossible to patch in some useful way. And this should scare the crap out of all of us, because whereas there are large companies who will try and ship fixes for my laptop, nobody's going to give a flying rat patootie to try and fix the security bug when it's in my light switch. So there's some other interesting things going on here. So there's all these vendors out there. There's all these devices. And from a consumer perspective, I really want to control this device from anywhere. I don't know why you want to control this from anywhere. It doesn't seem to make a lot of sense to me personally. But I am told that consumers really want to control the light switches from anywhere. So, okay. You as a product development company figure out, well, there are these stupid things called firewalls, and they prevent me from talking to everything. Okay, that's obnoxious. How do I fix that? I'll reverse proxy the Internet connection, which means that this will connect out to some cloud-like device and create a tunnel that I can then connect to from my phone to turn it on and off. It's great, except when you start thinking about security. Or the fact that these companies may not actually be telling you the truth about things. So on the left there is a rather unfortunately heavily edited photo of a display from one of the IP cameras in my house. I believe I have 9 or 10 cameras around my house right now, mostly because I have two and a half acres of land and trying to keep an eye on that big of a property as a bit of an issue. But one of the things that I specifically don't want these cameras to do is talk to the Internet because the Internet is full of scary, scary things. And so there's this magical line that says Platform Access, and there's this magical thing that says Enable. You'll note that it says it is disabled. This thing should not try to talk to the Internet at all. And as soon as I move my big head, you'll notice the dump from IP tables that specifically says on my IP camera network trying to access the Internet, that device was attempting to talk to this IP address on port 8555, or 8555. IP address just happens to go to Amazon AWS. For some strange reason, I was not particularly impressed with that device. What's worse is that every IP camera I own, regardless of manufacturer, does this. And some of them don't even give you the nice fiction of being able to turn off that. Some of them don't even give you it. This is great. Now, put all these neat little devices in our houses. They're lying to us a lot. If anybody's familiar with their update strategy, it's great because they never update them. And there's all kinds of neat other just problems. I mean, how many people have something wrong on their laptop right now? Okay, you know, a matter of a half. Think about that when you have like, instead of like 10 devices on your network, and you have 100 devices on your network, how many of those devices probably have something wrong with them? That you've mis-configured it. You just haven't figured out how Lua works on that particular platform. The vendor stopped supporting it so it only sort of works now. So, okay. There's some neat things we can do to fix this problem. Or at least maybe not us exactly, but there are some things to fix this. Companies can build better devices and provide long-term updates. That's a pretty solid thing. If you have paid good money for this thing, it should be updated every so often. You know, maybe once a year. It's a switch. It's got a relay. You know, maybe update it once a year. You know, I'm not asking for much here. Devices could be more open, you know, hardware and software. In a previous life I worked for Intel as their open source hardware evangelist. And so I have actually a pretty strong bent on trying to convince everybody that they should be using open source software, which I've been trying to help facilitate for the last 20-some years. And hardware because, you know, well, frankly, hardware's gotten cheap enough. We can all build hardware. There's probably a few people in this room who have literally laid out their own PCB and gotten stuff back. Okay, some point. But, you know, be more open so that we as a greater community can at least come in and maybe help. You know, maybe we won't run your firmware, but how many people have a rooted phone in this room? How many of you are running non-stop for your particular hardware manufacturers? Oh, that's on your phone, tablet, whatever. Okay, maybe a quarter of the room. Smaller than I expected, I will admit. But, you know, clearly in the phone space we've all proven that we can, you know, build some maybe not quite, maybe not necessarily always quite as good as the manufacturer, but at least an interesting, you know, remix for your phones. So there's that. Devices can be less reliant on the cloud to spread out the attack surface. That makes sense. There's a lot of good, nice things. And devices can be better, possibly even automatic updates and alerting users about updates. I like updates. Well, there's just updates now. You know, most Linux boxes, if you're slightly brave, will auto-update. Unfortunately, while I'm wishing for, you know, somewhat impossible things out of the universe, I'd also like a unicorn preferably in green and purple paisley. Because none of this is going to happen. Let's be honest. So when I said the first one, you all started laughing. And you've mostly been, you know, chuckling under your breath on each of these that I've said. And that's depressing. Because we now have a ridiculous amount of cheap computational power that we're literally, like, connecting to everything that could, like, turn on and off our refrigerators and our TVs and listen to us and cattle at us and record our entire movement as the IP camera in the back of the room is doing. This is genuinely kind of scary. And it gets slightly worse when you take a look at what a normal home router looks like. And unfortunately, this is even more complicated than what most people have. Because most people don't have the wireless guest network. You know, this is a more recent thing. This is advanced. That we now have a wireless guest network that can't normally talk to the main network. It can only talk to the internet. And this is, you know, maybe a little bit of an oversimplification but trying to describe networks is really hard. So let me give you an example of what this mostly implies. That we think the internet looks like this wonderful high-key party. You know, everybody's obeying the rules or all being polite. You know, why, yes, I would like to talk to that port, sir. Would you like to call it? Why, yes, you know, the key is lovely. Can I have a, you know, a thing of sugar? This is what most everybody right now and maybe not necessarily in this room. Because, you know, if you're here, you know, you either want to hear me, you know, song and dance for an hour, which is possible. Or you're genuinely kind of scared about how the state of IoT is. You know, okay, then you're kind of in the right place. But most people who are consumers of straight-up electronic devices think that this is the end-all be-all for how the internet should look. At least from their home perspective. Except that there's, you know, things, they come, they bump into the firewall and they go away. It's great. That works most of the time. Except that's not the way these things work anymore. You'll remember when I pointed out that the IP camera was continually trying to talk to the internet. Oh, that's a problem. Because, oh, yes, okay, I changed that point. Because this is the way the internet actually looks. It's a mass. And it's a rodeo, and it's the Wild West, and everybody's shooting everybody. And it's really quite deadly. If for no other reason that this device now connects to a random cloud provider of some sort, it then pulls down whatever it's going to pull down. I have no idea how this is authenticating anything. And I have no idea what's running in the cloud. If for no other reason that, you know, let's say that flybynight.com built this device. It did not. I have no idea what flybynight.com is. And they built this, and then they go out of business tomorrow. Well, somebody comes in and buys up their domain name. This is connecting to a domain name, not necessarily that Amazon AWS IP address. Because, well, IP addresses in Amazon AWS are relatively ephemeral. Great. So now, because I've just bought up the domain name for flybynight.com, I can do anything I want to this device. That's great, right? What happens when I don't even have to buy flybynight.com? What if I just poison your DNS to the point where it connects to my copy of flybynight.com instead of the actual flybynight.com? ISPs don't, you know, muck with your DNS queries at all, do they? That's never been done before. And for those of you who don't get the joke, they do this all the time. Usually to make money on ads. Neat, huh? Great that we killed that neutrality. Yeah, that's kind of that. So yeah, so then we infect this magical device that's inside my network with the Ebola virus, which is arguably the most deadly virus known to the human body. And that's great, because now we've just infected our entire network and everything's doomed and now we're back to the every IoT device is hackable and part of the botnet now. So great, so all of you who have IoT devices in your house, congrats, you're probably participating in some botnet somewhere. You know, say hi to the nice people who are controlling it. So yeah, this is to say the least. And this is where kind of the crux of this entire talk comes from, because a number of people have known over the years that my network has gotten more and more ridiculously complicated. And a lot of my friends would argue that I'm just an overzealous systems architect who likes things complicated. They're probably not wrong. But it also means that I've done a lot of things to my network that are weird. This is what my home network looks like right now. You'll notice I have a lot more VLANs than your average consumer. I'm also buying much nicer network here, unfortunately as a result, but my network's expensive. But this also means that I can do a lot of things that we frankly should be doing by default already. This IoT device, there is absolutely no good reason other than the belief that consumers want to control this while they're in Australia, when the device is in America, that they want to control this. There's so good reason! Seriously! It's just a bad idea! If your oven's on and you're in Australia, call someone, they'll go over to your house and they'll turn it off, because most people are kind of nice like that, and I'd rather tell people where I did the Mickey to my house that I know and trust than putting this on the Internet that John Q. Stranger, who just happens to find it on the Internet, can turn it off for me. Or on when I'm not there and they figured out that I'm not there and they put my house down. Of course, that's never happened before. My IP cameras, and I'm going to kind of blather through this a little bit, but there's some really neat things that we can do here. And yeah, that this is beyond overkill. So my main network has access into all of these things. This is a somewhat dangerous policy, but if you're on my main network, I kind of trust that you're pretty much me, my wife, or somebody who actually literally runs the Internet, because I think those are the three people who have access into my main network who are not me, or who generally have access into my main network. Which means that, you know, from my main network, I can still get it in my IOT devices, and my IP cameras, and my multimedia stuff, and my wireless, and... Yeah, so I can get all these things, and it's kind of nice. Because I'm me, and I like having access to these devices, but when you're this magical little device that turns things on and off, I don't want you to have access to anything, including the Internet. I don't care if you get updates from the Internet. If for no other reason, then, if this can't talk to the Internet, and it can't talk to anything else, it's probably going to stay in roughly the same state that I just started in. And if every device that can reasonably contact stays in that same state, you know, yes, it may not have the latest and greatest firmware, it may not be able to, you know, it may not be connected to Alexa anymore, but this device can no longer do any damage, or the damage it can do is very limited. It can't get at my laptop, which is good, you know, because things like light switches. These are random devices now that we buy, we connect to the Internet, and that can turn lights on and off, they can calculate us, they can do all kinds of things. So, yeah, give me one good reason. Can anybody in this room give me one good reason why that IoT light switch should be connected directly to the Internet? Okay, that may be the only reason, sir, and I would argue that you win the prize, but we haven't gotten to the prize yet, so... Yes, so the best reason that this entire room full of... I don't even know how many people are in this room, but there is standing room only in the back, which is kind of impressive. The only reason we can come up with why that light switch, which controls the lights in my living room, should be connected to the Internet is to screw with my kid, who's five and a half months old. It is way easier to screw with at this point than the light switch, but I will remember this for a couple years from now. So, you know, good on you, sir. Okay, now, I've talked about not wanting my IoT devices to have access to the Internet, and yet I have this VLAN that says IoT with Internet access. That seems really odd, because IoT things scare me in a lot of ways. So, there's some reason for this, and I go back to the generally consumers want access to things. There are a lot of gateway-type devices for Z-Wave and ZigBee, and all these kinds of things. And the open-source ones are all great, except they are about as user-friendly as bashing your head on a text console with a brick, which, you know, isn't entirely fair to them. Some of these things are actually genuinely complicated and hard to set up and make good UX for, and some of them are eventually going to try and solve this problem. Great. You know, hand-editing text files to add a new device to my home automation network is not something that I'm going to be able to convince a normal consumer to be able to do. And so, most people buy up these devices from various companies. There's a whole slew of them. I'm not going to name and shame all of them. But if you just go to some large online retailer and type in, you know, IoT Gateway, you'll find plenty of these things. And most are almost universally, these all connect back out to the Internet, because again, people want to be able to pick up their phone, which currently is showing a lovely clock, and be able to turn their lights on and off when they're not at home. Which again, you know, if you have consumer friends who believe this, please, you know, sit them down and ask them why they want to turn their lights on and off from Australia. Because I'm still not convinced that any of them have any better excuse than they want to screw with their kids. Because, you know, oh, we have a new idea from that front. Okay, correct. But if you're home, frankly, you're already on your home network, and if you're not there, you probably don't know that they're anywhere. But your point of using it to scare off an intruder is a potentially valid use case. Although, you know, in some cases if your home automation network is complicated or sophisticated enough, it would be able to detect movement when you're in a way state and then automatically just turn the lights on. And then maybe, you know, in an hour, you know, it'd be able to email you or text you or something. It may make slightly more sense. Oh yeah, automated tool RAM and time also works. So yeah, so there are some ideas here, but realistically, you know, at best, the best one I've been able to find that does the least amount of communication with the Internet is a device I have in my house. Which is from a company called Vera. They've had various names over the lifespan, and it's a pretty decent little device, except when the EMMC on it sales and they don't tell you how to, you know, reflash the device over TFTP or anything. It's kind of a not just button. But it does everything locally except for one thing. All authentication to the device happens at the cloud. So I can still go to the IP address for my device and I can turn my lights on and off and I can check the temperature. But if I want to clear out an alert, you know, that somebody moved in my pantry or something, I have to go and authenticate to their website that then kicks me back to my device with a token shoved into my browser. It is the most depressing thing I can think of if for no other reason everything else runs locally on the device. It doesn't need any other Internet access. And yet that is the best device I've found. The worst is, strangely enough, things like this. And you'll note at the bottom it says prize time. This is a switch that I bought off of Amazon or, you know, your favorite large online retailer that claims to connect to Wi-Fi. I'm like, great. You know, there's a few more things in my house. I would like to connect to things. Z-waves obnoxious. ZigBees also obnoxious Wi-Fi. Great. I know how to deal with IP. Great. So I bought this. Guess how you control it? It only works through the cloud! Awesome. There's this entire instruction manual, which I did bring with me, that explains how you install an app on your phone which, by the way, hasn't been updated since 2015. So it's totally secure. And the only way to get this relay in this device is to now connect it to my Wi-Fi network because I figured this out quickly enough. The only way to connect this to anything is to let it log into Amazon AWS and you click a button. So the person at the end of this talk who comes up with either the best question or the worst question I haven't decided yet, wins a prize. And you may take this home and you may do whatever you want with it because it wasn't worth my time to ship it back. I bet you tell you how cheap these things are getting that I think I spent like $10 on this thing and it wasn't worth my time to put this back in a box and ship it back. And yet here I am, I'm getting a good solid laugh out of it that I'm going to give this away today to some either very happy person or very sad person depending on their perspective. But yes, these are the devices we're putting in there. And this is why my network is as complicated as it is, is that the interactions that these devices have is insane. Why does my light switch need direct access to my IP cameras? What do my IP cameras need access to anything? And yet these are full-blown Linux computers that have BuildRoot and Yachto-based distributions installed on them. These are non-trivial kinds of devices at this point. They're as powerful as your phone. And that should scare you that these things have gotten so cheap, so powerful. And yet we're just putting them on the Internet and letting them talk to random websites. They've got a gig of local storage that they could do things with because IP cameras, they sometimes store things locally at least for a set amount of time. This is insanity. And so yes, I take my IP camera and they get shoved off onto their own network. And part of that reason is, again, since I have a little one, why would I want to give anyone else access to the ability to look at my son while he's sleeping? That is a genuinely creepy thought that just because I want to make sure that my son is still breathing at 3 in the morning and I really don't want to leave my bed because it finally just got warm. How many of you are parents in here? My son is five and a half months old. I sympathize with all of you on those late night feedings now. I grok. I did not grok before. Those of you who do not have kids, you do not grok until you have them. I love my son dearly, but gosh darn it, could he sleep through the night once? My wife's going to kill me. Oh, soon, soon, yes. Well, hopefully sooner than later. But yeah, sadly my wife has stopped dealing with it. It is overnight feedings right now by herself. So hopefully I don't return home and be killed. But yeah, this is the kind of stuff that we're doing. This is one of the specific reasons why people are putting cameras in their homes. People are putting these voice assistants into their homes. These voice assistants do not stop listening to you. If you have a voice assistant on your TV, it does not stop listening to you. It is listening 100% of the time, because at some point you might say the magic word. And for some strange reason, the universe doesn't work that it can trigger and interrupt that says you should wake up now, they said the magic word. No, the way that works is it just listens forever. So if you have one of these devices that you can talk to and say things to and convince to go into an infinite loop if you have enough of them, these things don't stop listening to you. And if you have a smart TV and it has a camera in it, there is nothing that you can really do to prove whether somebody is actually looking on the other end of it, whether you're watching TV or not. Because video cameras on laptops and all used to have the LED that was attached to them, used to be attached to the power line. And so they would only be on when the power was applied to the camera. It's not the case anymore, it's just a soft LED. Which means if you want, you can come take a look at my laptop, you'll notice that the camera has a cover over it. And if you don't have one of these, I highly recommend them. They're not that expensive, they're usually cheap plastic. Or you get them free at conferences, I'm sure when the exhibit hall opens tomorrow somebody will be giving them away. And yeah, yes. So for no other reason I wanted to point this out and try and get people to think about this. And that's the reason when Scale asked me to do this talk is they wanted me to kind of explain why I was insane. And the real answer is I'm not exactly insane, I'm just paranoid. And when you run kernel.org or a large internet infrastructure for a decade, you get paranoid because the target on your back is much larger than the target on most people. But it also means that you're more aware of what that target means than your average person. And so they asked me to kind of come and explain why does this make sense, why do we need this, and this is frankly why. And there are some neat tricks you can do depending on how you set your network up. So you'll notice that my wireless network is not necessarily directly connected to my main network. It is on its own VLAN. And that is because I bridge the two VLANs back at the router because for two reasons. One, you can filter IP traffic on a bridge network which means that you can filter traffic as it crosses the bridge. This is really neat for a number of reasons because you can, you know, let's say I'm on my wireless network and I don't want somebody on my wireless network, regardless of who they are, to have access to the IoT network. But I do want them to have access to my Chromecast so they can throw things onto the TV and make pretty cat pictures or something. Well, great. My wireless network now routes back through my router. I can block the traffic appropriately and everything's still on the same broadcast domain. Because again, most home systems and a lot of systems even in business all assume that they are on the same broadcast domain. What this basically means is when I, you know, scream out into the void, you know, I'm on this network, everything else goes, we know. But when you're on different network segments when I scream out, I'm on this network, the Chromecast, if it's on a different network, doesn't tell me to shut up. It just goes, I didn't hear you at all. And that becomes a complication. So this is a neat way that you can get around segmenting things into different, you know, hands and, you know, domain groups to help with your security. So, you know, if you want to follow in my insanity, which you may or may not want to do, there are some tricks that you can play here. You do have to have pretty much full control over your router, which means you're not running, you know, stock routers anymore. I'm sorry, the open WRT stuff's not going to cut it anymore. But there are some very lovely small board computers that can do all this and Linux and, you know, FreeBSD and all those kinds of things. They all work great. But there's also some interesting problems when you get into the wireless guest network. Because I still want the random people who I actually allow into my house and then onto my network, the guest network specifically, to have access to my Chromecast because they may want to show me pictures of their cats. The record I'm definitely allergic to cats. I don't know why I want to stare at them, but everybody tells me that I should look at cats. How do you solve this? Because you don't want them on the same broadcast domain because that's bad, because then they can get all kinds of things about your internal network. But you also want to be able to give them access to your Chromecast. Well, there are a lot of neat ways to solve this. And Avahi, if you're all familiar with it, does have some ways of repeating broadcast traffic and doing it in a filtered way. Effectively, you're doing MDNS at your router. This works great. If you have devices that don't play with MDNS correctly, you can always set up tunnels, little software tunnels across the networks at your router. And this all works. Works great. I'm living proof. If for no other reason then, I haven't had any complaints in the last six months of things not working on my network from the users who have very strong views on this. Yeah, this is all a mess. I've got a little bit of time for questions. But before I give you the opportunity to win this fabulous prize, there are some things here that I want everybody to kind of take as a take-home. I may not have explained them nearly as well as I would have liked to in this talk, but I only have an hour, and this is an insane topic, mostly because I'm explaining my own insanity. And yes, my network is likely to be overly complicated for what we actually need. There is, in some respects, no reason for me not to just lump my IP cameras in with my IoT devices and just accept that they're all one type of mess. Yeah, I bought high-end network gear. I can have, you know, 512-some VLANs. VLANs are free. I'll just do that. And the equipment we were putting into our houses was nicer, and we had better infrastructure available to people. We could do things in even more clever ways. Things with, like, WPA2 Enterprise. When you authenticate to a WPA2 Enterprise device, you can actually set VLANs based on that authentication, which means you can expose one SSID instead of the six I'm exposing at my house. And then you can route things into different VLANs. But that means that this type of device needs to be able to talk WPA2 Enterprise. And most of them don't. So, that is a thought. We need to do a better job of educating users. And by users, I mean probably not the people in this room because you're all very smart. You're sitting in this room. We need to explain to users that, you know, their network topologies, as simplistic as they are, don't work anymore. They're fundamentally not safe, particularly if you're adding IoT devices to it. Because the IoT device will be the infection point for your entire network at this point. And if you have them, I guarantee it. And it may not just be because you may go to a website. The website will launch a bunch of JavaScript that will go and probe your internal network, find the IoT device, and then infect it. And this is a known attack. So please run Adblocker. If you're not already, please. Home networks are no longer simplistic things. They can't be. The belief that they are is just silly. Most users don't understand what they're actually getting into. Frankly, most of us don't know what we're getting into with networks, but that's a different problem. And we have to be better at every level about the defaults that we give to users. So if you're working in projects like OpenWRT or router stuff or you're building products, take a look at what you're doing and seriously ask, does this make sense? Why is there not a default IoT network exposed by every router on the planet now so that I can connect this stupid device, I mean, stupid in every sense of the word, to the Internet, and then it can't infect things that don't make sense to infect. Yes, it may still participate in that distributed denial of service attack, but at least it can't do any more damage to the things I actually care about, like the baby photos of my son, that I'm trying to think about. And while a lot of the things, when I showed you my network, they're very complex, it's not something a lot of people would even think about trying to set up, because VLANs are almost mystic knowledge, depending on what switch you've gotten, how you set it up and where in the configuration, and it's not simple. We all have to think about how we can take these incredibly complicated things that we know this stuff can do, and we have to get it to the point where it's simple enough to understand that any of us can use it at least at the most basic level. This is my fair-mongering talk of the year. And at this point, I thank you for sitting through me being silly. And if you've got questions, I think we've got a mic that I may run around or something. Actually, if you have questions, line up behind the projector. We do this at work. And I will badly answer your questions in the order in which they are received. De-solder it? I want to do something. I don't have a good answer to that because laptops are designed in such messy ways. What? Purism is creating devices that have physical hardware on-off switches, so take a look at their stuff. Yeah, an earplug or something just shoved into the mic. I don't have a good answer for you because it's hard. You could, but what does that mean? But just because software tells you it's disabled, is it? That's the gotcha. It could be connected in any number of ways. The question is valid. I don't have a good answer for it. De-solder it. The best answer I can give you, but I'm not a security person and I might be overly paranoid. Place where something is posted is to what would constitute diligence for a ordinary person to have reasonable security for some value-prisoned security at home? Number one. Number two. If such a thing exists, how many man-hours per year would it take for the person to invest to actually make that work? Ooh. Because you're learning how things are changing. Oh, okay. So the answer to your first part of your question is I do not believe that anything exists, but frankly something should. And now I think I'm stuck being the last one touching that idea, which means that I'm probably going to have to do that. Because this is the way the universe works. The last one touches it at their head. And two, I don't have a good answer on that, but rough needs, well my knee jerk reaction will be if somebody goes and takes a look at some sort of updated document, maybe once every six months and just sees, even if there's just a diff of what changed since the last time I was here, that I should be doing differently or better or something. That might be good enough. So initially it may be a lot of work, but the upkeep should be, I mean, once my VLANs have been set up, it's not like I've had to do a whole lot to continue that security model. That beyond section. I don't know. It's an excellent question. I genuinely do not know, and I'll have to go think about that. So. I don't see what stops them from crossing the VLAN barrier, or crossing the Wi-Fi barrier either for that matter. If they have a Wi-Fi transit receiver, how do you know what's going on? So the simple, you know, how paranoid is John? I think the simplest way to put that question. At some point you have to draw a line on how far down the rabbit hole you want to go, because if you keep going, you go all the way down to, I need to make my own computer chips. And I have to make my own computer chips. I mean, me, I have to build my own fab from the, you know, literally dirt to 29-millimeter fab so that I can build my own chips. So yes, there are attacks on VLAN hopping and all that kind of stuff. And mostly expecting the people who are doing these attacks, the bot-bets and whatnot, they want to go after low-hanging fruit. I want to make myself so on low-hanging fruit that they either can't figure out what I'm doing and go away anyway, or that I don't fit into their logical model and I just, I become too much of an annoyance and they literally just go away. So yes, you know, if I convince everybody to do this, I've literally lowered my own safety threshold because now I've made everyone, I've floated all the boats, which means now the low-hanging fruit is much closer to me. Yeah. The bikes are better for the neighbors. Yes, I don't need to be the fastest person. I just need to be faster than the last person to outrun the bear. We have that vital of you there producing, I guess being more responsible, I don't see much use. Oh, okay. You mean like brand life? There are some... Okay, the best answer I can give you to that is go and take a look at the devices that Home Assistant, specifically. It's a Home Assistant and open source project that does gateway kind of stuff. If they support it, the company is at least giving you, is likely either giving you direct access to the device via your local network, and that should be mentioned probably somewhere in the description, or they're giving you an API through their cloud. If they're giving you an API through the cloud, you'll probably need some sort of API key when you would set that up. So that should be pretty obvious in the documentation. That is not the device you want to buy. Unless you trust the magic of the cloud. The company shouldn't be disappointing to try and... The two switches I have mostly in my house right now that are Wi-Fi based, TP-Link, has an excellent one that you can completely control directly on your own network. No cloud access necessary. And there's Sonoff. Sonoff, S-O-N-O-F-S, I think. Yeah, but you still have to set up something else to control them. You still need something like Home Assistant, or if you're willing to, you know, give Vera a little bit of Internet access and a little bit of access to your home, those kinds of things work. Otherwise, we should probably be looking inside the open-source community and trying to help projects like Home Assistant get better. The other thing you mentioned was the WPA2 Enterprise. The kind of look for switches that have that. So the most wireless access points do support WPA2 Enterprise. The things you're going to want to look for, like if you're looking at access points, are going to be whether they support VLAN tagging based on WPA2 Enterprise. I know that devices like Unify do support this, which is from a company called Ubiquiti. But the gotcha is that you need to be able to support the authentication scheme that WPA2 Enterprise needs, and that requires radius and radius and kind of a pain in the butt to set up. So it's a really good idea that even for me is relatively obnoxious to try and deal with. And I haven't quite gotten my setup working yet. So you think we're having a free-of-the-marge for anything, just wait for the next generation? No, I'm not saying wait for the next generation. I'm saying go buy the stuff that's there now and help us fix it. Wait, come help. Do you use IPv6? Do things like people don't really run NAT anymore? No, NAT fundamentally should die with IPv6. Okay, and devices are a lot more uniquely addressable to the internet. So the question is, do you see this helping or hurting the problem? So I have very mixed views on this because having run large internet infrastructure, having a flat topology makes everything easier and better. However, also having devices sitting on my home network with IPv6 scares the bejesus out of me. It's kind of like you're losing a layer of firewall. You are fundamentally losing a layer of firewall, but NAT is fundamentally a lie anyway. So all we're really doing is we're getting back to what we should have had in the first place. More or less what that means is that you just have to be more diligent when you're on firewalling. And this isn't to say that having this pingable on the internet is a great idea. That may not be, and you should be able to stop that at your firewall. Or you just assign it, there are private IP ranges that are specifically non-routable and non-native. I believe they're supposed to not be NAT-able. So local link layer IP addresses that you could assign to this so that it doesn't talk to the internet. Do you think home routers should want incoming connections by default? Yes. If your router is not blocking all incoming connections by default, go home, throw it in the trash because it's not even remotely doing its job. The problem is it's not just that people think it's like that, but also that the original designers of the internet thought the internet would be like that. Yes. So I believe the real approach or one other approach to solving this problem is that we completely redesign how networks can stand. I won't disagree with you on that. The gotcha is IPv6 is already effectively deprecated and it's taken us 20 years to get to this point. And how many people in this room have IPv6 at their house? Yeah. So I don't disagree with you. This is something we could fundamentally fix by taking everything we know about IP, throwing it out and starting over. But that will be completed sometime around 2100. Well, I can put that, and I'll actually have a talk with it on Saturday. What's the project? It's called the last up. I will talk about it. The talk is about let's fix the internet. Okay. I hope you will be here. I will try. Give me some feedback. I will try. So there you go. Pitch for his talk. Let's fix the internet by burning it all down. Let's fix the internet is the talk. It's on Saturday. So from a hardware perspective, it gets really complicated. Probably the best thing I can suggest to you that is off the shelf, that it can do everything that I kind of described, that has a user interface that is not like, please edit this text file. Please edit these 20 text files to make this firewall work, which is what I do. PF Sense, which is an open source project. There's actually commercial support and you can buy specific hardware that does all of this. I think it's stored up, pfsense.com or something like that. The boards are sold from a company called Netgate. And open sense? Okay. Okay. So apparently PF Sense doesn't build properly. Thank you. I will go bug some people about that. Actually, I probably know what they're doing. Anyway, but yes, open sense or pfsense. I mean, just go take a look for those. Those are the things I would probably recommend. If you're much more comfortable with the command line and you like hacking on things, my firewall is entirely based on IP tables for the wrapper called shorewall. And I just run it on a literally just a random x86 box that I basically pulled out of the trash and a couple of 10 gig nicks that I threw them out. So, yep. Yes, I did mention ubiquity. Their firewall does not do what I want it to do. But their access points and their switches are pretty decent. Okay. For your lightweight... Yes. ...so cameras, just to get them on your system. Yep. So what happens if somebody can show up with an open Wi-Fi connection distance of the IP? Do they... Even if you lock them down, do they start checking in with the letter shift? So, let me rephrase your question and see if I rephrase it correctly. If someone knows... Well, my SSID is being broadcast, so everybody knows what my SSID is. And if somebody was to know what the password was for that particular network segment, which one would hope that they don't, but let's say the WPA2 has a giant crack in it. So, the only thing you can do is to pop your network down and kind of stand up and open them. Yep. If you've got two and a half acres, you... Yeah. They've got plenty of spaces to do it in. And I wish to see if she was working with a cold area and people had to go up on a hill but that she couldn't find in order to steal her dish, but they came over and sent it back to their network. Yes. So, yeah. So, if somebody was to fake enough of my network or something to get those devices to actually reconnect back out to the Internet, yes. There's not a lot of ways that we can stop that particular problem, other than one would have to hope that the device, once it's connected to the network, would try and attempt to forcibly only connect the way it's expecting. So, you know, if you stand up and open network, one would hope that it doesn't actually just, you know, fail over. That's not to say that that wouldn't happen, but now I'm curious. I'm probably going to go home and try this and then be like, oh, God, it's all broken and this will probably turn into a fucking death con or something. I have no idea. We've got a TV. I connected up to it. And now the cable guy has an update. Just connect to the network cable. Now I have no idea what the device might be doing. Correct. Well, it's fundamentally the same problem of, you know, if you own a large house or whatever and you've pulled the Ethernet to the outside, like, I don't know, they connect to your fiber, you know, there are problems of, well, what happens when somebody comes up, pulls that connection and plugs a different one in. It's more or less the same problem. There are ways around this. 8021X, WPA2 Enterprise should fix that in the general sense because of how it authenticates because it's basically 8021X. But WPA2 Enterprise, if you're flipped over to it, it won't connect unless it authenticates correctly. So, but that's Enterprise, not PSK. And I don't know what happens with PSK. So, yeah, now I'm curious. It would have to be at the device layer. And the problem is, is that getting access to these things is almost impossible. These I effectively have to treat as a black box that we will never be able to fix. At least that's the way I look at it. Well, yeah, it's a pain in the butt and it's trivial to spoof MAC addresses. In fact, most of your phones probably are randomly assigning their MAC addresses as they connect to the network. It was a connected Barbie, was it Matthew Garrett? Yeah, it was Matthew Garrett who did the teardown. And they actually did genuinely do a bang-up job on that. So, if you're ever curious, that's actually a really good read. So, if you're building IoT devices, that's a way to go and look. So. I mean, this is what my favorite watch and home cat was. So, there's a couple of IoT devices running around right now that effectively provide their own access points. My Vera, for instance, has, my Vera actually runs OpenWRT, which is why I'm slightly annoyed that I can't convince them to let me have the bits to reflash it if I needed to. But I haven't really tried very hard enough, so that's a different problem. But there are things like the Fire TV. The remote for the Fire TV these days actually does what's called Wi-Fi Direct, which means it's actually broadcasting on Wi-Fi SSID to do connection with the remote. And there's a bunch of things like that. It's probably bad in the grand unified universe view. It's probably bad because now it's poking holes in a security layer that you don't control, necessarily. So, like the Vera, I've actually turned off all the accessibility on those devices in my house. Mainly because I have much nicer access points than what they'll ever provide. But things like the Amazon Fire TV with its Wi-Fi Direct, I would like the remote. The remote makes this usable for me. But is Bluetooth really better? Is Wi-Fi better? I don't know. And there's just the devices that bring their own access point for whatever reason, whether it's Wi-Fi Direct or they believe that they should be the one true gateway for everything. I think they're probably misguided, at least in some sense. Now, I do have some faith that Amazon, if they do get hacked for whatever reason or their Wi-Fi chip has an issue, that they'll push an update. Because, frankly, I think they have slightly better hardware engineers involved. But I can't say that necessarily about all the companies involved that do these kinds of things. Because who knows, maybe the device that I bought today that I don't think I bought any devices today. But if they ship an access point in their device, how do I know that they'll get updates? How do I update it if it's a problem? How do I even disable it? Which goes back to the question about how do I disable my mic? How do I know that the Wi-Fi chip on my vera is disabled? Well, I don't see any SSID broadcasts that would match anything to it. So I'm reasonably sure it's off, but that's not... That's absolutely guaranteed unless I go in there and I rip the Wi-Fi chip out. Yep. Yeah, it does come down to a, well, how do you reasonably authenticate stuff? And yes, if they provide their own access point, and you've got to go over and push a button so that it sets itself up and then pushes some data, I don't know. That may or may not be a good thing, and some of that's just going to be a... If you look at it and your risk assessment says that that is a scary thing, don't buy the device. Again, some of that does come back to a, how paranoid do you want to be? Yes, I'm going to get it decided. Yeah, so there's a lot of neat devices these days, because GSM has gotten so cheap in some cases that in fact I'm backing something on Kickstarter right now, which is a little device that you slap it in the back of the mailbox, the ones that have the key to everything that are up the street from you, and then tells you when your mail came. They're doing light sensors and scanning and all this other kind of stuff, which is neat. And you can buy up the credits for the GSM for a year in the five bucks or something like that for the GSM credits. But yes, how do you protect against that? Effectively, if it's on the GSM network or if it's on the public Internet or on the Wi-Fi here, then you basically have to treat it as a public device. It is just out there. If you want to trust it with anything, you're probably insane. Now, if you want to just believe that it told me that my mail came, and everybody else on the block, oh, that goes back to the Amazon door thing. Yeah, it's neat. I like the idea, but it's going to suffer from the same problem of the connected car. The mailbox generally will not kill me unless somebody else has done something else malicious. But don't do those things. That's bad. I think I'm going to give the prize to the question that I could answer the least, so congratulations, sir. You want a prize? You may do what that is you wish, up to and including thermiting it, which I would probably recommend. Anyway, let me get to the end here again. My name is John Holly. That's my contact information. These slides I will get up to scale, and they are CC-BiSA. Except for the pictures, those are all copyright-respective whoever they did it. If it does not have a copyright notice, it's probably my picture, which is probably CC-BiSA. Thank you. We'll be doing more embedded tomorrow in this room. I'm starting at 10. Thank you. We knew that the cameras were insecure to begin with.