 Hi everyone. I'm going to give a talk about a series of workshops that was run in London over the last year about introducing NetBSD to a group of hardware enthusiasts. In London, we have a series of events that have been running for a few years called the open source hardware user group. At some point last year, I was asked on the theme of operating systems to come and represent the BSD family to these hardware enthusiasts. I thought about how I would kind of present this because most people don't really care about software. It's something that they have to kind of suffer to get their job done with their hardware. On this event, we had a series of talks from myself doing the BSDs, a group of my friends representing the Linux family, and then just to kind of keep it really out there, we had Charles Forsythe presenting the Plan 9 family operating systems. And the feedback was quite good. And moving along a few months, I got the opportunity to actually run a series of workshops to get people started. So kind of thinking about how I would actually present the BSDs to them. I've never done any kind of teaching before, so I thought that trying to actually run a workshop with like three or four different operating systems is quite a big take on. So I thought I'd just stick with NetBSD, because I could think of some things that I could demonstrate to them that would be beneficial. And so I did that. And what I've kind of focused it around was basically around saving time. The reason was, going back a few years back in about 2013, we actually did a series of workshops about introducing people to FPGA, FPGA programming. And it was meant to be an event run over a weekend. And basically our first day was completely lost to basically trying to get the Altera toolchain installed. So what happens is with like the Altera toolchain is if you have lots of money that will sell you a very expensive ID. And if you don't have it, you will get the same thing, but it's like the free addition. Now you'd think that if you don't have a license, you know, you could kind of operate it in the free mode, but that wasn't the case. And most people kind of, including myself, found this out where we kind of worked along with the application. And then when we actually came to actually synthesize our work, it goes, oh no, you don't have a license, you can't use this. And so you had to literally uninstall and go back and reinstall 12 gigabytes worth of application. So from this experience, from the group actually, some folks got together and worked on a project to actually make a new board. So you're not a victim of these kind of closed tools. So the project that they've kind of put together is this MyStorm board, which is based on this Lattice FPGA chip, which is also open source. And yeah, you've got a nice, very light toolchain, and you can do stuff with open hardware. In other places in the world, you know, things are kind of still growing. 60 gigabytes worth of toolchain just to install before you can kind of get started. So why? Kind of to raise expectations. Like, in this case I'm presenting in net BSDs, there's a lot of things that we do in the BSD family that, you know, kind of just avoids a whole bunch of problems that other people in other areas have to kind of suffer. And just, yeah, demonstrates a little bit of respect for the user that you can, by using these, this piece of software, you can save yourself a whole load of time and headache. But the thing is, is kind of, if you actually wanted to kind of get involved and develop, there's quite a steep learning curve. So I really want to kind of keep it high level because, yeah, we're not going to kind of get into actually trying to do something properly in the space of a short workshop. But the thing is, as a culture, you know, resources are available. Like, I think we're approaching, next month we're approaching like the 40th anniversary of like one BSD and that's, you know, all that is available open to track online. So you can follow the ancestry and see how the operating system has evolved. There's great texts, you know, like the Richard Stevens books, but that's not a light read. You know, the application programming in, advanced programming in the Unix environment is a fairly hefty, hefty book. And, you know, learning C, yeah, you can learn how to do like hello world and stuff, but can you actually handle pointers and things like that in a safe manner? And can you actually demonstrate that and teach that in the space of a very short workshop? Unlikely. Design and implementation series, you know, I think the first book came out about 20 years ago. So this is not a culture of, you know, describing your intention through comments or, you know, forcing people to read source code. There's actually decent text to kind of supplement the software development. And aside from that, there's proper courses as well. So this is completely the opposite of what's there. It's kind of more of a gallery of how can we, over the space of like half a day, get to do some stuff fairly quickly and fairly safely. So what I was kind of thinking about was these brittle tools and painful experiences that people have to kind of go through. Have we all seen like Brendan Greg shouting in the data center video? So, you know, that video is coming up to like kind of 10 years of age and still like most people still don't have the ability to instrument their systems in real time. So I thought we should kind of really demonstrate something around that. There's a gentleman by the name of Brett Victor, who's a researcher. And he did a talk called Inventing on Principle. And he kind of showed this environment that he'd kind of built as a proof of concept where when you have a rich environment, which allows you to kind of experiment with stuff very quickly and your feedback loops almost instant, it becomes a case of it's not always a reactionary, your workflow isn't always reactionary. You can actually start to project where you want to go and it becomes kind of much more intent-driven and forward-looking. So I was kind of thinking about that as well and how I could present that to the attendees. Brian Cantrell, I'm sure most if not all of you have heard of him. He did a talk back in Lisa 2009 about de-trace and how it came about. Before de-trace was a thing, Sun had a very big computer called the E10K and to actually go from a reboot to the system being functioning was about 45 minutes. And he was on a customer site trying to debug a problem and the system kept on panicking. And every time it panics it's like 45 minutes and then you come back and you do something. That kind of becomes even worse when you're actually in operations and things like that where your kind of feedback loop from debugging the situation to actually implementing it in production can be quite a long time and that's you lose lots of context. And yeah, on the theme of kind of brutal tools. And so when you don't have any, your coping mechanism just ends up becoming like kind of abstracting your problems away, just kind of piling stuff on top of each other. Or if you're dealing with like kind of closed systems and there's no way for you to kind of participate in the system or change it, you start manifesting these kind of really bad habits. Windows users who try to avoid all costs rebooting because as soon as you reboot, that's like three reboot cycles of updates installing that you lose time on. Or you're dealing with a project that doesn't do project management properly. So you're in a rolling release model. And you don't know if you apply a patch or update to a new version, what's exactly going to change when you're actually just trying to fix one bug. And so that kind of end up manifesting it into like fear to explore ideas because the system is so clumsy and brittle that even the slightest change, it's going to come tumbling down on you. Or I'm sure most network engineers have this, there's no room to maneuver. The only thing you have is the life system because redundancy is a licensed feature. So some more bad habits where we could avoid this whole problem by going serverless. Unicernals, be away with the ops team, anti-cernal, everything's in user space. So, George Neville Meal I think, I think last week or maybe a week before did this quite nice article about these kind of cultures, about avoiding kernel programming and trying to push stuff into user space. Some of the things that you may not be aware of. And PHK a few years back did a thing called the generation lost in a bizarre, which was essentially a book review of Frederick Brooks's the design of design which kind of talks about institutional knowledge and kind of keeping a log and documenting your processes. So what are we going to explore? Constraint is how do you make this, present this operating system to someone who doesn't care about operating systems and ignore the software features because it doesn't matter, they don't care about the bells and whistles. So what did we explore? Explore the documentation. Your first port of call isn't Google and someone's blog. We have our manuals online and you can kind of drill down from there. If something isn't documented that's a bug. This isn't actually specific to NetBSD but that's what I kind of presented. The source repo I think actually right at this moment there's a talk on the UNIX architecture somewhere else about the UNIX history repo. They're a privilege where we can kind of go back I think 37 or 38 years so we can actually see how the operating system is involved. Which is maybe not a privilege for somebody in a fragmented operating system environment. And cross compilation is really easy, there's no elaborate setup process. So the idea is an attendee would get a copy of the NetBSD source code, a copy of the package source code and would start building. We'd avoid trying to do stuff safely and see by leaning on the lower environment that's in NetBSD. How to do rapid prototyping on your actual machine because it's bound to be faster than the Raspberry Pi or the ARM board that you're going to kind of work on. And sometimes you don't actually have the hardware to kind of, you don't have the hardware that you're actually targeting. So we have some various simulation devices which allow you to kind of work away from that. And so in the world of IoT security is an issue and so I thought about kind of demonstrating Tampa resistance. So should it binary change, the system won't execute it. And we kind of got away with it. There was some, the process was a little bit rough in certain places. So I had to kind of explain to the people who turned up with like a Linux laptop that they have to install the UCP package in order to obtain the CU command. So they can connect to the serial adapter and then you have to change the permissions. On OSX it can be a lot worse because if they're a bit confident they'll stop googling about how to kind of install the OSX drivers and they'll probably land on a blog post that says you should turn off kernel signing of drivers just to get some obscure driver that you've downloaded from a Chinese mirror. So we still, the main tool chain in FBSD is still GCC. And if you're on a Mac and you're building with Clang, the nesting in GCC for C++ kind of blows up. So I had to kind of explain to the users about adding extra flags to get the build done. And ideally we want to kind of do away with serial header. But the reason I kind of went down the serial header route was I didn't want to actually have to walk through a classroom full of people on how to set up networking and you know, they have to be in the same subnet and this is how you connect. So I kind of went down the serial header route but then obviously I had these problems. But one thing we've been thinking about is for our ARM images we have multicast DNS set up. But we have this problem at the moment that there's no initial user. So there you have a device that's connected on the network and broadcasting itself. You can't actually SSH in and connect. So I kind of, in preparation for this, I was kind of making sure that I covered all the angles. So I did a lot of testing on SIGWIN and Windows for the cross compilation side of things. And we kind of quickly got into Autoconf Hell where some of the things that we had in base just had no knowledge of like Windows 64 bits addition. So it kind of fell short very quickly there. Unbootable images. For the ARM boards, there's no unified way of booting a system depending on what you have. You have something as drastic as you know, DDing a blob to the file system to just having a file on a DOS partition that's marked active which is a thing from like the DOS era but for some reason it's kind of transferred into the world of ARM for some reason. And we actually had this with our Beaglebone images where we were generating the images but the FAT file system wasn't actually marked active which means when you apply power to your ARM board, you boot will not actually look at the DOS partition for the boot loader and wouldn't boot. I didn't keep any notes, I can't remember what that actually was. Sorry. And so the two boards that I actually targeted was the Raspberry Pi and the Beaglebone Black and it turned out that there was some inconsistencies between what was there in the configuration files for the kernel between the two boards which meant some attendees couldn't do some of the steps and the other ones were, yeah, that was the thing. So I actually had to take a minor detour into getting people up to speed and break and you boot scripts. So we have a generic ARM image and the script tries to enumerate what board you're actually booting on and then load the relevant kernel based on that which means that we don't have to have loads of different images for different boards. I think moving forward this is a non-problem because we end up actually relying on the flattened device tree and we just actually have one kernel and we can do a way with this but the way that we actually enumerated which board you were running on was broken which meant on certain Beaglebone Blackboards it wouldn't boot. So since doing those things we made some changes to NetBSD. Mark Baume brought the lure up to date in our base. Jared McNeil made the U-boot packages which means it's literally adding three new variables in a make file if you want to support a new version of U-boot. Very exec which was the tamper-resistant mechanism with supporting kind of things like MD5 and SHA-1 which are effectively broken so we removed that and the setup process is no longer required because it's working out of the box. But there's still a lot more that we can do to just kind of make the process really smooth and really easy for kind of people to try to try things out. As always there's more documentation needed. More documentation needed to kind of explain the various subsystems. More lure and vrump examples. So what I wanted to do was basically instruct people to show up with an onboard and a computer and we can basically compile or you'll be given a copy of the source code. We can build an operating system and you can boot it on your onboard and then kind of work through some exercises. What I wanted to kind of avoid is actually having to request that they show up with a breadboard and you know various components for me to have to supply a breadboard and various peripherals to try. So I looked into what I could do actually on the board and I think for like the GPIO exercises the Raspberry Pi lets you control one of the LEDs which is there for the Ethernet so you can kind of switch that on and off and not much else. So I want to kind of do something a bit more elaborate than just trying to switch an LED on and off. Yeah so we've been kind of discussing how to actually add users to an image so that the user just basically downloads an image for the operating system or takes the image for the operating system puts it onto an SD card maybe adds an entry for a username and a password and they can boot the system and we don't have to fuff around with serial console drivers and things like that. Mark has this Unix binding which is gives you various system calls and functions in Lua so we can do a bit more systems related things from our Lua environment and maybe kind of look at the testing framework that we have so we can further move away from doing things on the actual board into doing testing locally. So spinning up an instance of QMU running through some tests and then shutting the system down you know if the system worked okay. Some people that I'd like to thank Andrew Back who invited me to do the talks in the workshops, the open source hardware user group, the London Hackspace, some NetBSD developers who helped along the way. Do we have any questions? All right thank you.