 So this talk is not a very code heavy talk, but there are a bunch of command lines that you might like to copy and paste if you want to try things out. So you want to scan the QR code, go to the URL. There's a copy of the slides right there, so you can follow along at your own pace. My name is Taylor Campbell. As you can see, there's very small text that we had to write on this badge because printers seem to have missed it. Or I also go by Riastra in NetBSD. I'm on the core team of NetBSD. I've been hacking NetBSD for about 15 years. And I started hacking it fairly early on. I've been a NetBSD developer for about 12 years at this point. I work on various things in NetBSD and anywhere from networking to file systems to concurrency to drawing pretty pictures on the screen, which I'm not very good at. And yeah, so I got started hacking NetBSD fairly early on in using it. And I'll go into the origin story a bit in a moment. But first, this is the main thing that you need in order to start hacking NetBSD. You just need to check out the sources. You can do that with CVS or with Mercurial or Git. The main repository is CVS. There's an automatic conversion to Mercurial and Git that is ongoing. It's off by a couple of hours, updates every couple of hours. And then run build.sh. And that's it. And you don't need to be running NetBSD to do this. You can do it on your MacBook, running macOS. You can probably do it on an OpenBSD. You can do it on a Linux under any reasonably POSIX-ish system. The requirements are fairly minimal. What will happen is that build.sh will first build a cross-compiler tool chain and then use that cross-compiler tool chain to build NetBSD. In NetBSD, unless you go out of your way to do it differently, every build is a cross build. And this is very nice for keeping the build isolated from the host environment. So you don't have to worry about structured definitions being mismatched between the host and the resulting NetBSD. Everything starts with a cross compiler and a clean environment, very well-controlled build system here. So this is how you get started. And this is how I got started, too. It's been like this for about 20 years that you can get started this way. And that's a good part of what attracted me to using NetBSD in the first place. So I had previously lived in the Apple world running macOS and well, it was called then macOS 10, written with an X but pronounced 10. And it's a Unix system based on some bizarre hybrid of BSD and Mach with free BSD kernels, some free BSD, some NetBSD user-land components, and then a bunch of Apple stuff on top. And yeah, Apple puts a lot of effort into making the system nice and cute to use, do a lot of work on polishing all the rough edges off, and you can also get reasonable development environment in macOS. But it is still very much controlled by Apple, and they get to decide what you get to do with your computer. Sometime around then, maybe 2007, 2008, Apple decided you shouldn't be allowed to run iTunes under a debugger. And I thought, this is my own computer. I'm the one who should be in charge of what software gets to run on this computer, not Apple. And I was offended by this denial of autonomy of my own computer. It's a general robust computer. It's a Turing machine. It can do whatever computation you want. In principle, I could, and I'm sure someone has figured out some work around that will let you actually run GDB under iTunes. They run iTunes under GDB, rather. That would be an interesting concept running GDB under iTunes. I don't know what that would mean. But anyway, I don't think they forbid that. But they did forbid running iTunes under GDB. And it was the autonomy of using my own computer to do the computation I want. That was really off-putting. And I knew that Apple controlled the operating system. They controlled what software updates and stuff happened. But I could still generally use the system. But you can't use a debugger for the things that are not blessed. That really ticked me off. And so I started chopping around for something better, something that would respect my autonomy. And that would also run on my PowerBook G4. Because that's the laptop I had handy at the time. And the student didn't have a lot of money, didn't want to have to, didn't have the cash to shell out for a nice new laptop or anything like that. So this is after Apple had ditched the whole PowerPC landscape. And so I had to find something to run a PowerBook G4. And I chopped around a little bit. I'd run OpenBSD previously on some Intel tower that I used as a home router for a few years. I didn't do very much with it. I just ran it as a home router then. And so I looked around at some Linux systems, some BSDs, and in that BSD. And now, I had this spare PowerBook G4. And I also had an Intel MacBook, a 32-bit Intel MacBook, in fact. This is one of the one generation of Apple hardware that was a 32-bit Intel system. So it was kind of inconvenient for a while. But I was using that as a main laptop. And I wanted to test things on G4. And so I wanted to make sure I could build it from the sources and have the control over what I'm actually running. And the only thing I found that made it really easy to do that was NetBSD. And so I gave it a shot. And this is the first email I wrote to NetBSD mailing lists back in June 5th, 2008. I guess they're maybe too small to see. I tried to get a small screenshot, but anyway, showing my tribulations with trying to follow the install instructions, which were complicated, but they worked smoothly once I figured out which steps to take. And then within a couple of weeks of messing around with it, installing packages, doing some package source, I found a bug, which turned out to be a kernel bug. And this is actually kind of exciting. I mean, it's not great that the thing I was trying to test out as a potential replacement for my computing platform has bugs. But I think we're all going to have to live with the fact that there will be bugs. And so it was exciting that I found this bug. I reproduced it in sTunnel, this program to make a TLS tunnel between two servers and expose it on a local socket so you can transparently make your IMAP client if it doesn't support TLS, talk to an IMAP server that's worth TLS, something like that. And I went digging around with this really awful API in posits called the CMessage API for ancillary data on a socket. This is both used for weird IPv6 features and for passing file descriptors on a socket from one process to another. And the API is just awful and not for any good reason. You have to assemble a buffer with the right alignment. And there's these macros, CMessage length, CMessage space. And you have to use to get the right allocated buffer and a frame within the buffer to assemble it all yourself. And if you get the alignment even subtly wrong, then it won't work on some architectures like PowerPC, except actually it was the kernel's bug that it didn't work on PowerPC because, well, PowerPC has strict alignment and X86 does not. And I guess nobody tested this particular path on PowerPC. So I decided, you know, this seems to be a reproducible bug in the system call. It seems to be doing the wrong thing based on the documentation I could find of how CMessage stuff works. And so I decided, OK, I'll just open up the kernel source code and read it. It's just a bunch of C code, right? I mean, a lot of people have this impression that kernel hacking is this dark, mysterious, black magic. And it really is just a bunch of C code that it has some constraints. You know, you can't just forget to free memory because you need the system to continue operating. You can't just kill the process and start over because the kernel is the thing that is responsible for freeing the memory that the process failed to free. And so there's some constraints in kernel program. But it's just C code. And C is a fairly simple straightforward language. You know, there's dark corners of C, but it's nowhere near the level of complexity in the language of C++ or Rust or what have you. And so I just decided to poke around. And I tried to guess how CIS calls were implemented. I think what I did was I just grepped for some long CIS call names. And I found there's a bunch of functions called CIS underscore and then the CIS call name. And, huh, these seem to be system call functions that they seem to match up with what the system calls are called. And so I looked at them and they have these arguments that match the arguments and laid out in the structure, but they match the arguments that are in the man pages. And I just kept reading and I found, okay, well, this seems to be the send to function, the send to system call. And I dug around and found where the C message header framing is parsed and I found that there was some alignment criterion that had a, maybe it was an equal sign where there should have been a less than sign or a greater than sign or something like that. And I thought, huh, that's interesting. It looks like if I trace down the values that go in here, then with on the PowerPC, then it will take the error branch. And if it's on X86, then it won't take the error branch. So I don't remember the detail. This is 15 years ago now, but something like that. So I proposed a fix on the NetBeast users mailing list, just one line thing to change that criterion. And within a few days, Christos Zoulas committed it. And that was exciting. Wow, I had just found a bug in the kernel and found a fix, the fix I suggested wasn't quite enough, Christos made it some other changes as well. And it later turned out that in fact, it wasn't quite fixed until several years later. This C message API is really bad and file district of passing is also difficult. But anyway, I was excited because I'd found a bug in the kernel and I've never done any kernel programming before. And I proposed a fix and the fix was actually a reasonably good approximation to the correct fix and someone committed it. And I could actually do something. I'd never done this with macOS. I had no chance and I've ever doing that until well. Maybe Apple would hire me now but I wouldn't want to work for them anyway. So then I thought, okay, well, I can do kernel hacking. It was great. So I bit off a little more than I could chew and decided to try getting the old Broadcom Wi-Fi driver working for the PowerBook G4's airport extreme. And I spent a couple of months working on that. And I had no clue what I was doing. I just copied the code from OpenBSD. You looked at the dragonfly code as well for reference and I went through and I have no clue about Wi-Fi. I still don't really have any clue about Wi-Fi but I just tried to make it compile, see what happened. And I got it to compile and then I went through and tried to make it link and got it to link and I went through and tried to make it run. And just whenever things didn't work is just tried to poke around at it some more. See what dragonfly's doing, see what OpenBSD's doing, see look at the man pages and just worked at it and I managed to get it working by September. I'd never done kernel hacking before, I just dove into it. And it was easy because I didn't have to worry too much about my development and machine crashing because like I said, I had two laptops at the time and I could just work on my MacBook, thank you, work on my MacBook and cross-compile and then, oh yeah, conveniently these Apple laptops had target disk mode where you could just boot into target disk mode, it would expose a firewire disk. You could just connect the two machines with firewire, copy and or kernel over and then iterate and so it was real fast. That's not really a thing anymore but there's other options like net booting with NFS and there's other ways to iterate kernel development which I'll get into later. And so yeah, it took a bunch of work but I got it working, of course. Like I said, I'm not really good at wifi, I don't know, I don't understand how it works. This driver isn't that great and also Broadcom hardware is a pain and the documentation is actively hostile and there's newer Broadcom cards too which with a different driver, BWFM. But anyway, so let's get back to how you can hack that BSD too. So as I mentioned, you can check out the sources with CVS or Git or Mercurial. At some point in the near future, we are going to switch away from CVS but that hasn't happened yet so for now you can still use any of these three options. And yeah, I know CVS is ancient and crusty but it works and switching is a pain so we're working on it. And then you run build.sh. Now let's go through, this command line is a little bit long. There's some options here that are not the default. I picked the minimal set of options that I use personally because it's convenient for my development. Once you've checked out the source in a directory called source and you enter that directory, build.sh-o.dot.slash-obj, it says put all of the build products right there. So nothing will be written into your source tree, nothing will be written outside.dot.slash-obj. Go all goes into the one directory. If you make a mistake and you screw something up and you're not sure if you can recover from that state, it's very easy, you just delete that directory and start over. There are other ways to use build.sh-obj trees, you can have object directories inside the source tree scattered throughout, you can have it in slash users, slash obj is traditional but what I do is this and this is handy. Dash big U makes it an unprivileged build which really we should make the default because these days privileged builds are kind of silly but traditionally from the 80s, possibly even earlier, you would build as root, run your whole compiler tool chain as root in slash users slash source and build products in slash users slash obj but these days it is build unprivileged and you don't have to worry about root running, GCC or Clang or something. It's just a little bit sketchy from how heavy weight and complicated those programs are using anyway. Dash little U makes it an update build so if you make a mistake in some code you're editing, it'll pick up where it left off. Take some time for it to scan through all the directories but it'll pick up where it left off once it gets there so you don't have to recompile everything if you just change to one .c file. We can even speed it up more by doing builds in one sub directory at a time, I'll get to that later. Dash M alpha, this says build for the sleek modern deck alpha architecture. Now you can build any architecture you want, you can cross build that BSD for the Vax if you want or for the, I mean, not every architecture on the planet is supported. We haven't yet finished the PDP10 port but you can use build.sh list arch to see what the options are. There's a lot of options especially for ARM and MIPS. There's a bunch of different ABIs and variants and it's a kind of a mess for those but dash big n one changes the verbosity level to one so it'll just say compile foo.c and it won't show the whole command line but you can use dash n two for whole command lines or dash n three for more or dash n zero for very quiet builds. I find that dash n one is a good level of verbosity but tastes differ. You want to do parallel builds because everyone has multiple CPUs these days. If you don't have multiple CPUs then you're gonna be waiting a while. And then there's a target. You can specify several build targets. Release builds a whole net BSD release, the whole thing for one architecture and it will produce various things. It'll produce a cross compiler tool chain in object slash slash tool there. It'll produce distribution sets based on TGZ, comp.tgz, et cetera. It'll produce a desk to dir which is a staged installation of net BSD into a directory that's what the distribution sets are built from. It will produce a bunch of build trees for individual programs and libraries like find and that's all the .o files and the executable from find are stored in that one directory. If you screw something up with that then you need to start over. Just blow that one directory away. And it'll produce kernels in OBSIS arch and you can find a net BSD, a file called net BSD or netbc.gdb in that directory and that's a new kernel. You can just boot that kernel, move it to slash, maybe move the old one to slash o net BSD so that you can recover if you made a mistake but move it to slash and then you can boot it. Some other useful targets from build.sh help is always useful if you forget how to do anything. List arch, if you're gonna build just the cross compiler toolchain which is handy for some things you just want an ARM compiler that you can run. I mean you can use like godbolt.org if you're just playing around with seeing how compilers produce but it's handy to have a whole toolchain sometimes. Distribution produces just the user land, doesn't build any kernels. It can be handy if you just want to update your user land with some changes to some files. Sets build just distribution sets, modules, kernel modules, pretty straightforward. For kernels, if you just wanna do kernel development then build.sh tools to get a toolchain and then build.sh kernel equals generic and that works for most ports like X86 and alpha and fax and a bunch of others. For some of them, like 64-bit ARM and 64-bit RISC-5, the kernel is called Generic64 for administrative reasons that make it convenient. And in some ports like FBPPC that's short for Evaluation Board, PowerPC, these are ports that were originally created for just generic hodgepodge of evaluation boards that were made by various different makers. There's a lot of them 10, 20 years ago that all this kind of small embedded system so you don't wanna have a whole kernel for all possible ports and the kernel is somewhat smaller and it's tailored to that one evaluation board for embedded development kit. So you might find some of those if you're dealing with FBPPC or FB ARM or what have you. If you want to make a custom kernel config, you can either just put some customizations in generic.local, this is a file that the generic kernel config will automatically look at if it's there, if it's not there no problem but if it is there you can just have that file in your source tree and it will never have merge conflicts when you update from CVS or Git or Mercurial. Or you can just create a new kernel config, call it say debug, this is what I have in my debug kernel. I have a few more options in that but they wouldn't offer on the slides. And it's just to turn on some debug options so that I get some diagnostics if I, while I'm trying to do some driver debugging or something. Now if you wanna boot a kernel, let's say you built one for AR64. So you can take the AR64 image that build.sh release just produced and G unzip it and into a dist.image file. And you can make a, let's say this image is about one gigabyte or so, maybe a few hundred megabytes. Once you extract it, you can expand it with DD to be, let's have a pointer, cool. Spend it with DD to be a sparse image by adding a single byte after 10 gigabytes. And there's other ways to do that but this works. And then you can even make a sim link to the kernel that you just built in the case for booting QMU with a netpst kernel. You want the netpst.image file and this is very handy because you can then pass it on the command line to QMU and anytime you want to update the kernel, just execute QMU, control AX and start it up again and it will instantly boot the new kernel. You don't have to worry about copying it over like I mentioned the target disk mode even. You don't have to copy it over to the file system. And you can do that but you can also boot with kernel on the command line. And so this just specifies boot from the disk DK1 which will be discovered from the file disk and image. This makes sure that we have the machine type of just a generic virtualized ARM system with a reasonably modern CPU instruction set, not some ancient ARM v5 thing or something. And it pretends to have two cores, a gig of memory. You want to have the disk to boot from. You'll also want to have a Virudio RNG device. This is handy for entropy, especially if you want to do anything serious inside the VM like generating keys that you care about. And you'll want to have a, and there's lots of options for this, but if you want to have network access to talk to the host or something, then, so this is a complete command line for you can do some reasonable net PC development with. There's lots of other options. The QMU man page is huge. You can run through it, whatever, but this is, I just create a script called run.sh and put that in a directory with the disk image and then whenever I want to start it up, I just run that as h. There's some more options that I put in here, but you can play around with this stuff. You can also use V&D. What does it stand for? Virtual node disk. What does it stand for? Anyone remember? I don't know. Okay, so it's the virtual disk. And so this is just a virtual disk that exposes a file system as, sorry, exposes a regular file in your regular file system as a block device. You can mount file systems from that and so you can mount the root file system and update your kernel or update the user land or whatever. Be nice if you could also pass the kernel on the command line for X86. Right now we don't support that if somebody would like to volunteer to do that. That would be a very welcome starter project. You can also run GDB on a live kernel under QMU and it's real easy. You just pass the dash s argument to QMU. This is actually short for a longer argument. There's more options for that, but dash s is the default thing. And then in your kernel build tree where you have netpista.gdb that was back up in this directory here. Obsess, arch, the machine, compile and the kernel configuration name. So you'll find a netpista.gdb in there. Yes, what's up? Don't you need one? Yes, I guess I should have mentioned that. Yeah, so if you are doing it, right. Yeah, so, right, if you're doing a cross build in the sense that you're like, not just in a way that netpista is always a cross build, if the host is an x86 laptop, like I think that here, and you're trying to run GDB on an ART64, ARM64 guest, then you'll need to pass an extra argument. So I hope this is big enough for the video audience to see or people in the back to see. So yeah, so you can use build.sh-v make cross GDB. This will, in addition to the cross tool chain, build a cross GDB so that you can then use your toolder and you will have a GDB executable that you can use for the purpose of that, for the purpose of running a live kernel under GDB in QMU. And it's, so yeah, once you have, either if you have the host GDB if it's x86 to x86 or armed ARM or whatever, or if you have a cross GDB from build.sh, then you can use that to debug the live kernel under QMU and it's, you can like, you can examine data structures, you can look at what's going on in kernel memory. I'm not sure if you can set break points. You may be able to, but I don't know, you can mess around with it. You can even, oh yeah, so in QMU-S is short for a dash GDB, cc124, that's not right. There should be two columns there. Sorry, that should be two columns, not one. But there's other options you can use for connecting QMU to GDB. In fact, you don't even need to use QMU, you can run GDB on a live kernel that you're running right now. I could do this on my laptop right now if the system is not running at elevated secure level. So you can just, if you have the net-dust GDB file that you use to boot the machine, then you can run GDB on it and you can look at the data structures, changing in front of your eyes with the GDB prompt. This question. You're talking about the net-psd-user-mode port or are you talking about Rump kernels? So the question was about running net-psd in user mode under GDB. So are you talking about like a Rump kernels? Yeah, so I'll get to this later. There are ways to run kernel code in user processes and run those under GDB so you're not, running GDB on a live kernel, it can be dangerous if you write to some memory then you can really screw stuff up. You need privileges, of course, to do this. But there's also ways to run kernel components in user-land processes and run those under GDB without having to interfere with your live kernel. This is mostly useful for diagnosing if the system is hanging and you don't even know why it's hanging but you wanna do some retrospective diagnostics as part of it is hung or something. I don't do this very often. Just kind of need that you can do it. But I do use net-psd under GDI in user-land processes very often and I'll get to that later in the talk. Now if you get a kernel crash dump, if your kernel crashes and it saves a core to a dumps device, you can also use GDB to inspect memory in the core dump. There's also another command called crash. Crash knows a little more about net-psd specific things. It can sometimes handle stack traces that GDB doesn't understand. Obviously we'd like to make GDB work better for all of those stack traces but crash sometimes is easier to do that. Crash is just a version of the net-psd kernel debugger that runs in user-land. You can enter it without pausing everything. You can also just hit the control on escape or I should have mentioned CN magic. So you can also set a sys-cuddle knob hw.cn magic and when on a serial console, if break on a serial console doesn't work, whenever you type plus, plus, plus, plus, plus, as I set it to here or as you can set it to anything, set it to hello world, then the kernel will trap into a debugger and you can interactively use the GDB, the kernel debugger. But if you have a crash dump, from the kernel debugger, you can always just continue, hit C and then the machine will proceed as before. But if you have a crash dump, you can use crash to get some information like get it a stack trace, look at the list of processes that are running. There's a bunch of commands. You can see the DDB man page for various commands for inspecting it. So when you're doing driver development and something goes horribly awry, then these are tools that you'll use all the time. Sometimes you just want to see the D-message, the messages that the kernel has printed, including like if a crash to the stack trace and D-message can do that from a core dump. Now, sometimes we find that on some machines, for some reason, core dumps don't work and it's really frustrating when that happens after a weird problem we don't know about has occurred and now the core dump is useless or there's no core dump. So you can use these syscotal knobs, the crash me tool to force a panic to pretend that something had gone wrong and see if a crash dump works. And so there's a few different ways to force some kind of crash. You can just call it panic function or you can, similar to panic, I call it panic function enter ddb directly. You can recursively lock a mutex, which is forbidden and there's some logic to detect that. You can enter an infinite loop with interrupts blocked and we have some infrastructure in place now for each CPU to occasionally look to make sure some other CPU is making progress. And if you enter an infinite loop with interrupts blocked that doesn't work, so the system will panic and say, hey, this one CPU is stuck, doesn't do anything and or you can do something like launch go lang, a well-known test tweet for the NetBSD kernel. No, go actually works pretty well in NetBSD. It's just, they found a lot of bugs, which were by doing very weird things that nobody ever thought of even trying before and turns out, oh, that path has a weird bug in it. One thing that you'll wanna do on, let's say a new architecture as you're bringing it up or if you're just generally making changes or it's a new machine or something, you wanna run some automatic tests. So you just go to CD user tests and do ATF run and pipe it to ATF report and that will produce a big old report of several thousand automatic tests. Depending on the machine, it can take hours to, many hours. But you can also, usually it's run privileged in a VM or something and this will mess with the system configuration to test lots of paths in the kernel as well as user land. You can also run it unprivileged and that will just skip some of the tests that are dangerous that would require privileges. And I also often take the output and tee it to a file so that I can scroll through it later. The ATF run.out file is very verbose but it has nice things like automatic stack traces of test programs that crashed and the ATF report.out file has just a summary of all the test results and like a one-liner of each failure that says, test program crash with SigSegV or test program failed with a one-liner message and you go back to ATF run.out and you can look at the details. There's also nice HTML output that you can find at, capitalize that correctly. You can find some nice HTML output. With ATF you can build it yourself but we also do automatic runs and if you just look at relengedinvest.org and look at the automatic test suite runs you'll find some examples of the ATF test suite output. See what it looks like. Now if you're doing development on that BSD this doesn't work in cross compilation but like when I'm doing development on my ThinkPad I'm running that BSD here. This whole presentation is all done with that BSD and this is my development machine. This is where I do all my work, all my build, most of my builds and a lot of my testing. I run a current kernel that I build an update from time to time and I have a destator. I use build.shrelease to build a net BSD user land or distribution to build user land and then once I have that it's a stage installation. The permissions are all unprivileged so they're all owned by my regular user but in spite of that you can still just cheroot into the destor and then make some device nodes and mount PTYFS and tempFS and then you can run ATF tests in a cheroot and so I have a net BSD nine user land running on this laptop but a current-ish kernel and whenever I'm doing development for say pull-ups to release branches to test security patches or feature back ports to net BSD eight, nine, 10. 10 is also imminent by the way. Release kernel should be coming any day now this month. This is actually imminent, not like real soon now but actually imminent. And so I've been doing a lot of work with, you know, we updated OpenSSL to OpenSSL three. Man, that was a big pain and I don't know if anyone has ever worked with OpenSSL or worked with OpenSSL major version changes but oh man, it is a big pain. It's a vaporized system. Yeah, yeah, well, I expect that the number of differences between them will make that also a big pain and it's just the whole thing is a big pain. No matter what you do, it's gonna be a pain. It's just a difference of what flavor of pain you like, I think. And so we did a lot of work in testing a large number of patches in these true environments for net BSD eight, nine, and 10 and I just did it all on this one laptop, didn't even have to use a VM to test it just true in and go ahead. Yeah. Useless. Oh, sorry, for the video audience, there was a question whether LibreSSL cross builds to Vax and the answer was no, Vax support was dropped and it doesn't cross build. So, evidently, this is a useless project. Can't cross build to Vax, you know, why even bother? No, right now, OpenSSL on net BSD, it does cross build to Vax and it mostly works even, although I discovered recently that the Edd 448 logic in OpenSSL appears to be miscompiled by GCC or something or there's some kind of bug in it and so it fails tasks but otherwise it seems to work pretty well and Edd 448 is pretty obscure, there's really no reason to use it these days. Now, let's suppose you want to develop, you know, make changes to one library at a time, then you can just CD into the source directory and there's this make wrapper. It's a, in the tool dir, the executable mbmake hyphen then machine arch like mbmake mb64 or mbmake, mbmake, you know, Vax or whatever. This is a special, it's just a script that sets a bunch of make variables so that it points to the right object directory, it points to the right tool chain, it points to the right everything for just building one component of an FPSD build at a time. So once you've gotten build.sh tools, you can start using this to build one sub directory at a time. And again, you can iterate this and it will install into the trute so you can test it straight from the trute. For libraries, it also works, you just update the dynamic libraries and bam, everything in the trute is using it now. Static libraries, you have to rebuild the dependent things. And the make files, there's one make file per program, per library, they're usually pretty short, there's a fairly nice language for saying, this is a library, it has the following source files, it is called this and it maybe has some special properties and that's it, that's it, that's all the make file. Very easy to read, you can get some more information in the share mkbsd.readme. If you wanna change an include file, like user include standard IO.h, then you also have to run a separate includes target. Oh yeah, I should mention depend all, automatically generates a file of cached dependencies so it notices that if you change a header file, then the program has to change and also build the program. Let's see, how much time do I have? I'm not actually sure when I should stop here. Top of the hour. Top of the hour? It is? Yeah, yeah, but how much time do I have? Like should I? Okay, okay, great, great. Okay, so let's, now that you have the tools to start hacking something, do something, let's look at what things look like in NetBSD. Like I said, this isn't gonna be a very code heavy talk, it's gonna give a high level overview of what some major things look like. So, device drivers. This device driver is also not the sort of scary black magic that the public perception often has of them. They're just programs that deal with cantankerous pieces of hardware and sometimes, the hard part is sometimes you have to do science in the hardware if you don't have the right data sheet, but the writing a device driver itself, the writing a code is fairly straightforward. Now there's two kinds of device drivers or things that you might call device drivers in NetBSD and other BSDs. There's auto conf drivers, which if you have a laptop like this ThinkPad, it's got a PCI bus. There's a bunch of devices that are attached to this PCI bus and it's just a standard for the interfacing between hardware like an ethernet driver or a USB controller to the CPU, so the CPU can talk to things. The auto configuration, this is not the GNU auto conf, we're not talking about software auto configuration, we're talking about hardware auto configuration. This is where the kernel will go and enumerate all of the devices it can find by looking at the PCI bus, by looking at the device tree on embedded systems, by looking at ACPI and see, okay, so what hardware do I have? Oh, I got a USB close controller, I got an HDMI port, I've got an ethernet thing and some of these will be exposed to software in user land, some of them will just provide hardware interfaces, we'll just provide some kernel interfaces like a power button, the kernel might wire up an interrupt handler for a power button so that it will execute the shutdown routine and there's no user land interface to that, but if there is a user land interface, if there is like a USB device like this FIDO key, then it might be exposed to a dev node, a slash dev node in the file system and that's just an interface between kernel and user land. It might not even be backed by physical hardware, there are virtual slash dev nodes, but these are the two things that are often meant by device driver and sometimes they correspond, sometimes this device will expose a UHID node in slash dev and there is a corresponding autoconf device for that, but sometimes they don't correspond, sometimes autoconf devices have no slash dev nodes, sometimes there are slash dev nodes that have no autoconf devices like slash dev slash tap virtual ethernet interface, there's no hardware behind that, it's just a software abstraction. So an autoconf driver is something that will be detected by device enumeration on the PCI bus or device tree and it comes in a few parts. There's a, you have Soft-C is the traditional name for the state that is used by a device driver. I have no idea what it stands for, maybe someone at this conference knows, but one suggestion is software copy of, all right, software copy of device registers. Do you need to find a new chair? Okay, and so there's, any software state you need to interact with a device, there might be, you might have a reference to the device T for software purposes, a lot of drivers do this, often it's just for printing a message that has with the device print app function if you wanna just give messages to the console about the device. You might have a mutex for software state, you might have a bunch of other stuff, anything that you need for interacting with the device goes into the Soft-C. And then you write this bit of annoying magic called the CF attached declaration. And well it started as CF attached decal and then someone added a new parameter, so it became CF attached decal two and then someone added another one so CF attached decal two new and now there's CF attached decal three new and at some point we'll switch to C99 designated initializers for this, but we haven't done that yet. So it's just write the name of the device, the name of the auto-conference device, you'll see things like food dev to zero, food dev one, food dev two in D message when you boot the machine. And you tell it how much space it needs for its private state and you give it three functions for device enumeration. And so the match function will take some arguments describing the potential device that this driver might have found and it passes through some bus arguments, some information about the PCI bus for instance if it's attached to PCI. And this match function will answer the question can this driver handle the device? So does it have to write PCI vendor and product IDs? It doesn't have the right device tree compatible string. And then there's an attach function that once match, and the match function returns a priority number in case you have like a driver for generic USB devices to expose to user land versus a driver for specifically serial consoles. And in principle they could both attach but the serial console one should have higher priority because it's specifically for serial consoles, not for general devices. And oh yes, well I'll get to this in a moment. Then once the OS has decided which driver is responsible for a device, it will call the attach function and that actually allocates resources for interacting with the device. It will allocate memory for buffers that it has to do and make it create mutexes, create threads, what have you. And then once the device goes away when you take a USB device and you yank it out that the detach function is called. Now detach functions are a little bit tricky and I gave a talk at last year of BSDCon about those. I don't know if this QR code is too small but if you wanted to take a look at the talk from last year, I went into some details about how detach functions are tricky and how to make them work reliably so that you can actually just take like a USB device and yank it and put another one in and even if you're using it heavily at the time, this machine has not crashed, I hope. Yeah, I forgot to do a live demo last year. I just, I was giving the talk and it just didn't occur to me to do a live demo. So now if you have a slash dev node that's like a character special, this is the interface between kernel and user LAN. So the auto-contract device is interface between hardware and kernel. Character special is interface between kernel and user LAN. And so you just give a collection of functions that will be called when a process opens the device node, when the process closes the file script when a process says read or write or whatever. And there's some other administrative, if you want to create a new one, you have to create a major number and perhaps minor numbers for this to determine, the number, the major number is what actually determines which logic it's used when user LAN talks to a kernel. Traditional slash dev nodes correspond to physical hardware devices. And so if you want to open a device, you're actually opening access to a floppy drive or an ATA device or something or a SCSI device. And so the state is just global state for the driver. They're also cloning devices where every open has its own personal state. And this is a little more flexible for some things. So like the audio mixer interface, which automatically mixes audio from multiple streams and then sends it out to physical hardware device. That is a cloning device. Some others like the VHCI, USP virtual host control interface used for debugging and for testing the USB stack. And with a cloning device, you just specify an open function and then a file ops set. It's a set of functions that's similar to Cdev and the previous slide, but you get a little more flexibility and per open state rather than per device state. And there's a special structure to how you have to do the open function. You can look at this in the slides later if you're interested. Quick rundown of bus space. So hardware devices, the way that they're usually exposed is that you get a data sheet and the data sheet says, okay, here are the device registers. Register is sort of like, device registers are sort of like a location in memory. It's like, you know, if you have allocating an array of ints of UN32, that's a bunch of spaces for four bytes when you write an integer and when you read the same address back, you get the same thing you wrote. Well, device registers are just like that except that when you write something out, the CD tray ejects. Or when you read something back, then the CD tray ejects. So, you know, device registers are, they work just kind of like for locations in memory. You can write to them, you can read from them, but they do magic hardware stuff instead of just remembering what you wrote. And so when you have a driver, the bus will give you a bus space tag. There's a few different ways to get at device registers like in legacy X86 systems, you might have heard of IO ports. This is a, it's an address space. Addresses that you can get at, like Xerox 3F8 is the serial console in biosystems. And, but sometimes you'll have memory mapped IO that's where the CPU actually exposes it just like a memory address. But when you do a write instruction or read instruction or load or store instruction, it, you know, CD tray ejects instead of writing something to memory. So you, if you wanna use any device registers in NetBSD, you have to, you start with the bus space tag and an address that is given to you by, in a, like typically given in the AUX argument to a match and attach functions in autocons. So PCI will, you know, will give you a couple of bus space tags for memory mapped IO and IO ports on some machines. And you map it into a bus space handle and a handle represents a small contiguous chunk of a, you know, a window into the space of device registers. And then you can just read and write registers. The foo code will be some define that you get from data sheets and number, just a number of a register relative to the start of a block. And then you can write, you know, write it back, twiddle some bits, whatever, it's just, you know, you get a four byte thing from bus space read four and write it back. There's a bunch of other bus space stuff. You can look at the man page, it's big, but the concept is generally very simple. It's just you get windows into a little address space for device registers that match what you see in the data sheet. If you have a data sheet, which is nice. Sometimes you don't have a data sheet, which is not nice. So often you'll have a file full of registered definitions that just says, okay, this is from the Rock Ship Crypto Engine Driver. So there's a bunch of registers in the register block. Maybe it'll be like, you know, one page long or something. It's at 0x200 is the TRNG control register. And then we have some definitions for bits inside it. The bit macro creates a bit field with just that one bit set. Bits plural macro creates a bit field with all the bits between those two numbers set. And you can assemble a new control by shifting the value 100 into the field of the cycle, into the cycles field. And that will put it in here. Well, in this case, it doesn't do anything because it starts at bit zero. But you can say handy macros for putting bits into fields inside a 32 bit word wherever you want. So you just transcribe the data sheet. You'll see a table of registers and table of fields in the register. You just write down what the bits are. And then when you want to use, put something right into a register, you use shift in. When you want to pull something out of register you read, you shift out. It's pretty handy. There's another thing, bus DMA for, okay, do I have time for questions after this? Or is my, okay, all right. Real quick then, just, this is how you say, someone wants to write a bunch of data over the network. So they put it into a memory that they pass, in user land they pass to write, the write system call or send to. Bus DMA is the API that the driver will use to make that memory available to the ethernet driver. So you don't have to mem copy it in a CPU. And bus DMA abstracts a lot of details like getting the IOMU right, getting bounce buffers if they're address based restrictions and so on. Now someone asked, but I think you might not be here anymore, about almost done, about running kernel components in user land. So you can use this thing called Rump, which lets you run kernel code in user land processes, like the run threads, run device drivers, run file systems. We have extensive file system tasks that all run inside Rump. And so you can just use nbmake under the source slish Rump to build subdirectories and it will install into the truth. And you can run like the VFS tests in the truth with ATF and run it under gdb. And that way that's how I did a lot of file system debugging to fix like the rename system call, which is very difficult, involved hammer on the file system extensively and then in user land and then running gdb on the results when it crashed and seeing, oh, I forgot that case right there. And so that's it. So now you can all get hacking at BSD. And if anyone has questions. Thank you.