 Okay, so hi, I'm Nia, I'm on the Board of Directors of the NetBSD Foundation. I maintain programs in the base OS, also in package source for the past four years. I do embedded and desktop stuff, particularly audio production. And in this talk, I'm going to tell the story of NetBSD 10. I'm going to go into the gritty details of release engineering, the wins and the pitfalls, and some of the technical and community side of making what is going to be one of the biggest releases that we've made in recent times actually happen. And this lovely diagram here in ASCII shows how NetBSD releases are supported. So what you can see is that we can support two stable branches at a time which get continuous updates. For example, NetBSD 8 and NetBSD 9. When NetBSD 10 is released, NetBSD 10 and NetBSD 9 become the supported branches and NetBSD 8 stops being supported. The so-called current branch is changing constantly and might include breaking changes. A new stable branch is made when the current branch is deemed ready by the release engineering team. Changes are backported from current to the stable branches when they're deemed non-intrusive enough and don't break compatibility. Occasionally, we make a release which is a fixed snapshot of a stable branch. Releases don't get updates so you should pay close attention to security advisories and figure out whether it's necessary for you to replace a component update to a stable snapshot or whether you can take other countermeasures, which is often possible. This next diagram shows some of the major tools we use for ensuring that NetBSD branches are stable. Build.sh is the main tool we use for building NetBSD from the source code. It supports easy cross-compilation across many platforms and CPU architectures. For example, you can cross-compile NetBSD for MIPS or ARM on, say, a X86 machine running macOS or a NetBSD machine running something else entirely. In tandem with the builds, we use lots of static checks which cause immediate failures if the source code contains anything problematic. Over 20 years now, I think Linux has just recently adopted some strict errors as warnings functionality in the last couple of years. A Python script called Anita is a really important part of release engineering. It helps install NetBSD automatically in virtual machines for over nine CPU architectures. This runs separately to test some real hardware, which might be slow and awkward. A lot of people like to complain that we rely on cross-compilation and virtualization too much for our test suite, but we're doing lots of tests on real hardware, including for some of the less common CPU architectures. Inside the virtual or physical machine, the NetBSD suite containing over 10,000 tests run. Through Rump, it runs various parts of the kernel in user space, making tests very reliable and allowing us to simulate real hardware setups. For example, Rump allows various network components to be virtualized, and that basically prevents the test setup from breaking and allows monitoring it safely. We're also using various sanitizers which detect common programming errors in the kernel at runtime and feed various fuzzing tools, which feed randomized data into the kernel to detect bugs. By now, it should be very clear that this is state of the art, and very much needed for our small team to maintain a very large code base that's over 30 years old now. Late 2019 is when NetBSD 10 starts to take form as something really special. When the first release from the NetBSD 9 branch was being stabilized, Duran starts making frantic commits that drastically improve the performance of the NetBSD kernel, especially on multicore systems and those with a mixture of low power and high power CPUs like modern Intel and ARM chips. So here is some benchmarking that was done two years ago now, nearly two years, and all we can really say is, wow, about the performance improvements that have happened since the release of NetBSD 9. Yeah. Just amazing statistics like build times being completely halved. But it pretty immediately becomes clear that some of Andrew's changes are causing instability. Our tools instantly detect new failure cases in the kernel, and Andrew now has to dedicate his time to ironing out and fixing them. So many bugs were found by fuzzing the kernel. Many bugs were just found by users while they were testing the current kernel. So what have we really learned from this? We learned very quickly that user participation is critical for testing the kernel. A lot of kernel errors are detected at runtime during operation, so it's good that we can get lots of detailed reports from our users about our error checks failing. Kassert, which is the kernel assertion tool, it allows the kernel itself to indicate its state is wrong or confused and report a message useful for developers. It was really critical for ironing out some of the bugs that were found during the optimization stage in late 2019 and early 2020. In NetBSD 10, we're removing an unprecedented amount of code for old wired network technologies, which were replaced by gigabit ethernet. Some are still useful for the retro computing community and were kept. NetBSD is a group of people with a range of interests, and it's important that we keep everyone relatively happy and avoid major conflicts. So this was done really well on a social level. Just for illustration purposes, this is some of the older networking cards that support was removed for. This is FDDDI, if you've never seen one before. The port looks very strange, and this is an IBM token ring connector, which support was also removed for. And really this was done because in many cases this hardware just no longer existed in the world. No one was using networks with this type of cabling. So the results of the great driver cleanup is that many drivers were removed entirely, and it went really well compared to previous feature removal proposals. And it's allowed us to, in general, do lots of widely sweeping changes to the networking stack to improve performance and just to clean up the APIs internal to the kernel and make the kernel a lot cleaner. So as we move through 2020, it becomes pretty clear to us all that NetBSD 10 is going to be an amazing release, but we're still really far from being able to call it stable. Our release engineering and testing tools make that much obvious. And in summer 2020, the randomness saga also begins. A lot of what I'm going to discuss here is covered in Taylor Campbell's EuroBSD con talk from last year, so I'm not going to focus on it too much. Please listen to that later. Fundamental to NetBSD is the fact that we support lots of old, small and quite low-end hardware and some really new and fancy hardware. Some of this hardware is very suitable for modern internet security, but some isn't, specifically really old and deeply embedded hardware. My lack any good sources of random data importance to internet security, but it's plentiful on large modern x86 boxes. We need to provide a good user experience in both cases. It took more than a year and many flame wars in the community. My hands are not clean of this for us to figure out how we wanted things to look. POSIX did make some decisions on our behalf, which simplified things or alternatively made things more complex. And there were a lot of problems that weren't easily foreseen. Like there were some problems with Python due to it. Randomly deciding it needed critical entropy early in the startup process, which prevented many applications from running on hardware that didn't have a good source of entropy and required manual user intervention to stop Python from hanging. So there's 2020 in summary. We got some new security features for 64-bit ARM in the kernel. We got pointer authentication and branch target identification, which are amazing for avoiding some particularly nasty exploits. We got massive improvements to support for the Xen hypervisor. So we got really fast multiprocess networking for Xen and some support for some new fancy virtualization modes. We got boot load improvements for our EFI ARM boot loader. So more configuration options, the ability to load kernel modules in the ARM boot loader, and Raspberry Pi for support if providing that you flash the EDK2 EFI firmware on your Raspberry Pi because it's been broadly decided by the ARM team that we want to move forward with mainly booting the ARM ports with EFI now. And support compatibility with WireGuard, which caused some drama. The maintainer of the code base in Linux was not very happy that we had committed an implementation without consulting him first. And we also got some surprising improvements on the retro-computing side. We got support for multiprocessing on some deck alpha machines. And we moved the location of the default database of installed packages, which caused some headaches and delayed some stuff in the package source world. It's another example of what theoretically would have been a quite straightforward change causing headaches later on and causing the release to be delayed. It's now 2021 and a few developers are starting to focus on some potential wide improvements to the NetBSD kernel's code quality. It should be pretty obvious to most experienced programmers that an API that's not simple to use will result in lots of code being copy and pasted badly. And this was the case for an API called config found in some device drivers. The obvious solution is to make config found easier to use for programmers, which was done, but what's more complicated is making sure it can't be misused even before the code is compiled. NetBSD9 had some of this for config found, and various developers objected to the simplification because it lost some of the error checking. A long discussion started before an agreeable compromise was reached. And here's the code we eventually settled on, all the style of usage for config found. It resulted in some quite prettier code in the kernel, I think. By 2021 some of our users are starting to complain about our GPU support being limited to hardware that's a couple of years old by now. And they're quite lucky because a major graphics driver update was being worked on with our code being synced with Linux 5.6. But this comes with thousands of lines of code changing all at once. This seems to be an easy change by any measure. And some errors are particularly difficult to debug. So some problems are difficult to figure out. Even with our kernel error checking tools, trippy colors appearing on the screen is a good example of that. This doesn't give developers much to go by if there's just graphical corruption happening with no useful debug information. A lot of users were interested in testing the update, but didn't have the knowledge required to report problems well, which is a really difficult problem to navigate in open source in general, I think, getting users more involved in the testing process, getting users more actively involved in the bug reporting process. Some users were confused by our priorities because from the perspective of a developer, you're updating existing working code and fixing bugs there before adding any new hardware support. So they're like, oh, you're updating the code, so my fancy new GPU should immediately work, right? No, we're focusing on updating the older drivers first and then we'll do the new ones. The kernel assertion tools were very useful for getting diagnostic information from the kernel, but code ported from Linux doesn't necessarily use any equivalent. It was a problem in the testing phase because the NetBSD kernel hackers were used to having loads of diagnostics and loads of checks to make sure the kernel state was consistent, and the code ported from Linux wasn't really being as thorough. I'd love to hear about any ideas for making large kernel changes easier to test for inexperienced users if anyone has any good suggestions or ideas in that department. And this was our 2021 in summary support for loads of new ARM hardware, including the Apple M1, and I'll get to that. Here is a nice Wiki page we have on the state of Apple M1 support in current, and quite a lot of hardware is supported, even X11, so you can get graphic support on the M1 Mac Mini. The Odroid N++2, which is a smaller, cheaper ARM board that's still quite high performance, that's quite popular with NetBSD developers, as well as some lower-end ARM hardware. Lots of performance improvements specific to ARCH64 as well. Major improvements to Linux compatibility, loads of new system calls, support for the Linux compatibility layer on 64-bit ARM, so you can run 64-bit ARM Linux binaries on NetBSD. There's a support for our virtualized GPU driver, not GPU driver, it's just virtual machine drivers in general. So lots of enhancements for running in KVM, and stuff like Behive as well. And we added NVMM, of course, and support for native language support and GNU-REGAX extensions in the regular expressions layer, which the GNU extension part being turned off by default. Some GNU extensions that just happened to work were enhanced previously, no longer work unless explicitly enabled, which caused some compatibility issues, and we spent a little bit of time ironing that out in package source because some programs were just using GNU-REGAXes without realizing that they were using GNU-REGAXes. We did a large audit of the kernel APIs and found many bugs and fixed them. We reworked the system for password logins and disk encryption and adopted a new algorithm that dynamically scales with the hardware quite considerably based on the amount of memory that's in your system and the number of CPUs you have, which is, again, very good for NetBSD because we support such a diverse range of hardware platforms, everything from stuff from the mid-1980s to stuff released this year. We started improving support for the UDF file system, which UDF is basically a portable standardized file system that was originally made for rewriteable DVDs, but it's actually useful for a large range of applications where you need a file system that works across operating systems. NetBSD actually has some of the most advanced UDF support of any operating system out there. Of course, the GPU driver update happened, and we switched the default compiler to GCC10, which is likely going to be the final compiler that ends up in NetBSD10. And Grant grants the support for running code developed with new C++ standards, and this is quite important if we want to maintain a release for several years because one of the first things that tends to break is third-party software that's depending on very new C++ standards that the base compiler that we're shipping in a stable release doesn't necessarily support. By early 2022, it was pretty clear that we wanted to do a release soon because we'd managed to get all of these amazing features in. Unfortunately, when developers hear that there's going to be a release, they've got a tendency to rush their changes in, and this adds more instability to the equation and tends to delay the release even more. To make matters worse, we're running out of version numbers for current, which cause developers to group large changes together. It would be really nice to figure out an effective way to avoid this problem in the community. I think even the people who had rushed changes in had eventually agreed, this is kind of unsustainable and we need to find a better way of managing the project. By mid 2022, the release engineering team was very unhappy with the amount of delays and basically no more major changes were allowed. We ran into the final regression, which is currently delaying the NetBSD 10 release, but we had introduced support for FFS access control lists in the FFS file system and this was done in a way that was accidentally incompatible with NetBSD 9 and there were various file system corruption issues. So basically we have to introduce a new version of the FFS file system that fixes this compatibility issue and then finally we'll be able to branch NetBSD 10 and do a release. This is a couple of unfinished projects. We are currently working on a massive update to the Wi-Fi stack. We're synchronizing the code with FreeBSD to a large extent with many of our own local customizations as well and we need more volunteers to work on this project. There's a nice wiki page that basically lists the drivers where we have the hardware on the foundation side and there's somewhere we still need the hardware and various Wi-Fi drivers that we've managed to convert already and this is most likely going to make it into NetBSD 11, hopefully. We need to... It was determined that we wanted to deprecate the PF firewall for NetBSD 10 and basically decide that NPF is the one through firewall that every single NetBSD user should be using and basically there are still some obscure features that need to be ported over and it was decided to basically delay the problem. NPF is no longer compiled by default but users can still get it if they want to and of course by the time we get to NetBSD 11 we're going to have to update the GPU drivers again because by that point another year's worth of hardware will be released and people will be complaining and we're getting to the end of the story so this is our NetBSD 10 in summary. We have made leaps and bounds on performance and especially on multiprocessor systems this is going to be absolutely astonishing. To many extent this wasn't a release that was major for switching kernel components over to work in parallel with that had mostly been done in the past but this was more release about ironing out improving locking semantics and basically reworking kernel algorithms to better scale on systems with many, many CPUs. Lots of new and extended device drivers and lots of removed ones support for tons of new hardware. Lots of new user space programs which is not something that usually happens but at least one of them was written by me so I'm quite happy about that and significantly improved user space programs we had a project to work on the iNet daemon because it's kind of becoming fashionable again and people decided that it needs to be made more flexible and lots of work on the default shell as well and tons of work on the test suite which is a kind of underrated feature but it's very important if we want to basically ensure that NetBSD remains a very reliable platform and people managed to add over 2,000 new test cases which is amazing and various new tools for analyzing the kernel in order to better detect bugs before we make a release. Compatibility with wireguard VPNs is the obvious one faster, better disk encryption which means basically that disk encryption now takes advantage of hardware cryptographic acceleration that's found in many new CPUs and loads of enhancements to various little nice bits like the Linux compatibility layer and also something I haven't mentioned yet is that the user space file system layer received a massive overhaul to support newer user space file system APIs so we get nice things like NTFS support for free and I'd say the weight has definitely been worth it I think this is going to be one of our best releases ever and I'm extremely excited to be able to publish it so that's all. Do we have any questions in person or from the stream? Yes. So this is a question about the deprecating PF and the replacement that we've got lined up. So for a number of years now NetBSD has had free firewalls in the base system which is obviously not an ideal situation we have IPF from Darren Reed who was a NetBSD committer in the past and a lot of people are using IPF and I'd say it's still quite popular IPF is not going to be removed for probably another few years at least there's PF which is obviously ported from OpenBSD and hasn't had a maintainer for quite some time so that's the big target for removal because the code base is quite old and getting quite rotten and there is NPF which is the unique NetBSD special source basically and it's very well integrated in the kernel it has excellent support for scaling over multiple cores and it has an active maintainer so NPF is the future basically and we're very happy with it I'd say anything else? Yeah, the RISC-5 is the code existing in the kernel I'm not entirely sure of the current state of RISC-5 support we also have foundation funding available for people working on 64-bit power PC support but NetBSD is increasingly arm oriented a lot of us are kind of obsessed with ARM and this is where a lot of the development these days is focused we're big into the ARM stuff All good? Okay then Oh, thank you