 Hello everyone. Welcome. Thank you for attending this talk. Today I'm gonna talk about a small but handy tool called syspatch. So quickly who am I? So my name is Antoine. I'm known as A.J. Kuto or A.J.A. at OpenBSD.org. I'm also a part of the GNOME Foundation and I mostly work on anything that helps me put OpenBSD into production. And for work I'm head of SRE at VankPrivée. So what are we gonna talk about today? Syspatch. Syspatch is both a tool and a build framework for patches. The main tool is basically a very small shell script, 220 line of codes. And then we have the build framework that's under a user source, the district syspatch, which is the usual source tree of your OpenBSD installation. That's mainly BSD make files that replicate what the release build process does on OpenBSD as well as our port tree. During this talk I'm gonna talk about binary patch. That's not really accurate because we're not doing binary diffing but we're actually providing a new version of binaries, libraries, etc. It's just easier to refer to them as binary patches though. So first of all a little bit of history. Anyone who had to actually maintain OpenBSD boxing in production, more than a dozen of them, hit this horrible requirement is that you cannot easily keep your system up to date. What I call up to date is not upgrade from one release to another but just to keep up to date regarding security and reliability patches. So up until syspatch you had basically two solutions. Either you CVS up and then you have to recreate a complete release. Of course you have to do that to do everything manually. You have to use signify to verify that the actual stuff that you download is valid. So the two different things you could do is build a release from scratch, CVS up from stable or you keep the DSD, for example, 6.2 release and you manually patch it and you build only the small part that you want it to be patched. The big issue with that is that if, for example, you have a security issue in your LibCrypto then of course we rebuild it with the patch that was installed a shared libraries but also a static library. The problem with static libraries is that anything in the base system that is actually using this library will not be rebuilt. So basically you're like half patch really. So the only real way was to build a complete release from scratch and that's a huge pain because that means that if you want to update a group of machines then you have to do a really, really big upgrade which is like the same, basically the same thing that you would do from one upgrade to one release, sorry, to a release plus one. So it was really tedious to maintain boxing. A few years ago I was hired in a company called M-Tair. One of the company's job was to put OpenBSD machines into production, both servers and workstation. So we had a huge pool of OpenBSD machines to maintain all over the globe and of course using official update facility was really tedious. So we came up with a tool to provide binary patches for the OpenBSD base system. The tool was based on the package tool like when you have a new patch you just install the package. There was two issue with that. While it works relatively fine it really abuses the package tool framework. Also you will get mismatch binary, well mismatch signatures in your packages when you upgrade to a new release. Let's say for example you upgrade LibCrypto using a patch then the package tool framework will record the checksum of the binary. Of course when you later upgrade to the new release then the package will still be installed but of course the library will have changed. So there were all kinds of stuff like that that were not really nice. So we came up with a small wrapper script called OpenUp which is again a shell wrapper that we used to keep both the base system and the package up to date. It's a wrapper because it was doing some stuff behind the scenes to deal with checksum mismatch and stuff like that. So it worked nicely, it was a bit ugly and the main issue was that it was not official. So while there are a few OpenBSD developers hired by this company it's still not an OpenBSD provided way to stay up to date. So now situation today. So dispatch ended in OpenBSD on September 2016. It was the result of a hackathon in Cambridge and that's a joint work between Robert Nagy and myself. The first supported release was OpenBSD 6.1. So that is the previous release since we're at 6.2 today. So this tool only makes sense on official releases and kernel. You cannot use it if you're using like current or if you use it using stable. It's really meant to be an additional layer on top of the regular release. As of today we support AMD64, Arri 386 and ARM64. We only support the last release. So that means that as of today we don't support 6.1 anymore. Although it is still supported patches are still available that has CVS patches. Reason for that is lack of manpower. And also what's important is that this tool is really a very small and stupid tool. So we really only want to use it to be up-to-date regarding security and reliability. It's not a new release of great tool. And why shell script? Well basically because everything we had to basically implement this feature were already in the base system we just had to orchestrate them. So shell script. Before coming up with Syspatch we started looking a bit about what the others were doing. So in Linux most of the time everything is a package. So you don't really have to deal with that. You can run your GNF grade or APT grade or whatever. As I said our intern was completely different. We don't want to be a release of great tool. We want to be a maintenance tool. Also sometimes rollbacks are tricky when everything is a package. So then we looked at FreeBSD. To be honest we've been jealous about FreeBSD update for quite a long time. We looked into implementing it but that was quite complicated. Much too clever basically for us or at least for me. Also it's very slow. It's horribly slow. And it's also a tool that will basically allow you to upgrade to a newer release. And so that's something we didn't want to do. And that's also a part of the complication of the tool. And regarding NetBSD and Drive and Fly as far as I know it's source only. So Syspatch. As I said Syspatch is a really really simple tool. You have a four option that we will detail later. But basically if you run it without any argument it will install all the missing patches automatically. You don't have to pass it any option or anything. So it's just a small utility that will basically fetch and install binary patches. And rollback of course. Within the script we implemented a small unprivileged function. This function is used to lose privileges while doing some sensitive stuff like going online or like doing cryptographic verification and stuff like that. So it's really stupid. We're using our own SU to do that. We drop privileges to a user called underscore Syspatch. And then we do the dangerous stuff. That's exactly what the OpenBSD installed it as. And we wanted to mimic that behavior. We also, well you may be able, you may want to pass an argument, sorry, to this function. Like you may want to, as an unprivileged user, go online. But of course you want to have the right to sometimes download something and write it somewhere. So with the minus F function the privilege, the losing of the privilege will happen after that the file has been created. So as I said everything that's network related or crypto related will be done using this function. Which is basically what we do in Syspatch to fetch the SHA-256 signature file. So let's look a bit about the option. So we have a Syspatch minus C that's basically check. So check what the, what available patches are around. So what it does, it does, it will fetch the signature file. We're using the command that was in the previous slide. It will verify it using signify. And then it will press and compare what's inside that signature file to the installed patch on the system. And from that it will build a list and display what's missing. Nice thing to note is it's automatically run bar or RCA system once you upgrade to a new release. So that means that you will see in your gmessage output if you have missing patches and it will send a mail to a route as well. We have the minus L option. So that's to list the actual installed patch. It's a really stupid function. It will just look into a specified directory. That's Varsyspatch and see if there's anything there, if there are. That means that we have patches that are installed on the system. And in this case it will list them. Now we have Syspatch without any option. As I mentioned, that's what will install all the missing patches on your machine. So it does a few check before. It removes anything that doesn't match the actual release and that's stored under Varsyspatch. The hierarchy there contained the actual CVS diffpatch so you can know exactly what the patch does. And it also contains your rollback patch. If something bad happened on the machine, you can easily rollback using that. But there is no garbage collection between releases. For example, you have 10 patches installed on your machine. You upgrade to a new release. Then this directory is still full of the old patch and rollback files. So the first thing that Syspatch does is remove all the non-matched stuff there. And then it will compare your installed patches against the available one using pretty much the same function that used to list and check for patches. And then it will loop during the un-proof download and verification of the removal torbol that are on our mirror server. It will do a few checks like make sure that we're writing on the local file system, not on NFS that we have enough space that the file system is not read-only, stuff like that. And from there, we pick the actual list of files that we're going to install from the target set. And then we create a rollback torbol using the actual file that are on the system. And then once we have done that, we save, install the patches that are contained, the files that are contained in our patch. We're using install minus capital S, which is the safe install that unlinks the existing target before installing a new file. And then at the end of the run, if a new kernel was installed, that is, if it was a kernel patch, then we run reorder kernel. Reorder kernel is what we call call on OpenBSD. So it's pretty new feature that I think it landed like six, eight months ago. It means kernel address randomized link. And it's basically a random random randomization feature that we added to our kernel. It shuffles the object linking order. And it does that at every reboot. So like each kernel that you guys run on OpenBSD are completely different one from one another. So like the internal delta between functions inside a kernel on not where one would expect them to be, which makes some particular case of tags way more difficult. It's different from KSLR, which is what I think they're doing on Linux, but it has a similar goal. So basically, we don't need to load the kernel in a random location because the kernel itself is randomized. So that that's the main difference between call and KSLR. We have the minus r option, which will roll back the last installed patch. It's important to note that you cannot choose which patch you want to install. All patches are cumulative. If you want to roll back a patch, you have to roll back the entire sequence of patch that were before this one. So that's why you can only always roll back the last into an installed one. So what it does basically does the same kind of checks as we do when installing patches. And then it will get the rollback table from the versus patch directory and extract it. And then, of course, remove the old dispatch directory. And then we relink the kernel if needed. And we have the minus capital R, which is basically remove all the patches. We had a few use case in the past for that. So that was it was implemented. And of course, as anything in syspatch, it stops at the first error. So that's how to basically handle the patches. So let's have a look at how the process and the building of patches works. So first of all, someone notice an issue with the code. So they will tell the developer at your code sucks or whatever has a security issue in it. So we create a fix, we get, of course, validation and review. It is committed to current and then boom, we'll backport it to stable and creates the initial errata file. TJ will then review the patch, write the W changes on our website, and we create an announcement. Then Theo will sign the batch. Only a handful of people are actually able to choose to sign the patch file. Then Robert will build the patch using the assigned pipe file. And of course, then we test it and then make it available on our mirror. That's roughly our process for dealing with errata, but it's not set in stone. Can vary from time to time. So that's actually on AMD 64 and I 36, those are actually the build machine for syspatch. So they're quite small. They are built on machines that don't have any access to the internet. And we for each patch, we do a complete rebuild of the release. I will go into that a bit later. There are different types of patches. We have kernel. We have user land in which case we build an entire release or we have Xenocara or Xorg implementation and in which case we are rebuild the complete X.org. Of course, all the patches are installed on the build machine. And all the the result of building a patched release will be put into a fake route directory. The patch are created by basically diffing our current fake route with the previous patched one. And that's how we know what's changed between the release and the patched one. Yes, USB. And yes, and the initial fake route directory is just the extracted official release. So we have a make file for that. syspatch.mk. I won't go too much into the details. Interesting. Basically, we stripped a bit what the OpenBSD release process does as well as some part of the port stream and we assemble them into this file. So we fetch and verify the patch with signify. We create a fake route using the no perm option because we don't build as route. I mean, we are to build as route, but we are losing privileges quite early. So when we install file, we need to be able to install them on by route, et cetera. So we're using an open mount option for that. And then we diff the current build fake route against the former one and create the syspatch storeable. So the main issue we had when we first started implementing syspatch is that on OpenBSD we don't have reproducible builds. There's no effort whatsoever into that right now. So we didn't look for reproducible build, but we looked for what are called deterministic builds. That is that we know that if we build this release on one machine and that release on another machine, we will end up with at least the same set of build options. First issue we had was with the R cover. We had to implement the D flag, the deterministic flag to prevent some randomness in the archives. For example, we hard code UID, GID, and M time to zero and the mode to 644. Otherwise it could be completely random and we had difference in each and every archives. So that was easy to fix. There was a few requirements here and there to use that new options, but that wasn't as a fix. Then there was the timestamp of the actual archive, which were obviously different for each build. The diff.sh script that I mentioned is actually using cmp for comparison of the files and we can actually use a 34 bytes offset to skip the timestamp index. We were actually quite happy to see that it was so easily possible. So this problem was also very easy to fix. Besides static archives, we also had a shared object. On OpenBSD we rebuilt all the shared objects in a completely random order. So of course you can imagine that when you're diffing one release with another one, again everything is different. So what we did is that we basically told syspatch to use redlf on the shared object of the previous fake root, the previous build, get the linking order from there, and then apply it exactly in the same order when we build a release with a patched release. So that was a bit tricky to get, but in the end it actually worked fine. In the middle of implementing syspatch, we moved from GCC to clang, at least on md64, i36, and ARM64. And we hit a few bugs that were clang-related. Like the assembler does not properly set the .file directive when compiling assembly files. So it was missing from the object, which basically prevented redlf to see them. And if they can't see them, then we can't have the link order. So there was, I don't think it has been fixed upstream, but there is a bug report about it. We also build most of our stuff with minus g. And sometimes it's not 100% reproducible. The location list section changes. So this particular issue is not fixed yet. What you end up basically is that you have a lot of false positive, that you have to manually edit in the PList of the syspatch. It's still under investigation. We have no idea where this actual non-deterministic thing is coming from. Then we had the funny stuff with Pearl. The configure test for long double implementation details probes the content of long double length, and that gets stored in the config module. But some of these bytes are currently pretty much random and vary between builds. Because of their long only 80-bit and the remaining bytes are uninitialized. So again, it was not always the case. It was completely random, and it was a hard one to track down. But finally we saw that it was fixed in Pearl a few months before we started looking into it, so we just backported the patches. There was also the case of manual pages created by Pearl. For that we didn't come up with any solution yet. The diff.sh script users said to remove the timestamp header. Then we had a few funny things like time and date that were in several pieces of code within the OpenBSD source tree, like to display the build date or whatever. So this one was easy. We just removed it from the code. Well, I mean, what use does it have? And also something that was really, really, really well something that was really unexpected is that patches are unbuilt on the release machines. The release are actually built by Theo. The patches are built by Robert. So they're not built on the same machines, obviously. And on OpenBSD, we don't have any automated garbage collection of old files. Like, for example, if we remove the header, then nothing will automatically remove it from the system. You have to do that manually. And we realized that actually Theo's machine had a lot of header that we didn't have, that weren't cleaned. I mean, basically, that doesn't introduce any issue. But in this case, for Pearl, Pearl actually picked up a different option because the header was there. So of course, the final binary was completely different. And of course, that specific issue is only relevant for the first dispatch, since for the second one, we compare to our own built release. So what do we want to do in the future? So far, the tool is very nice. It works really nicely. It's super simple. What we're going to do is support more architectures, obviously. I would really like if we were, if we would be able to support the current release and the previous one. So at least you have patches for an entire year. That issue is, again, not technical. It's just that we don't have the manpower to do that. And something I'm hopeful as well is to pave the role for stable packages. We do have stable packages as part of the port tree, but they're not built. So it's kind of a pain to actually maintain these. And it would be also a bit nice to have a more verbal output when listing, syspatch, like listing installed one or the available one. Because the name or a bit tricky doesn't always tell you where the actual patch is within the source tree and what it does. I mean, 002 underscore libssl. Okay, you know that's something under libssl, but you have no idea what the patch fixes. So it would be nice to have that. And something else as well is that currently when we install kernel patches, a kernel patch is not a new kernel. It's just the new object file that we're modified. So that means that we can ship pretty small patches. And your kernel basically will be recreated anyway by the call pre-order kernel mechanism. So that's why we only ship the different object file. The problem we have is that on OpenBSD we have a different kernel for SMP machine and single processor machines. So that works perfectly fine. But if you want a VM, for example, you install it on a single processor of machines, you patch it, and then you want to upgrade your VM to dual set CPU for CPU or 8 CPU. Then syspatch, then the next patch syspatch will get a kernel patch that will try to build against a MP machine because you are now an MP machine. But your installer, we only have installed the single processor object for kernel linking and boom. So yeah, so I hope that one day we can have one slash BSD for both MP and SP machine. That's pretty much it. Off topic, we are hiring. So if you're looking into development, operations, whatever, come talk to me. We're hiring all kinds of different profiles. Any questions? Yes? Yes. You can't pick one or the other. That's by design. We thought a lot about it, but it's just way easier to do it this way. Yes. Sorry? No. We know how to do that. I would very much like to implement it, but I'm not sure how to integrate it within OpenBSD itself yet. There is a solution for it. I would like it personally, yes. People put syspatch in a Chrome for automatic? Whatever, man. I mean, it's your box. I don't know. It's not a question I can answer. I mean, it really depends on your environment. I mean, if usually patches are really well tested, of course, we can screw something up. I don't see what I do like is having syspatch check in Chrome, but apply it. Most of the time you have to do something after you apply your patch. Like you have, I don't know, you have a libcrypto or whatever. You have to restart your daemon using your library anyway. If you try to restart syspatch, oh, if you have a problem. The question is what happens if during the run of syspatch you lose the network? If anything happens at the moment of installation of the patch, if any error occurs, then it will stop and roll back. So if you lose the network, it's not installing anything. So it will just hang or whatever. Then you can control C and there is a trap handler that will handle everything. So it's pretty safe in that regard. Of course, if you unplug your machines, then... Okay, I think we're done. Thank you very much for your attention.