 Yeah, I guess we can start. So good morning, everyone. Thanks for coming. I'm Martin Pitt. And the motto of our conference bears the question, how do people actually build a spaceship? If that was your average open source project from like five years ago, it would probably go like, hey, take this bare metal pipe, fill it up with fuel, stick a pallet on top, light it up, cross your fingers, and then let the bug reports come in. But for some reason, someone told these engineers that this wasn't susceptible. So spaceships are built by carefully designing each and every component, and then carefully testing every change to it with every conceivable corner case. And quite fortunately, in the open source world, we are now slowly moving towards that attitude as well. Because we finally realize, hey, writing all these tests and running them in CI is not a burden, but it actually makes our life easier and keeps our sanity and productivity. But in the special case of the Linux plumbing layer, how do you actually write tests for things like UDIS, U-Power, Network Manager, Modem Manager, LibMTP, or LibInput? These things talk to a lot of devices, which you usually don't have in your test VMs. And maybe not even the developer has that. And that's what I want to show you today, give you some tools and tricks how to approach that problem. So of course, I can't dive into the details, but the idea is to give you some initial taste and some ideas what's possible. And then you can RTFM later on when you need it. Of course, this thing sleeps. The first thing I want to cover is disks. Now, of course, we all know loop devices, and they are fine for the simple cases. But they really don't feel and smell like proper hard disks. Like they have lots of special cases, special names. You already run into trouble when you try partitioning them. You need special tools like KpartX. And you can't do fault injection. You can't do different device types. They don't scale very well. So a much more powerful tool is the kernel module called SCSI Debug. This is an actual in kernel SCSI driver, which is backed by a RAM disk. So we can just load that module, like for example, with a size of 100 megabyte. The default is 8 megabytes, so it may be a little small. And then we see in the kernel, like it detected something like an SDB, which is already quite familiar. And in GNOME disk, we can see it appears here as a normal disk. We can even put a partition table on it. Is that really my? Yes. Don't want to format my internal hard disk. I did that once, and yeah, not pretty. We can also put a partition on it, and we can mount it. Yada, yada. And we see, OK, it's mounted. So far so good. Now, the first cool thing that you can do is to simulate that a user just rips out the device. Like for example, if you have a USB stick or a CD-ROM drive or you just eject while it's still being mounted. So let's do that. And we, the kernel complains a little bit about something went wrong with this device. But here, we, for example, see that new disk that is duty and cleaned up the mount point. And this is broken repeatedly in the past. So now it really shouldn't anymore because this is running in CI. No, not this. So another cool thing is how to, like, let's add, like, 100 devices at once. I'm sorry. Add hosts equals 100. And I'm screwing this up, sorry. So learn to type. It's at host. OK. And now we have, like, a gazillion devices here. We see it here. And this was actually very useful in testing and fixing some bottlenecks in new disks. Like, there is server environments which have, like, gazillions of disks attached. And new disks previously just took ages to process them all. And yeah, try that with loop devices. Or what about adding a CD-ROM instead? Now we get an SR0, again, very familiar. And also, new disks agrees. It's a CD-ROM drive. It's pretty nice. Or what about fault injection? So the SCSI module has this magic ops, or what is it? Ops, yeah. Bitfield, where we can tweak, like, various things. And one of those is that you simulate a medium error at the sector hexadecimal1234. And if we try to write something on it, the right device, there we see it stops at, like, the decimal equivalent of that number. And we see the kernel reports. We've got a medium error. And again, this is probably not something that you want to force on your real hard disks when you test the behavior of your plumbing stack. So this is used in real life in the UDISC's integration test, so you can look at how it's being used there. And it's documented in the normal kernel documentation directory. OK, so much for block devices. Let's move to a different class, network. The simpler case is, of course, Ethernet. So very useful for testing network. Ethernets is the virtual kernel driver. So for those who don't know, you can create this out of thin air, and it gives you a symmetric pair of connected devices. And what you do is you consider one of these as the local end, which you want to put on a test with, for example, network manager. And the other is the remote end, which you set up with a known good tool like DNS mask, for example. So the commands are a little too laborious to type, so I put them into a shell script. Let's quickly go through. The first thing is just some bureaucracy. So network manager ignores these devices by default for good reasons. So we need to tell it to unignore our local end, so that it manages that, but not the remote end, of course. So this is the interesting bit that basically tells the kernel to create this device pair. And we use each 42 as the local end and remote 42 as the router. And then we just need to give our router an IP and start DNS mask on it so that we can do DHCP and prove that you can actually talk through these devices. You see here, or maybe it's a little small, that I'm not connected to anything right now. But let's run this and see what happens. And we see we, network manager, spinning a little bit. And we see that it's picked up some DHCP. We also see it on the server. We can ask connection information. And we are at a wired connection. And when we stop it, everything goes away. And we are once again disconnected. But now let's let it up a notch. And we want to simulate a Wi-Fi, because that's also a very common case. By the way, let's filter out some lock noise here. Otherwise it's a little too annoying. So Wi-Fi is, of course, a little more complicated, because there is a whole new concept around that. But there is another useful kernel module. It's called Mac 802.11 IHW Sim. Try to pronounce that 10 times. So how do we use that? In the beginning, that looks very similar, but now we need to do the opposite. Of course, network manager manages Wi-Fi devices by default. So now we need to tell it to not manage our remote end. But otherwise it's pretty similar. So if we load that module, we once again get a symmetric connected pair of two devices, WLAN 0 and 1. And we do the remote local thing again. So we set up our router, give it an IP. And this is the new bit here. So on our remote end, we need to set up a Wi-Fi router stack. And the standard tool on Linux for that is host APD. So this is by no means like a test-only thing. This is an actual thing which you often find in Linux-based hardware routers. So for here, we give it in hardware mode like 802.11 G, which is like middle ages. And we can give it an SSID and this super-secret passphrase. And this bit, once again, is familiar. We just put DHCP server on top of it so that we can prove that communication works. So let's run that. So network manager does some yada yada. Did we pick up something? Yes, we did. We have a zi Wi-Fi. So let's connect to that. Yeah, I didn't mistype it. And so it's spinning around a little bit here. Come on, second green dot. And we are connected. So once again, connection information. We picked up a DHCP address so communication works with our fake router. And yeah, we cleaned it up and everything goes away. So both of these, vEath and Wi-Fi, are in active use in the Debin and Ubuntu Auto Package tests for network manager and also Netplan. And the vEath part is also being used to test network D in its upstream test. And yeah, once again, communication is in the IP documentation is in the IP-linked main page for vEath and in the kernel documentation for the module. OK. So block and network devices are really rather special beasts in the Linux world because they have very complex behavior or are not even in the standard device format that most devices are. But most Linux devices and drivers are much, much simpler. So usually you have a character device in slash dev which you read from or write to, or sometimes send a few Ioc tools to. And for these more generic devices, a couple of years ago I wrote a tool called umockdev. And this is able to essentially mock your whole slash sys tree and slash dev and slash proc and redirects them to a temporary directory. And it does that with a preload library so that the whole thing needs no privileges at all. You can simulate devices and hardware with unit tests, with make check, and you need no root privileges or any kind of special kernel. It can also emulate you events by doing the same redirection trick on the netlink socket. So you can also satisfy you there, for example. At its heart, it's a C library. It also has GI bindings. So you can use it from Python, from JavaScript, or your favorite programming languages. But there's also CI tools which are very nice to interactively play around with it. So we don't need the root shell. So back to normal user. By default, if you run a project program under this wrapper, you get a completely empty slash sys of s3. So nothing in it. And of course, that's not very useful. So let's load something into it. The most convenient interactive form is to put those into text files. Just pretend we have a foo device. If you look into those, these look very similar to what UDEV-adm-info-xportdb gives you. Maybe some of you are familiar with this. So you have the device path, and the device name, and the e-colons are the UDEV properties, like the standard UDEV-name and subsystem, for example. But in addition, this also says e-fixing. In addition, this also says the assistive attributes. So a standard one is the device major and minor. And a non-standard one is this mode thing, which I just invented. So let's load this into our sandbox. Let's just run bash that we can mess around a little bit. If you now look at sys, we see that it, maybe we close this a little more space, we see that it creates a pretty standard surface layout for us. So we have our foo device here, who we have our magic mode attribute, and it also does all the standard buy class and bus sim links for you. And we also have this foo device in slash def. And you see, it looks pretty much like a standard thing what you expect. It's a character device with a correct major and minor. Of course, what it really is in reality is that it's just a simple PTS, which is reasonably close to a character device, so that the preload library doesn't have too much work to do in shaping it into the expected form. OK. So this already gives you the appearance of the device, so not yet the behavior. But this is already pretty useful. For example, you powered us there to simulate various kinds of empty or full batteries, UPSs, or combinations thereof. You can also simulate input devices, like lit switches or touch screens. And none of those are usually available in NVM. Of course, the latter can also be done by FEMU, but that, again, needs root privileges and kernel support. But again, let's go further. We not only want to emulate the appearance of a device in SysFest, but we also want to emulate and record the behavior of it. So for devices which are merely read from and written to, UMog Dev can basically record the entire conversation with that device node, including correct timing information, and put that into a file. And then in your test suite, you can load that back. And basically, UMog Dev will simulate the remote end and replay the whole thing. So this has been pretty useful to write tests for, for example, Modem Manager. You can take a real 3G stick, plug it in, record the conversation, and then put that either into a back report or your test suite so that the developer who does not have this device can reproduce the problem or, like, your test, make sure that it doesn't break. It also can record and replay Ioc tools from FDF devices. So you can simulate actual clicks and movements on Wacom devices, for example. And the third thing that it can currently do is use B-regressed blocks. So this is the thing that you get with MTP, like communication with your phone, or G-Photo, with your camera, and various other things. And I want to demonstrate that. So here I have my phone connected in MTP mode, I hope. Yeah, here it is, 3-3. And I first want to record all the SysFest properties and the UDF properties from it. Yeah, more. Def bus USB, what was it? 3-3. So if we look at this, we see it's pretty similar to the food at UMockDef example that I've shown earlier, except, of course, there's a whole bunch of more properties and attributes and everything that SysFest gives you. And it also contains all the parent devices of that thing so that your SysFest 3 is complete. And now we can also record the behavior from talking to the device. For example, we send this to temp sony by octal. And let's just run something simple, like MPT detect. All right. If you poke inside, you see this is basically a giant tree of all the iOctals that happen on the device. And it's a tree so that when you replay this, you aren't bound to the exact same order as the record. But you can actually repeat steps or do steps in a different order and something. And you can also add stuff for different libUSB devices, libUSB versions, where the form and shape of the iOctals might change, which happens fairly often, unfortunately. But so this .iOctal and the .umockDef files, these would be the two artifacts which you stick into a test read or put into a bug report, for example. So magic. I disconnect that thing. So now let's see what happens if we use these records. Let's first just run the static thing. This is already, yes, typing. So if you look at this, we see that our testbed contains my sony and the parent hub. But of course, if we try to run MTP detect on this, it will just freak out because there appears to be a device. But since there's no record of anything, it will just try to start to talk to the device and then say, whoa, there's nobody answering. But of course, we can also load our recorded iOctal trace. Equals temp sony. And here we see it's doing stuff. So as a little disclaimer, you see at the end, there seems to be a bug there. And indeed, this broke sometimes in the past year. So before that, this was working pretty much perfectly. You could record a conversation in Shotwell and import a couple of pictures. And then we play everything of that. But apparently something in LibUSB or MTP made the iOctal structure a little more flexible. And you might have cannot currently do this. So with the blame on myself, I need to track that down. But everything else, like FTF iOctals works perfectly and everything else too. Right. So that's basically my demo. I think we have a couple of minutes left for questions. Steph. Can also repeat that. Do you have a library of common devices or iOctal traces that people seem to use across the Debian project, not just one place that are reusable? I have a couple of examples in the upstream project. But there was no such thing like an official GitHub ePlace where people can collect those. So usually I use this to write integration tests for Debian's Shotwell package, for example. And I just stick the traces in there. So I'm not sure how useful that would be, because it's really nice for bug reports. But as I said, this ages relatively quickly. Like as soon as your LibUSB version changes and it changes the format of these iOctals, of course this can only do so much magic if the real device sends a block which is not in the trace, then the whole thing will just go up in flames. But yeah, no, there isn't. I just keep a couple of examples here. Any other question? Please. Is there a place where you record the version of LibUSB and stuff like that so you can at least detect why it goes in flames? I currently don't, because like the recording place doesn't actually know that. I mean, you basically provide this preload library which intercepts things like read, write, iOctal. But iOctaf record doesn't know what's running inside. Of course, you can add some metadata to the files. And I think that's possible. Like we do have one instance of that. We record the device name which we recorded. It doesn't need to be the same on replay, but sometimes it's useful to know that. So I am very happy to add that. I mean, it's trivial to do. It's just a simple text file. But yeah, maybe that would be useful indeed. Right now, you usually see this kind of stuff in the Git log when you see out there the iOctal traces to LibUSB version 1, 2, 3. OK, anything else? So a bunch of folks that I work with get kind of crazy with building various Gen2 builds for their laptop. Could you foresee any value in setting up a trace like from the very beginning of the boot of an individual's machine to record all of the kind of USB devices so that before they attempted to reboot into a new kernel or something, they could verify that or basically set up a test rig for their own machine to verify that everything would come up correctly? Well, I'm not sure about the continuous aspect of that. But what you can currently do is tell you Mockdiff record to just say, dump my entire system. And you will get, I don't know, a 100 kilobyte file. And this has each and every device in it. And then if you reboot, you can do the same thing and basically compare the thing. It's similar to what you would do with UDEV-Admin for export DB. But as I said, this is lacking the Sisyphus attributes. You can also get them from UDEV-Admin, of course, but with that, you have everything in one file and maybe it's easier to diff or something. So that works, yeah. And in fact, I've used for something like that. So I think we have time for one more quick question and then it's over to Steph who will tell you where to actually run all these tests in a sensible manner. Not okay, otherwise grab me the hallway and just one note, the talks page has a link where I put all my notes and the demo scripts and pointers to documentation for like the SCSI debug and everything else I talked about. So yeah, thanks for your attention and please go write tests. Thank you. Sorry.