 So up next we have How I Learned to Stop Worrying and Yank the USB by Taylor R. Campbell. Hi, I'm Taylor or Riastra. I work on NetBSD. I've been a NetBSD developer since early 2010s. And this is a talk about making it, getting rid of the PTSD that we've had from the prospect of pulling out a USB drive. I was hoping the system, oh no, is it going to crash? There's a lot of detail in some of the slides in this talk. There's some code samples. So I put this up on the web if you want to look at the URL and browse the slides at your own pace. And yeah, so I'll leave this up for a second for you to grab the URL if you like. So yeah, for a long time using NetBSD, I would try to make sure I very carefully ensured nothing is using the USB device first before I start to pull it out. And sometimes, you know, pull it out and just, oh crap, if I had a program running, oh no, is it going to be okay? And sometimes it will just wedge up and crash or do something horrible. And last year I decided that's stupid. There's no good reason for this to happen. This is just software bugs and set about to go and fix it and try to do it systematically to make it easy to fix it in a lot of drivers. So some background. The BSD device driver model has sort of two parts. There's autoconf devices that the kernel goes through and detects when you're booting up, goes through enumerates PCI buses, enumerates the ACPI firmware nodes, or device tree nodes in modern ARM systems, or things like that, open firmware. So it goes through and enumerates these devices that the drivers attach to and the drivers then have this state called a soft C, which is stored inside an autoconf instance. And there's three parts to an autoconf driver, three moon parts, match, attach, and detach. In free BSD, this is called probe and detach, but it's a similar idea. You have a function where the driver tells you, do I want to take care of this device? And if so, I'll take care of it, allocate the state, set up state and whatever. And then when it's done, when you want to remove the device, the kernel will call detach. This is purely a kernel side thing. This is not an interface to other parts of the system. This is just how the kernel organizes detecting devices and connecting device drivers to them. On the other side, there's user-ended interfaces, slash dev nodes. So if you look in ls slash dev, you'll find a large number of files. Well, on that BSD, you can open these systems, a large number of files. On free BSD, there's devFS, so it's limited to the ones that you have drivers for. But a similar idea. You have a bunch of names for files that you can open, like slash dev slash UHID0 for the USB human interface device, the first one. Or slash dev slash TTYU1, USB serial console. Or RSD3A, that's the raw, scuzzy disk, the third one, partition A. Or other ones like the dev zero software, purely software backed, just a pseudo device node that returns zero whenever you read from it. So there's a couple of different kinds, character devices, block devices, depending on whether you're talking to user-land programs or file systems. Distinction's not that important for this talk. We should kind of get rid of the block devices, but whatever. And again, the state may be backed by an auto-conference. There might be, in the case of like a USB human interface device, there will be an auto-conference for the USB device driver. It might be allocated in software, like the audio device in that PST is allocated in software. Then that lets the kernel do mixing in the kernel, so that multiple processes can use the same physical hardware devices, physical audio devices. Or the stateless, like dev zero, dev no, these are just software abstractions. There's nothing, no physical devices backing them. When you use them, when you use device nodes, you open them first, then you can do IO, read, write, IO griddle, and then you close them. Or read, write, IO griddle, or strategy. That's how you submit a block IO request to a block device. You'll have to ask Kirk why it's called strategy. So, simple example of an auto-conference driver. Sorry, the text is a little small here to fit it on the screen. The other code samples will be larger. Just a match function tells you, it says, do I want this device? Is this one that I recognize? In this case, does it have the right vendor and product ID for a USB device? And if so, then it returns, yes, I would match this. In NetPSD, you can have multiple drivers say, yes, I would match this, but give it different priorities. Not that important to you, just that's what a match device does. The kernel will decide which driver uses it. Then once you've committed to picking a driver, then the kernel will call attach. We'll have some private space that's allocated according to the size that you put in the attach declaration. And then attach sets, allocates memory, allocates buffers, allocates queues, initializes mutexes, and so on. Once you're done, once you're about to remove the device, when you pull out a USB device, the USB stack will call the detach function that then has to free resources, destroy mutexes, and so on. So, pretty straightforward here. Then for the usual interface, we're going to have some functions. This screen is a little bit small, but I guess it's just a closed brace here in case you're worried about this one being matched. So you have a collection of functions to implement open, close, and some IO functions. And in this case, this ULPT driver, this ULPT device node, or something's called a CDEV switch, CDEV SW for character device switch table from back when I guess the code was written as switch, switch the device minor number, and they would call these functions instead of using a collection of function pointers. But anyway, something called CDEV switch or just slash dev node, whatever. This will look up some private state for an auto-conference device that was attached with ULPT attached using the device lookup private function. And if there's no auto-conference device, then we say, well, no such device here. Otherwise, open will maybe start talking to the device, maybe allocate a USB pipe, allocate some transfers, get ready to talk to the device. Then close will reverse that, and other functions like read, write, and so on will do IO on the device. Here's an example of a tree of auto-conference devices with some device nodes, some slash dev nodes that talk to them. I took this from my laptop last night. So we have a graphics driver with a frame buffer device, frame bus for pseudo device, fictitious device sort of, and then WS display, which is exposed via user-land to slash dev slash TTYE0. And when X11 is drawing to this screen right here, it is going through slash dev slash TTYE0. Then we have another tree of things here under the PCI. There's an XHCI USB host controller, and then some USB bus and a hub and a mass storage device and then a SCSI bus on the mass storage device and a SCSI disk that then has a user-land interface slash dev slash SD0A, SD0B for different partitions on it, for the whole disk. And I have another two USB flash drives plugged in. Actually, one of these was not a USB flash drive, it's the SD card reader. So we have some other slash dev nodes, and then if I had an FTDI serial console, it would have some USB serial console dev nodes. So if I want to talk to a Raspberry Pi or a Beelowbone or something, I might use Minicom with one of those nodes or CU with one of those nodes. Now, device nodes are organized by major number, minor number, and whether they're character or block devices. So this is all that distinguishes two device nodes on disk. So you can use the make-nod command to create one with whatever block character, major, minor number you want. And the different drivers have different correspondences for the ones that have, for the slash dev nodes that have correspondences with autocomps devices. The correspondences are sometimes different. So with, you know, UHIDN, there's slash dev slash UHIDN, and the minor number is the same as the autocomps unit number, as the device T unit number. For USB serial console devices, UCOM, the minor number can be the same or there's also a dial-out node that works a little bit differently, a different kind of TTY, and that has a slightly different minor number, but these two are both associated with the same autocomps device. And I mentioned this because there isn't a one-to-one correspondence between autocomps devices and slash dev nodes, but often there is a correspondence and this will become important later. Okay, so the easy timeline for what happens when you're thinking about a device driver, when you have a device driver for a USB HIDD or something, is that the driver's attach function gets called when you plug the device in, then software happily opens the device node, opens the dev node, the slash dev node, and then software does reads, writes, iocutls, whatever, and happily proceeds along while the device stays firmly plugged in. Then software closes the device, and finally, once the device, once everything is safely closed because you're a good citizen of your laptop, you only unplug it after it's all done and then the attach function gets called and everything's hunky-dory. So I'm going to simplify this just to attach open IO operations, rewrite iocutl, close, and attach. Same timeline, just different, just less verbiage here. Well, this is the naive timeline, not just the easy timeline, because it's not really how the real world works because you're going to absolutely yank something out while you were trying to use it, and oh crap. So, yeah, so a program might still have the device open in a file descriptor when you yank it out and it calls the attach, and the device might even keep trying to read and write to the device, and before it finally closes. So that's no good. Another option is, well, software might try to open the device when the slash dev node, which is, okay, in free BST you probably won't have it unless there's a race condition, but in net BST open BST, the slash dev node is always there, and you might try to open it when there's no physical device backing it. So we need to make sure that open will fail when this happens. Another possibility is that while something is trying to open the slash dev node, the device might get yanked. Before it's even, you know, before open has succeeded, but while it's still trying to negotiate whether the device is there, what's going on. So what happens with this? Well, it depends on who wins what race, it might, open might succeed if it wins the race against your yanking, or it might fail, or the system might just crash. Especially if you do it over and over again, and repeatedly try to reuse the state. Another thing is that many device nodes can be opened by more than one process at once. For example, you can have a DD process reading from a character device for a disk, and have two different ones reading from different parts of the disk, or even reading from the same parts of the disk. And that's fine, there's no problem with that, except that the software has to contend with this possibility that the character device may be opened more than once at a time. So in some cases, multi-open is okay. You can, there's nothing wrong with that. You just have to make sure that if open allocated resources, well, only the first one does, or it coordinates with the multiple different opens. For other devices, it only makes sense for one process at a time to use the device, like a UHID device in NetBSD, that's exclusive. So you have to make sure that open fails when two different processes try to open devices at the same time. Or, you know, the easy thing is to just crash because you didn't think about this to begin with. Another possibility is, well, so if you, the traditional semantics is, and it's going to be hard to change this because all our drivers rely on it, is that the driver's close function from the character device switch table or the block device switch table is only called for the last close. So if you open multiple times, then only the last close will actually trigger the device's de-close function. Even though each open has a separate call to the de-open function. So when the first thread opens the device, the kernel will call the de-open function. When the second thread opens the device, kernel will again call the de-open function. And this might be necessary because you might have the device open for reading and then open it for writing. Or you might have to allocate some other state or who knows. When the first thread closes the device, then there's no driver callback because we only call close in the very last close. And then finally, when the last close happens, we call the de-close function. Another complication is that open can fail. So, you know, if you open a TTY and it's waiting for the other end to pick up the phone, so to speak, or even perhaps literally, if we're in 1980, then you might hit Control-C because you get bored. And when you Control-C, the program might be interrupted and failed before open has succeeded. So you have to consider that open might fail. So when you call open, when the program calls open, the software calls open, then the kernel will call the de-open function. When open fails, the de-open function just fails. It just returns and nothing will call the driver's de-close function. So the driver has to make sure that if open fails, it releases any state that it allocated temporarily because there will be no close if open has never succeeded. Now we get to a trickier case where we might have concurrent opens and the device might be fully open and then another thread calls open. If there's some reason, it's going to fail. But in the meantime, while it's in the middle of calling open, the first thread might close its own file descriptor. So we have a fully open device and a device that's in the process of getting opened, but then while it's in the process of opening a second time, the first thread closes it. Now at this point, the second thread might still be using resources that the close would free up. So the first thread, when it closes the device, can't free all those resources yet. It has to hand them off to the second thread because the second thread might succeed, might fail. Who knows? We don't know yet. But then once the second thread fails, now we have an issue because we have some leftover state. It needs to be cleaned up. So when you first call open again, the driver has its de-open function called. Then when you call open again, the de-open function gets called again. But when the device is closed, while there's another open in progress, there's no callback because we don't know if that state is done being used yet. And so even though the second thread's open failed, it has to call de-close. The kernel has to call de-close internally because there's nobody else to clean up the state at this point. So even though the second thread only witnessed a failed open, it still has to call de-close, which is counterintuitive with respect to the previous slide where there's no driver callback on a failed open. In some cases, a failed open actually has to close the device inside the kernel. In other cases, it can't. It's a tricky case that we have to deal with inside the kernel. Software doesn't have to worry about this, but the kernel has to worry about dealing with software that does this. Detach. This is what happens when you pull out a USB device, when you yank a USB device, or when you yank a maybe a Thunderbolt device or anything, or even if you type drivkernel in that PSD, drivkernel-d to test the detach path without pulling anything out. We do this with testing PCI devices too. So it's triggered by something in the bus, something in the kernel will decide to run and remove the device. The USB notifies it, hey, the device is gone, so the kernel says, okay, great, got to call the driver's detach function. And it has to free any resources that were allocated by attach. But what if the device is still open? Well, so I'm going to give an analogy here, which is suppose you have a road that you want to repave, and there's cars on it, and there's maybe cars driving on it, cars parked on it, and if you want to repave it, how do you go about doing that? Well, the first thing you do is you boggle it and lay it out for a tram. Sorry, this is the wrong place for that kind of talk. Okay, so you close it off, so new cars can no longer enter. You put off barriers so nobody can start driving on to it. But there might still be cars parked on it. So if there are cars parked on it, you have to notify them, hey, you've got to move your car, or else we're going to tow it. Or if you time out waiting for them to move their car, and then you wait for all the cars to leave, and at this point no more cars can be entering, and so you only have to worry about the cars that are already there. You wait for them to leave, and once they're all gone, then it is safe to repave the road. And put in a separated bike lane. I'm going to be pushing to the choir here. I'm from the US and we don't have a lot of trams or separated bike lanes. Kind of wish we did. So similarly, with a device driver, or any sort of software resource, really, if you're going to defer a resource that may still be in use, you have to follow this three-step pattern here, close it off so no more users can start using it. Then if existing users might be waiting indefinitely, like cars parked for who knows how long, you have to wake them and make them stop whatever they're doing, get out of the road, whatever it is, and once they've all finished using the resource, then it is safe to free the resource. So this pattern is going to appear in many ways, not just in attach and detach, but also in things like suspend and resume, not covered in this talk, but it's a similar idea. So with an autoconf resource, an autoconf instance, you need to prevent an autoconf instance that has a slash dev node associated with it. You need to prevent new opens of the slash dev node and interrupt any pending I.O. that might be trying to use it so that programs that are reading from your USB device will fail, it'll return the E-AXI or EIO or whatever, and then wait for all of that logic to finish before you can free the resources in detach. Now, getting all of this right inside the driver is fairly difficult. At first, I tried to engineer some drivers to do this. I spent a while hacking around a few different USB drivers to make this happen without changing any logic inside the kernel, and there's just too many things to coordinate to get this right, and some of them are actually kind of impossible to get right. I won't go into details, but even if it weren't impossible, it would be great to make it easy for all drivers to take advantage of a stronger contract from the kernel than VSC has traditionally provided since things like detach, you know, removal media like USB devices were introduced. So one mechanism that we've added is device T references. So often in an open function, we're given a devT, which is a number of a slash dev node, a major or minor number, a pair of major or minor numbers, so this uniquely identifies something like slash dev slash UHD0 or slash dev slash RSD0A, except without the character and block distinction, but never mind, don't worry about that. We'll need to get an auto-conference pointer from that and then get at the driver private soft C from there. So the traditional logic is that you call device lookup with the auto-conf driver, CF driver, and pass in some unit number for the device. Now recall, some devices have many slash dev nodes corresponding to the same unit number. For instance, every disk partition device node uses the same disk. A TTY has a regular TTY node and a DTY node for dialog devices. So we map the unit, map the device number to an auto-conf unit, lookup the auto-conf device. If it's not there, fail, whatever, not important, and then get out the driver private state so that we can get resources, you know, USB pipes, transfers, buffers, whatever. However, there's a problem, which is that nothing in this logic prevents DV from being detached. And when I say the principle being detached, I don't mean prevents, you know, the physical USB device from being pulled out, but I mean from this, nothing prevents the software, the kernel software, from calling the driver's detached function. So once the driver's detached function is called, well then SC, the soft C, might be freed and so you're just reading garbage memory that's been freed, used after free here. This is no good, this crashes, corruption, whatever. So we added a mechanism for looking up a device and acquiring a reference to it at the same time. This reference prevents the device, the detached function from being called and therefore prevents the soft C from being freed. So while this logic is running until device release, the detached function can't be called. Now, it's important in this region for the software to not sleep for too long, maybe sleep on a mutex, but nothing else because you don't want to make the system hang when you run plug the device waiting for something that might take indefinite time before you can actually run the detached function. So there are some rules for using this, but you just make sure this section where you're doing something here doesn't take too long. This kind of fragment would have appeared in a lot of different device driver open functions, but we have a lot of device drivers in the tree and so it would be a lot of work to go through all of them and insert device release into all of the error branches. We started to do that, but there's too much stuff to test which you need to test with actual physical hardware because we don't have simulators for all of it and it's just really error-coding. You have to make sure all of the error branches are exercised. So it's difficult to get this transformation right. The mechanism is useful. We use it in a couple of places, but mostly it's not necessary because we added a way to connect up the autoconf CF driver to the device switch table. So this fragment tells the kernel that the device's open function is going to do a device lookup with foo cd with that autoconf driver and it's going to use devminerunit, this function, to map a devt major and minor number to an autoconf instance number. So this way, when you implement the foo open function, this doesn't have to change at all. We just do a regular device lookup and device private and while the open function is in progress, when it's been hooked up to the CF driver and device unit number, the autoconf device dv and the driver softc, these are stable until the function returns. Now, the open function has to not hang for too long and it'll get to what happens later if you have to hang in, for example, a TTY driver where open actually can't hang indefinitely. But this way, at least using the software state is safe inside the open function. We don't have to change any open functions annotating the error branches with device release. We can reuse the open functions as is, make a small change to the cdfsw, cdevswitch table to make the open functions at least safe in case you're using the USB device while open is in progress. So you just have to add the dcf driver and ddevt unit to the struct cdfsw tables. Now, devt unit has to match. This is important. It has to, so whatever you use here, when you're mapping a devt number, a device, a major, minor number to an autoconf instance, that has to agree with what you put in the cdfswitch table. If you don't, things are very confusing and I want that afternoon back. There are a few built-in devt unit functions. For many devices, the autoconf instance number is just the minor number of the device. For disks with disk label partitions, it's a little more complicated with the partition number and the disk because we still have to deal with disk label, grumble. For TTYs, again, similar thing, just one bit for the dial-out versus non-dial-out node. Okay, now, a bit of a digression on revoke and BSD TTY security. So there's this funny system call called revoke in BSD and it doesn't appear in System 5, Linux, whatever. They work differently. And it's used in pretty much one place in the entire system, the entire OS, the entire user land. It is pretty much one call, traditionally anyway. And that call is in Getty. So the way that traditionally you boot up and you need to talk to a computer, you talk to a computer through a TTY, of course, because that's how you talk to computers. And so on boot, Getty will open a TTY and call login to read a username and password or whatever authentication mechanism from the TTY. When you successfully log in, when you type in the right username and password, login will shown the TTY to the login user. And then that user owns the terminal and that user can read and write to the terminal. Nobody else can. Nobody else can snoop on your MUT session to read your private mail. Nobody else can snoop on the TTY display of your MUT session. And when you log out, Getty will show the TTY back to root so that the next person who logs in, you can't snoop on their MUT session. So you can't open a TTY at this point, but there's a snag. Getty revokes the TTY because you might still have an existing open event that would let you just quietly listen in on whatever the next user's MUT session is or whatever the next user's session is. So the mechanism of the remote system call is to ensure that anyone who still has a file descriptor open can't continue to use it for the same underlying resource. It is a BST-specific thing. I'm actually not sure I can tell when it does it. This is basically one call in the whole system and it's in the Getty program. But it turns out that Detach is actually pretty similar in consequence to what's going on here because you need to make sure that all use of the file descriptor that refers to a device has ceased. So when you Detach, and many Detach functions for drivers, if there's a slash dev node, you have to call this vdevgon function which has the effect of calling, the same effect as calling revoke. It's the complicated mechanism internally in that BST, but it's the same effect as calling revoke. And that forces D close to be called for the device if it is a slash dev node. Well, which in the case it is because it's a Detach function for an auto-conference instance. Okay, so let us know there's a read that hasn't finished yet when you Detach the device and the Detach function calls vdevgon, revokes the file, and there's still a read going on. There's still a program that's trying to read from your USB serial console. You can see you running on the card serial console. You can talk to your Raspberry Pi and you just yank the whole thing. What happens then? There's actually a choice of semantics here that is a little bit different in Linux and BST. In Linux, the file remains indefinitely. You can't do any new IO on it, but existing reads or whatever just keep on spinning there until you hit Control-C or something. And in BST, the traditional semantics is that the IO is interrupted and the read will immediately fail with EIO or whatever. I'm just going to focus on the BST semantics, not discuss the merits of one or the other. This is what we have here at BST and this is what BST programs expect, the worth kernel implements and so on. So if you want to close a file that's still in use, where there's still a read pending or still a write pending that's in the middle of that or an IO kernel that's waiting on something, again you have to follow the three-step pattern, prevent new IO operations, interrupt any existing ones that are there waiting and then wait for them all to finish. At that point, it is safe to free the driver state. So in BST, I tried to assist all the drivers with doing this without changing any of the API, but it was very difficult to capture this pattern without adding a new function to the Dev Switch table. So for the legacy approach, which is with just a de-close function in the Dev Switch table, BST will first prevent de-open, de-read, de-write and stuff from starting. So once control enters the de-close function, you don't have to contend with new calls to the IO functions or new opens of the device until you finish with that function, with closing it. But you do have to contend with existing opens, existing reads, existing writes that might be hanging. So if you have a TTY that's open, waiting for the other side to pick up the phone, the close function has to go out of its way to interrupt that and make sure the open finishes at some point soon. So de-close is responsible for both interrupting the pending IO and waiting for it to complete before close can free any resources here. And there's a problem, though, which is that most drivers don't wait. And so as a sort of stop-gap measure for when we need to integrate this with a detached function, NetBSD will wait for these, for all functions, after close returns. So this is not great because they're still going to be broken drivers in here that are using this, but at least we have some classes of bugs where detach tries to free resources that were still used by the de-open, de-read, de-close, etc. functions. De-close might have bugs on its own if it tries to free resources, but at least this helps protect detach a little bit. But there's a much easier way, which is with... Oh yeah, sorry, I forgot to mention, TTYs can hang indefinitely in de-open. There's a much simpler way, which is with de-cancel. So when you closer evoke a device which does support de-cancel, which has a de-cancel function, then starts out the same way, NetBSD prevents new opens, reads, writes, i-cuddles. Then it calls de-cancel, and de-cancel is a very simple job. It just has to go and interrupt any pending IO. It just has to send a signal to if any threads are waiting, and that's it. It doesn't have to do anything else, just stop any waiting item, make sure that any concurrent read, write, i-cuddle functions will return promptly, and that's it. Then NetBSD will wait for all those to finish internally. You don't have to have driver logic to do this. NetBSD will do it internally itself. It will wait for open, read, write, i-cuddle, etc. to return. And then at that point, it will call de-close. The job of de-close is much simpler here because it has exclusive access to the device. There is no need for it to contend with concurrent opens, reads, writes, and so on. And this is all generic chronologic in NetBSD's specFS special device node file system. And for TTYs, there's a new TTY cancel function which you can use here that will just cause the TTY open function to interrupt. It's very easy to adapt existing drivers for this. Here's an example of a read function from the USB HID driver. It just looks at the driver private statement. Divisible private is safe here because the device is always open at this point. So you don't have to contend with open and close. This never gets called until open is done. And it can't start getting called once closed gets called. We look at the state. It's stable. No need to acquire or release a reference inside this function. We take a look at the queue. And if there's nothing in the queue, we have to wait. But before we wait, we have to check if this device closing. And if it is, we have to return promptly. And if we wait, then we loop back into here repeatedly. We may have a serious wake-ups or the interrupt routine might wake us up when we have data ready and so on. But the important thing to note here is before waiting, we have to check whether the device is closing. This will become important on the next slide where in cancel, all we do is set the closing flag and then wake anyone who's waiting on that condition variable. So this wakes up all the reads, all the writes, all the autocursors, et cetera, and makes sure that before they try to go to sleep again, they check SC closing and notice it's up. Guess we've got to give up, and then they will exit the loop and not go back to sleep. You have to stop internally for the human interface drivers that will have a similar effect for things inside the USB stack. Then in close, oh, did I miss? Yeah, I guess I missed close. Oh, well, close is not very interesting anyway. It's mostly canceled. It's interesting. Detach then. This is what happens when you yank the device. Well, it will look up, the detach function will look up the major number, minor number, and then say, okay, curl. Any open files for this major number and this range of minor numbers that are character devices, now, revoken, can't use them anymore. And by the time vdev gone returns, they have all been closed. So there is no open, no read, no write, no article, no close, no cancel, nothing. All of that is done so you can safely free all the resources at this point. There's one kind of small detail that's important to note, but it's kind of weird. If the open sleeps, like if you're waiting for a TTY, and de-cancer, de-close, whichever you're using, wakes it. For example, a TTY driver where it can hang indefinitely until it gets canceled. The mission checks may have already happened by the time the de-open function has gone to sleep, waiting for the other end to pick up the phone on the TTY. So de-open really has to return e-restart. It can't loop back in a CV weight loop or a T sleep loop or something. It has to give up and say, whoops, okay, restart the system call because those permission checks have to be redone. This is a kind of funny quote I found in a lot of drivers, TTY drivers. Many of them, if they like have a hang-up delay or you have to wait a second or two before the, or you can re-open the TTY, they would sometimes just loop in this and that's not correct. You need to actually go back and return e-restart. So anyway, here's a summary of the new contract for device drivers with autoconf and slash dev nodes. If you provide, if you set a CF driver and dev to unit function that match the device lookup use in de-open, then this is driver's job. Then in exchange, the kernel will guarantee that when you detach a device, no new de-open functions can be started and device lookup results in de-open are stable. You can safely use them, stash them somewhere until you're done. Similarly, if you set a de-cancel function in your slash dev node and it interrupts pending IO and returns promptly, then in exchange, the kernel will guarantee for you that de-close has exclusive access to that character block major-minor tuple. So nothing else can be used in the same major-minor number. Nothing among the dev-switch functions. And no further open, no further IO including in de-open is possible until de-close returns. So the de-close function is much, it's job is much simpler in a concurrent world when you implement the de-cancel function in this contract. Now there's also lots of detailed edge cases that I had to deal with in the logic around open and close. I had about, I don't know, 10 different commits each with multiple paragraphs of text in columns explaining a weird situation I encountered while sitting in the living room just running a program to open and close devices repeatedly and yank, yank, yank. Oh, that's interesting. Wow, okay. I'm not going to go into all the details. I don't have any time left. So here's a brief summary of the usage model. Kind of like a regular expression. You're open, then a bunch of concurrent IO. You can repeat this for a lot, then de-cancel, that can happen concurrently with this, but then de-close is exclusive. It's very, very simple. It's a more sequential model than we've had for a long time, going back to the days when it was actually sequential. And that's it. I think I have maybe five minutes for questions. I talked a little too long maybe, perhaps, but... Okay. Oh. Yes. What a benefit of the interwebs audience. There's a question. So there's... Just on open, can you restart, come back to user-land? Because that's a new error code for open, in that case. Does open go back to user-land? No. It will restart the system call. It will go back to the... As if the user had just called open again. So it doesn't go back. It doesn't return to user-land. No. Okay, thanks. And this is not new. This is part of the semantics of e-restart. For any system call, actually, it can restart the system call. Yeah, but there are some system calls where, obviously, you can get a really start... Yes. So in some system calls, if you get a signal and there's no essay restart flag, then you want to get a read. If you've done a partial read, then you want to return that partial read and not restart the system call. In this case, there's no notion of a partial open. There is only... It's either open has not finished and you start over, and you're done or open has failed. And those are the only three options. Thanks. So, obviously, sitting in your living room pulling out a USB drive over is a very thorough testing method. But I was wondering if you had any kind of thoughts on frameworks for automating testing of this kind of stuff? Yeah, so we have a system called VHCI, virtual host controller interface. It is a software host controller interface. And we've used that for a lot of automatic testing of the... mostly the USB stack itself. And it's been great. We found a lot of bugs that way, a lot of bugs in parsing USB descriptors. And so you can use that to, you know, write software that pretends it is a host controller, USB host controller, and pretends it has a collection of devices, a bunch of USB descriptors and so on. And on the other end, the kernel, well, it looks like it's a USB bus, whether USB device is on it and so on. All the driver activity. The reason I didn't use that for the drivers that I was working on is that you need to implement something that simulates a USB serial port, or a USB phyto key, or a USB keyboard or something. So it's easy to put in descriptors, but you also need to simulate all the behavior that the driver expects as well. And that would be great if we had that and we have the framework in place to do it, but it's a lot of work to implement a simulated USB keyboard, simulated USB, you know, mass storage device simulated and so on. So if you're interested in that, that might be a good project to work on. Be great for automatic testing of drivers. It's just that's not how I was working in my living room with the Iancan reinsert. Thanks. Any other questions? We have about two more minutes before this session ends. Do I have the clock rate? No, it's around 30. Okay, all right. So yes. That clock is slow. Did you find that there was any weirdnesses between brands of USB drives? Were there just certain unnamed generic brands that you were having issues with? I was mostly working with USB hidden devices and serial ports with some mass storage devices as well. I didn't remember anything about anything, any issues with the hardware itself. This is pretty much all software issues. There is a kind of funny thing with the phyto keys, which is something about the data parity bit or something gets tunneled and I forget what the term is. It's kind of a finicky USB thing and there's a hack in the phyto 2 driver to deal with that, but that's kind of a separate issue. This is pretty much all software bugs. All right. Thank you. We're going to interrupt the clock rate. I didn't mean to do that. I was just thinking, did you try MIDI QMU? Is there maybe a USB keyboard supporting QMU that you could like script or whatever? I did not try anything with QMU. I don't know if QMU will do a hot plug of USB devices. I mean, I guess, well, it could do pass through, but I don't know if you can use software to script hot plug of simulated. I don't even know if it does this. It's really possible that QMU has a mode where you can interact with it interactively or maybe even script it. You can, I think, work on devices. If you can use QMU in software to trigger detection and reattach, then that would be a great way to test this without having to write a new simulator for a USB device. All right.