 Let's start. My name is Johan Overwald, and I work as a consultant doing kernel work primarily with embedded systems. I am the maintainer of the kernel's USB serial system, which is why I take an interest in the topic of today's talk. I also maintain Graybus together with Graycore Hartman. Graybus, for those who haven't heard about it, it's the application layer for Unipro that came out of Google's Project R, the attempt to build a modular mobile phone. And today I will talk about the Serial Device Bus, which the executive summary being aims at making serial attached devices fit better into the Linux device model. So you are to NARS 2.3.2, it's been around for a long time now, at least since the 1960s and a few years before that. It's still a common interface, especially in embedded system for Bluetooth, NFC, radio, GPS devices and so on. The kernel provides an abstraction for the serial connection itself, but it devices more than just a communication interface, you have associated resources like GPIOs and interrupts and so on, that you need to be able to manage, not least for power management purposes. And unless you want to implement your driver entirely in user space, the kernel support so far has been more or less limited to so-called line discipline drivers within quotes, because they're not really drivers in the Linux driver model sense. And they had a drawback that they must be configured and initialized by user space. So brief outline of the talk, I'll start with giving a high level introduction or overview of the TTY layer and discuss those two options that have previously been available to us to write your driver entirely in user space using line discipline drivers, discuss some of the drawbacks and problems associated with those, before introducing Serative, talk a bit about how it's been implemented, the driver interfaces that anyone wanting to implement a Serative driver can use. And after that, I'll talk some of the limitations that Serative has today and point out some issues for future work. So the TTY layer, ultimately it provides this character device to user space, abstracting beyond the lane, serial port hardware. Beneath the character device we have the line disciplines, which is an entity abstraction which deals with any kind of IR processing. This is where the canonical input mode, line editing facilities is implemented. It deals with echoing, error handling, parity framing, but also things like emitting signals on various input conditions. Beneath the line discipline we have the TTY port layer which is a, where the input buffering currently is implemented in the kernel. And this is also a kind of abstraction layer for implementing common positive semantics. For example, when opening a port, you need to raise the TTR, yes, lines and so on. And beneath it all is the actual TTY driver which know how to speak to the actual hardware in question. In this picture I've divided the TTY driver into the TTY driver core bit and a lower level driver, because this is usually how it's implemented. The serial core would be the TTY driver or USB serial core, whereas an OMAP UI driver or the FTDI driver would be an example of a low level driver. And as I said, this is a bit of a simplification. In reality there are some, the line discipline actually calls down directly to the TTY drivers as well, but for our purposes this is sufficient. So the user space drivers, you could implement your driver entirely in user space using the default line discipline. This means obviously that your description of your system, your hardware needs to live in user space as well, the application needs to know which port to open, what line speed to use. And if you have, for example, a simple reset line enabled switch, you could still do that in user space because GPIO provides an interface for user space to access. But when you get to things like regulators and clocks, they don't have a representation in user space there, then you won't be able to implement your driver in user space. Obviously you can't use the kernel's power management facilities, you need to implement your own, you need to deal with things like system suspend notifications, which arise from user space to start with. Wakeup interrupt is another issue because you cannot specify an interrupt as a system wakeup interrupt through a GPIO loop currently. And obviously you can't use the kernel's infrastructure for firmware management. So, and for those reasons, and also because you usually want to interact with other kernel subsystems, you need to have your driver living in kernel space. And that's where the line discipline drivers come in. These other subsystems could be Bluetooth subsystem, input, NFC, and so on. And the line discipline in usually then registers further class devices, which is the actual interface with which user space will later use to access the hardware. This means that you need a user space demon to initialize the ports, switch the line discipline, and to actually keep the port open because it's only while the line discipline is opened that these devices are accessible. So we have these commonly known demons like LD Attach, Input Attach, and ACI Attach, which is being replaced now by Bluetooth Attach. Comparison to user space drivers, this means that the kernel's firmware infrastructure is available, but there are still other issues with power management and other resources like clocks and regulators. So a quick example here, Bluetooth. This is essentially what the ACI Attach demon needs to do. Open the port, configure the board rates, so on. Switch the line discipline, and in the Bluetooth case, you also need to specify which Bluetooth protocol to use. And it's only when this last IO control is executed that the actual ACI device becomes available for user space to use. So graphically, we have something like this. When the system is booted, we have the lower level driver being bound to his device, typically described in firmware. So the character device is available to user space. You need to first run your ACI Attach demon application one here before the line discipline switch and the class device becomes available for, for example, HCI config to actually start up the Bluetooth device. So what's the problem here? First of all, the description of what's going on, description of your hardware, what is connected where and how, lives in user space rather than being coded in firmware, firmware being typically device tree or ACPI. And this means that you don't get any automatic discovery. We need to launch these user space demons before we can actually access the Bluetooth device. And because your Attach device doesn't have a firmware description, we have a problem with describing and looking up these other associated resources. GPIO interrupts could be for reset, wake up purposes, ping control and so on. And obviously this is a problem for power management. We need to be able to control such resources. And a further issue here is that the port is always kept open. So depending on how the underlying TTY drivers been implemented, this may prevent the port device, serial device from going into run times to spend. Firmware loading is available, but firmware loading typically involves toggling a reset line or something like that, which then also depends on this fact that we cannot access GPIOs. So that's the background and that's why the part of the problem that Serative is helping us solve. So Serative is a new bus. It was implemented by Rob Herring of Linauro. And it's a generic bus for your Attach devices. I think the immediate aim here was to replace the TIShark transport driver and it's vendor-specific UAM daemon. But it's also based on earlier efforts, typically driven by power management concerns. There's been, was an example of a GPS devices which needed to be powered on. And you didn't want to have it powered on unless the GPS daemon was running and the port was open. Typically those efforts aimed at adding this at the Serial Core layer in the kernel, rather than at the TTY port layer. So that's part of why that those efforts never got anywhere until Rob and others picked up the thread. The bus itself was merged before 11, but there were some initial problems with how it hooked into the TTY layer. Lifetime issues that, so it needed to, that connection needed to be reverted and it was enabled only for Serial Core in 4.12. So SeriaDev introduces a new bus type. It's named Serial, not SeriaDev. And the concept of SeriaDev controllers and SeriaDev devices. The SeriaDev devices are also known as clients or slaves. And it's basically because devices is such an overloaded term, it may sometimes be more clear to refer to as a client device or slave device. And now we'll be going back and forth between these three as well. So just be aware of that. And there was only one TTY port SeriaDev controller implemented in the current today and it's the TTY port SeriaDev controller. So that means that sometimes we wrongly identify it with SeriaDev itself. The, this controller works as follows. So when a TTY driver is registering its ports, if there are any clients defined, then a SeriaDev controller will register instead of the typical character device. And the clients that are then described by firmware, they're currently defined by device tree, but in 4.15 we will have easy PR support as well. So if we return to the example with Bluetooth, we have something like this now. As before, when the system starts, we go out and parse our device tree. We find that we have a Serial port described in firmware. We bind the OMAP Serial driver in this case. It goes out and registers with the TTY layer. Which we now call into the SeriaDev core and go out and look if we have any Serial attached devices to find in firmware. If that's the case, we'll register a SeriaDev controller and a SeriaDev device client instead. And once the SeriaDev client is bound to its driver, it can register the HCI device so that once the system is booted, the Bluetooth device is already accessible and we can run HCI config immediately. So we compare it to the old situation here. We've done away with the daemon application one in the right end there. We're bypassing the line discipline entirely and we no longer have a character device representation in user space. This also means that the port is no longer being kept open unless the SeriaDev client actually needs to have the port open. And most importantly, the SeriaDev client has a firmware, a representation of firmware. So we now have solved the problem of describing these associated resources that the client may need to use. So some bit of details on how the TTY port controller has been implemented. I mentioned before that it's a TTY driver's responsibility to actually register these controllers and that's been done by hooking into the TTY port registered device helper. So when there's a client defined, this will register a controller in the slave instead of the TTY class device. And the TTY port struct in the kernel has been amended with two new fields here. It's TTY port client operations and client data. And the default client operations simply forwards data to the line discipline but when we have a SeriaDev controller registered, it will forward it to the slave device. And in the other direction, the controller interface is implemented by using these TTY operations that I mentioned in the beginning. Device tree bindings for serial attached device, a device tree serial attached node is simply a child of a serial port node. And the only requirement is that it has a compatible property. There is a max speed property defined as well. It's optional and it's not supposed to be used to specify the actual border rate to be used but rather to lower the maximum border rate. The driver is supposed to know what capabilities the hardware has and but it's only if there are issues with the actual physical board that you can cap that max speed by using this property. And this is also where you would specify your GPIOs and clocks and so on. So in this example, it's a Bluetooth node that's being defined as a child of the UART one. It has a TI compatible property and GPIO and clock. And in Sisyphus, this would show up as something like this. We have here two serial ports, both of being driven by Elmup UART. The first one does not have a child node in device tree so that there we get the character device, mean registered class device. And in the other case, we're using the snippet from the previous slide. So we end up with a serial controller, serial zero. It has one slave, serial zero dot zero dash zero. And it's only when that driver has been bound on the serial bus, the serial bus that the HEI device shows up in the system. So if you wanna write your own serial driver, this, the interface that you're gonna be using resembles that of the line discipline operations which is fairly natural because we're replacing the line discipline. So we have functionality for opening and closing a port, changing terminal settings, writing data, changing and reading back the mode and control signals and two callbacks for when you have incoming data and when there is more room in the outgoing buffer. There are a few additional helpers implemented around this but this is in fact the basic set of primitives that you have. Open, close, set, moderate, enable hardware flow control, writing a buffer, waiting until it's been sent, flushing and changing the mode and control signals. The thing to note here is that there is no write serialization enforced by a serial core. This shouldn't be a problem. TTY drivers are supposed to deal with the concurrent calls to write but since the character device implementation typically takes care of that, they may not have been fully tested so you may run into some problems there. Another thing is that there is no operation enforced ordering enforced by serial core at all. The only ordering that's been enforced is actually that you cannot write the buffer before you've opened a port if you're using the TTY port controller but there's nothing preventing you from calling it to set terminal settings before you've opened a port and so on and this will probably break some serial drivers. All these functions except write buff and write room may sleep. In the other direction, we have a couple of callbacks. Again, it's the receive buff and write wakeup callbacks. Receive buff is called in work queue context so you can sleep and it should return the number of bytes that the serial client has accepted and processed. Write wakeup typically called an atomic context so you can't sleep. So a quick example of what a Serative Driver may look like. You have a Serative Device Driver struct you need to fill in. It has an embedded Device Driver struct as usual. You set the name, you set your match table. This is where you specify power management operations so you can implement proper power management. Other than that, it's just a problem removal function and we have a convenience macro for defining our modular slave device driver. Serative, a probe function would typically look something like this. You'd allocate your private data as usually. You now have access to a firmware description which means that you can use calls like clock gets to go out and look up a clock resource or access the OF node directly. You'd store a pointer in your private data to the serial device that you can use in your class device callbacks and set the driver data to this private data. The important thing to notice is that you need to set your client operations. This is the callbacks for incoming data, right wake up before you open the port. And if you're application, if it makes sense for your application, you would open your port here as well before registering your class device. So Serative solves the problem but there are still some things to be implemented and some caveats for using it. As I mentioned before, it's serial core only and that has to do with these lifetime issues I mentioned in the beginning but primarily it's because Serative doesn't have any hot plug support at all and this was simply not a use case for the people who implemented it so it was something that left to be decided for later. And I'll get back to hot plug in a minute. It's also single slave only. There's currently no support for any kind of maximum protocols or using RS485. Two other issues to watch out for is that there is no input flow control implemented. I'd said you can enable hardware flow control but it only works in the outgoing direction if you're using something like auto CDS because there is no pushback implemented. So if your client can't keep up, data will be lost. There will be no pushback to the underlying TTY driver which can lower the RTS signal for example. Also, since we're not doing any input processing you don't have access to things like software flow control, parity framing over on errors aren't propagated up and you don't have any break signaling. Serial hot plug in is currently implementing using TTY hangups and file operations in the TTY layer but since Serative doesn't use any file operations this simply won't work. So this will require some changes to the TTY driver to basically have a call back to QSIO, flush out any ongoing operations before you can register to the clients. And this is, as I mentioned, the party version for the initial revert. We can't have Serative enabled for USB serial until it's been solved. Obviously we still have PCI hot plug and some PCI devices. You could actually rip out your PCI card and you will trigger these issues but it's fairly unlikely you will try that. Another issue with the hot plug is that we don't have any way of describing these dynamic buses currently. Only USB has some kind of rudimentary device tree support and for specifying basically static things like if you have an Ethernet controller or something that's connected over USB. We could be able to use device tree overlays but obviously we don't need some way to get those into the kernel and that's not available at the moment. And as someone pointed out when I had this talk a few weeks ago there is, some devices are entirely self-described. We have an example here of a HDMI CC USB device which presents itself to the user space or to the kernel really as an ACM device. It's handled by CDC ACM but you need to run the input attach demon to actually be able to speak the protocol, which would then in turn register a dev CEC device that you can use to turn on and off your TV for example. It should be possible to pass just the matching data from the CDC ACM driver but this should be solved in a generic way which probably means that we still wanna be using something like overlays but in this simple case it would be enough to simply pass a compatible string to serve dev to be able to do the final matching. Some smaller issues. There is no line discipline allocated. Sorry, there is still a line discipline being allocated even though we're not supposed to use it. We are allocating it and it's actually being called into. TTY layer is complicated and no one knows exactly what's going on everywhere and it's easy that you have some case you haven't thought about yet and oops, you got a callback. Another thing is that the series of controllers are always being registered. So for every port in your system there will be during boots when it's probed. A controller will be registered only then will we go out and look in the firmware and see if we have any child nodes. If we don't, we'll de-register, de-allocate and the character device will be registered. So that's inefficient at least. Obviously you don't have any character toys. Some people have been a bit surprised about that but it's really a feature more than a bug. The operation ordering I mentioned you need to watch out for. Another thing is that we don't have any bus power management. For example, runtime power management isn't enabled for the controllers which means that the client's runtime PM status won't be propagated up the tree. This shouldn't be a problem because it's essentially equivalent to having the ignore children flag set because you really wanna have your port be able to go into runtime suspender independently of your child devices and then when you're doing IO that's when you bring up the power on the serial port. There's also a possibility of running into submissions with code applications, the backwards compatibility because we have a number of line discipline which can now be converted over to CRDF but we still need to support the line disciplines for backwards compatibility reasons. For Bluetooth, this has resulted in the fact that we copied part of the implementation from the line discipline implementation and we now have two competing implementations which may need to be unified. There are some naming incidences to watch out for. For example, the bus is actually named Serial rather than CRDF and it's the CRDF device client slave being used interchangeably. Another thing to watch out for, Kconfig, we have two symbols for CRDF. It's both to enable the bus and CRDF core and to enable this particularity to wire port controller and you really cannot use CRDF or any of these Bluetooth drivers for example that are merged without having both enabled but the thing to watch out for is that the bus can be built as a module whereas the TTY port controller depends on TTY which is compiled in only, it's just now Boolean and this means that this option won't even show up if you choose CRDF to be modular. And since it's the only TTY port controller, CRDF controller really ought to be defaulting to yes and that will change starting from 4.15. So current users of this, we have three Bluetooth device drivers merged and a library function that again is mentioned based on HIL disk. It's for Broadcom, TI devices and Nokia. And there's one ethernet over UART drivers man for Qualcomm chip I think. And one of these drivers then HIL BCM, wanna say a few words on that. That's basically a sort of precursor to CRDF because it depended on some power management hacks which relied on having a child node, a child platform device of the UART device being described in ACPI or platform code. This device would be the one that has associated GPIOs and clocks and it would be when the platform device was probed you'd register the platform device in a global list in the driver and then when you get your HIL callbacks to the sibling device really, it would go to this global list, match on the parent, do we have the same parent and then call into the sibling device. This is layering issue at best, it's a hack but it's the kind of thing that you were forced to do if you wanted to implement power management before CRDF. Now this driver is gaining, has gained, Hans de Gutter has done some work adding ACPI, proper ACPI and power management support for the CRDF version of this driver and it's been managed for 415. But there are some possibilities that we run into regressions here. One thing being if you haven't enabled your CRDF controller, you can still use the line discipline but your power management won't work anymore and you can't really tell. You can still access, there's still a chance that you can access the Bluetooth device but power management won't work. And we just discovered as well that the HIL has copied these kind of hacks and it doesn't have any power management support for CRDF yet so that will definitely break if you have one of those devices using ACPI with HIL. Stuff that's coming is so, I mentioned ACPI support was merged, it was merged only four days ago, Friday by Fredrik Danis and the risk here is that we'll have some issues with HIL, BCM, HIL but it's been decided that this is the way forward so we'll deal with that when it happens. There's a patch that was posted a few months ago adding a Mox support to CRDF. It's based on the new Mox sub system and adds a reg property to these device tree nodes which is the Mox index. There's some issues with these areas. It doesn't do any flushing when it's switching from one CRDF client to the other and there's no locking in those IR paths so things will definitely break. In this series, there's an example of a iScrC control drivers. You have an iScrC controller connected over UART. It adds some basic parity support just to switch on parity checking but again CRDF doesn't propagate any errors up the stack so you won't figure out if something went wrong. And another driver, an MFT driver for doing some kind of supervisory I think it's for an in-flight system. You have a watchdog backlog and lead components and they are all accessed over this supervisory processor and it's implemented enough, an MFT driver. And there's some earlier examples of a GPS and a Bluetooth-led device driver that was posted. Future work would be obviously to address those quirks and limitations that I've listed. Specifically adding hot plug supports is the one that I care about because I don't want to enable CRDF or USB-CRL before this has been fixed. And lifting the limitation of only having a single slave would be good too. It may be possible to implement some kind of generic RS485 support based on this much work but someone need to dig into that because it may be better to do this on with a dedicated driver. We have further Bluetooth protocol drivers that can be converted. HIN is the top priority one because we now know that it's broken for ACPI. Other line discipline drivers would be for NFC. There's a line discipline driver for Canvas. The TI shared transport driver you can use now the Bluetooth component but there are some, it's shared transport so maybe the maxing subsystem can be used here to get these other components supported. I mentioned the CEC device and there are other CRIO drivers that could probably benefit from CRDF. And if you want to do some further studying on this I recommend looking at the code. It's only these three files basically include file and two files on the driver TTY CRDF and it's not really that much code to process. Device trip bindings have been defined where they should and there are bindings in there the thing to watch out for with accepted bindings for things that aren't implemented yet so you could have clock resources and so on being defined but the driver won't actually use them yet. And if you wanna read some more about the background about all this this is an article by Neil Brown on LWN which was written while CRDF was being developed and sort of iterates some of the history and some of the concerns and some of the previous attempts at doing this. Yeah, that was it. Thanks. Any questions? So I have few questions about runtime PM. You probably could correct me. Any driver could try to use it but it will not work, correct? If I implement in the each slave or control driver on Linus. Okay, so the second question is it does not cover console in kernel console is not covered by Z. Okay. And what about DMA support? Could you elaborate a bit about DMA support? DMA? I mean, that's something that's dealt with by the underlying TTY driver so CRDF wouldn't be involved in DMA at all. You'd only get your when you know when you have DMA interrupt triggers you'd get your receipt buffer being propagated up to stack through these callbacks I mentioned but CRDF wouldn't be involved at that level. It's built on top of the serial drivers on top of the TTY ports. First of all about the history the initial reason was the Nokia driver that triggered everything because I tried to commit a driver similar to the Broadcom one and got a neck for that one. And secondly about runtime power management if you just enable runtime power management on the host controller independently of the slave and the slave tries to write something then might be missed. That's why it's currently not enabled because at least on OMAP if you wake up from getting data on the serial line then the first few bytes are missed. Which might be a problem or might not be a problem depends on what is connected. So more thinking is required on that one. But I mean for OMAP the runtime power management for the OMAP viewer driver and that first of all it's always enabled. And when you don't have a runtime power management enabled for its child device which would be the serial controller this simply means that the child device won't be propagated up. It will still work. The port is always on when it's open currently. Any other questions? Thank you.