 Okay. Hello. It seems we're live. So welcome to my talk in these strange times. My name is Bartosz Golaszewski, and I'll be talking about GPIOs in Linux user space. While I'll try to keep this presentation fairly accessible to beginners, I won't be going into much detail about the things I covered during previous presentations when I was already talking about GPIOs in Linux. I'll just post some links to my previous talks and the slides. With that out of the way, I'll start with some background. So I'm with Pelebre. We are a base in southern France. Plus we have some people scattered all over the world, and we are right now around 40 mostly senior engineers, and we do a bit of everything. So we do upstream kernel development and maintenance. We help clients launch projects from concept through design and implementation up to manufacturing. We're also the founding members of Kernel CI and we regularly contribute to many open source projects. Personally, I've been working in the field for 10 years now. I work throughout the stacks from bootloaders through the Linux kernel app to low-level user space and build tools. What's important for this talk is I wrote and maintain LibGPIOD and co-maintain the GPIO subsystem in Linux and also contribute to many other projects like Uboot, Yocto, and so on. So this presentation will actually be much different than what I imagined when writing the abstract a couple months ago. It's not only because I thought I'd be physically present in Austin, but also the subject matter of this presentation change significantly. So in one line of the abstract, I said something like this, the GPIO character device has been extended with new features in Linux 5.5, and final new addition before declaring it feature complete our plan for 5.6 and 5.7. So this is all wrong. Basically, we got a serious reality check when we started receiving some bug reports on the mailing list and some feature request that were sensible by all means, but which we just couldn't implement without breaking the ABI. Breaking the ABI of the current interface, and all in all this line in the abstract should be changed to something like this. The GPIO character device has been extended with new features in Linux 5.5, but due to shortcomings, the first version of the ABI, the existing IOTL calls are being retired and V2 of the ABI is being introduced aiming at the release as part of Linux 5.9, hopefully. So there you have it. With this in mind, let's maybe dynamically change the title of the presentation to Linux GPIO, a lesson in user API design. So welcome again to my talk. I'll be talking about what to keep in mind when designing user-facing interfaces in the Linux kernel in the context of Linux GPIOs. So the agenda for today is as follows. First, I'll discuss the current state of the GPIO user API. I'll talk about the deprecated CISFS interface. I'll briefly describe the character device basics, and then go to the recently added new features, including but not limited to the GPIO aggregator. Next, we'll talk about the work happening right now on the new version of the GPIO API. I'll talk about what's wrong in the current ABI, what new features are being worked on, and I'll also talk on what to pay attention to when designing user-facing interfaces. Lastly, I'll talk a bit about the user-spaced part of the GPIO interface, which is the GPIOD. Again, I'll talk mostly about new features and the upcoming development, because most of the features already existing have been covered previously. So let's start with the current state of GPIO user API in Linux. What you normally do with GPIOs in the Linux kernel is to try to use them from the kernel space. That is, you should aim at writing drivers for your hardware and use the internal GPIO interface, but that's not always possible or maybe even desirable. There are many cases where you'd want to control GPIOs from user space. Examples of such use cases are power switches and relays, various devices that are controlled in big part from user space, for example, GPS. Also, certain users simply seem to prefer to control elements of their hardware from high-level user space applications. This is prevailing in robotics and home automation. In 2008, the GPIO subsystem was quite different from what we have today. There were no GPIO descriptors. There was only the global GPIO number space, and also there was no GPIO maintainer. It's during that time that the GPIO SysFS interface was merged. Today, it's really become apparent that it's outdated and has many drawbacks which were the reason to implement the character device in the first place. First, the state of the GPIOs is not tied to any process. It's global in the system. For instance, if a process modifies the state of certain lines or just exports some lines and then dies the settings aren't reset. Of course, that also makes it possible to access the lines concurrently from many processes, and no process can really own the lines. The API is quite cumbersome with many attributes and different SysFS files. Things like looking up lines by name are complicated. By far the worst part of the interface is the fact that it represents a concept that is a two-level hierarchy as a flat number space. Precisely what we did in the all-deprecated kernel interface. But this is especially wrong since the global GPIO numbers are dynamic. Unless a driver explicitly makes it static, it can change depending on the ordering of modules being loaded. There are even bizarre caveats with GPIOs and SysFS like the fact that if you export a line by writing a number to the export attribute, it normally shows up as a new directory called GPIO number. But if the line is named, it will show up as a new directory with that line's name, which is some way of looking up lines by name, but it's not really intuitive and makes the SysFS behave in a non-consistent way, let's call it. So I'd really like to stress the need to move away from this interface because even last year in Lyon during the ELCE, it was brought to my attention that there was a talk presenting among others, the SysFS GPIO interface. So we're working hard to remove the global number space from the kernel, but SysFS is one of those things that will block us for the foreseeable future. So yeah, please don't use it anymore. It's not disabled by default anyway. So since Linux 4.8, we have a new interface for GPIOs in user space and that is the character device. So now for every GPIO chip, we export a device file in the dev slash dev directory. It can be accessed and acted on using standard Linux system calls, Open, Paul, Reed, IOPL. This time, the state is tied to the process owning the file descriptor. It reverts to default when the process closes the relevant file descriptor manually or closes by simply dying. We now can manipulate multiple lines at once. We can easily look up chips and lines by name. We have several config options unsupported by SysFS, like the drive flags, bias flags. We have visible consumer names, new event notifications, and finally we have something interesting, which is reliable polling. Because actually previously in SysFS, many people are not aware of that, polling for line events was possible, but it was quite unreliable in cumbersome. So in SysFS, when you export a GPIO line, you can open the value attribute and poll it for input events. And when you get an event, when Paul reports an event, it means that there has been a change in line value. Now, what you need to do to read the current value is to either close and reopen the value attribute, or you have to LC to its beginning and read the ASCII value from the attribute. And as you can tell, this is quite racy because there could have been multiple changes, multiple line events during that time. And so with the character device, this works much better because now the events are queued in the buffer in the kernel. Every time an interrupt is handled, a new event is put into the kernel FIFO, and yeah, unless this buffer overflows, it can be read one by one from user space. The poll system call will return an input event as long as there are events queued. And yeah, this allows to implement simple bit banging from user space. This will be actually made even more reliable in V2, but I'll be talking about it in a minute. And so I don't want to get into much detail about the character device, about all the features. I will focus on the new features, but this is a link for my talk that happened in St. Petersburg during the Linux Peter in 2018, I think, which covers basically all features of the kernel UAPI and the GPIOD. So what I want to talk about are the new features. So recently we've had some development happening. The first interesting new feature is the new Set Config IOPO. So one of the drawbacks we identified in the character device was the fact that you cannot change the configuration of a requested line without releasing it and requesting it again. And this is obviously racy as someone may simply grab the line when it's released and we cannot even change the direction from user space while maintaining the ownership of the line. So the answer to that is the Set Config IOPO. Basically, it allows to change the configuration flags for a set of requested lines. Unfortunately, for now, this doesn't work with lines requested for events, but this will be addressed in version two of the API. Basically, you have to use a very simple structure which contains the field for the flags, which is analogous to the one we have in the context structure used when we call line-handled request IOPO. And we also have optional default values used when changing the direction to output. This is also a counter part of the field used in the context structure. And the next feature is something that has been requested for a long time, but we simply somehow could not really agree on a proper approach, which turned out to be relatively simple in the end. It's the bias configuration. So many GPIO controllers have internal configurable resistors. In the kernel, this is represented as bias pull up or pull down options in the pin control and GPIO subsystems. And now we can finally change these options from user space using the three new flags. There's the pull up, pull down, or bias disable flag. And these work both when requesting lines as well as with the set counting IOPO. So I know that this feature in particular made a couple folks very happy. And then we have the line watch IOPO. So this is something that I personally really wanted to see because I'm interested in the user space part of the GPIO interface. So the background for this is the fact that GPIO character device doesn't hold a global state, and this is by design. But some people are used to having some central authority having control over all the GPIO lines. And in this case, what we're working on is a GPIO demon that will, basically it would be responsible for controlling the global state of the GPIO. And in this case, it's useful to always be up to date regarding the current status of lines you control. Otherwise, so you want to know if their status, when their status changes, when they are being requested, released, or their config changes. And otherwise you'd have to constantly pull the lines for a config change at certain intervals. So the line watch IOPTO allows you to set up a watch on the line state changes. And when calling this IOPTO, the current line config is returned first. So this is how it is. We call the line watch IOPTO. We get the current configuration as it is at the moment. And then we pull the GPIO, the file descriptor associated with the GPIO chip. So not the file descriptor returned, not the file descriptor associated with certain lines, but with the chip itself. And then when any of the lines that are being watched changes, it's requests requested, released, or it's config changes, we can read the strike GPIO line info changed from the file descriptor, and it contains the new configuration. So this is the line info, the timestamp, which is the approximation of when the change happened and the event type, which tells you if the line will happen to the line. And I've also noted that the structure contains padding because it's gonna be important for when I'm gonna be talking about what's wrong in the current API. So right, so by far the biggest and probably my favorite feature, new feature is the GPIO aggregator. It's my favorite to the point where it got its own title slide. So one of the problems with the character device is that it's a device file in the Unix sense, which means it has modifiable permissions, but they affect all the lines of a chip. They basically affect the chip file, the device file, but not single lines. So you cannot have just certain lines accessible to a user and not others. And this of course may be undesired, but worry not, because now you can aggregate lines in virtual GPIO chips of which permissions can be different than those of real chips. So this works like this. You have, let's say you have two real GPIO chips, only readable and writable by root. So you have chip A and chip B both with four lines. So let's say that you can, you want now to make lines zero and two of the first chip and one of the second chip accessible to a user with different permissions. So you can now with the GPIO aggregator, create a virtual GPIO chip that will have, that will basically represent the line zero, one and over the first chip and two of the second chip. And you can adjust the permissions as required. And this chip, this new chip will be visible in the system, just like any other GPIO chip. And the lines in the real chips will be visible in the system as requested by the GPIO aggregator. So the GPIO aggregator will become the consumer of those lines. So here's a little example on how it works. So once you load the GPIO aggregator kernel module, if you go to sysbus platform drivers GPIO aggregator and list the files, you're gonna see some standard attributes and two non-standard ones. So this is the new device and delete device. So let's say you call GPIO detect, which is, I'm gonna talking about it in a second, it lists the existing GPIO chips in the system. So you see we have two chips, the zero and one with their appropriate labels. This is just the testing module. So now this is documented in the documentation, the new source. If you type the GPIO mockup A and GPIO mockup B followed by the list comma separated or dash separated, if you want ranges of lines that you want to make part of the new virtual chip. If you put that line into a new device, unless there's some error, it's gonna create a new virtual file, virtual GPIO chip. And now when you, unless again, the file sits here, GPIO aggregator dot two, because the two is the index of the chip. And when we call GPIO detect again, we can see that we have a new GPIO chip, which is labeled as GPIO aggregator. And then if we print its info, I'm sorry, if we print the info of the first chip, we can see that now GPIO aggregator is the consumer of line zero and two of the real chip GPIO mockup A. So that's it, very simple to use, really recommend it. So now that we have all these new features merged recently, why do we need a whole new version of the API? So the main problem with this user API with every kernel user API is the fact that it's stable and its stability is enforced. So once released, if anyone in the user space uses it, it must be maintained. If you get it wrong and only notice once this interface has been adopted, then either you left some margin for error in the way you implemented it or you're stuck with a bad API. Unfortunately, some problems were identified with the GPIO character device and some of them simply can't be fixed. So first one is a rather uncommon use case that is running a 64-bit kernel with 32-bit user space, which is perfectly valid, but in GPIO, this has been broken for a couple of years ever since the character device has been merged, but it's been a couple of years until someone spotted it. So basically what happens is that we have this struct GPIO event data, which is read from the file descriptor created when requesting lines for event monitoring, and this structure looks differently from the 64-bit kernel and 32-bit user space point of view. On 64-bits, the structure has an implicit 32-bit padding at the end so that it's 46-bit aligned. So if both the kernel and user space have the same alignment, it's fine. But in the case of a 64-bit kernel and 32-bit user space, suddenly the kernel is sending a structure to the user space, which the user space isn't expecting because what the user space is expecting is a 96-byte structure in NAND 128. So this of course cannot work. I think it was Andi Shevchenko tried to fix it at some point, but basically we would end up with a lot of ugly code and eventually it's been decided to just fix it properly in V2. So another issue is what I mentioned before, it's the padding. So as I said before, you may, when designing a user ABI, you may leave some margin for error. Unfortunately, we didn't do that. So if the structure you're using for IOTL parameters has some unused fields, you can use them to extend given IOTL with new features. But we don't have any such fields at the end, appendage, basically no padding. Or rather, only recently with the set-config IOTL, we started to add padding for future use. We probably just didn't know any better at the time, but it turned out that only adding new flags sometimes isn't enough as a new feature. Because when it turned out that the debounce period is something that the user space should be able to configure and monitor in line events, we unfortunately didn't have any place to put that value in. So yeah, padding is important if you think that you're gonna extend your API in the future. And another thing is using the right timers. So the GPIO event timestamps, for a long time we're using the real-time clock. But the thing with the real-time clock is that it's not guaranteed to always advance. It can go backwards in certain circumstances. So the right clock to use in this case is the monotonic clock. This has been actually fixed in the following commit, GPIO switch timestamps to K-time get NS. But this technically is ADI breakage. We just decided to do it and see if anyone complains and nobody did, so it stayed like this. And V2 will of course use the monotonic clock right away. So then we had several requests for sensible new features. So we tried initially to limit the GPIO character device functionalities to not encourage people to overuse it more than for prototyping or some limited scope of use cases. However, some requests make sense. So we're okay with adding them, but it's simply impossible in V1. I mentioned before the debound period for events, bias settings is something that we already have. But then something like event sequence numbering. This is another one that's useful because if the kernel buffer overflows or the order somehow gets lost between the hired IRQ and threat IRQ handlers, it's good to keep the right sequencing in user space. Right now we can't add it because it's another field for which we don't have space and the structures we have now. And so the work has started on the mailing list to implement the second version of the GPIO character device. So what's going on? The first version of the V2 API has been proposed on the LKMA, LKML, sorry. And it's being right now discussed. The V1, of course, has to stay in place so it will be maintained. The 64-bit kernel to 32-bit user space problems are fixed. So with the missing padding, we now have the possibility to extend the structures even further if needed. And also all the flags have been reworked. And the line event and line handle requests have been merged into one because there is no good reason to keep them separate. So just to give you some examples on what we did. So the merging of the two request types is for a good reason because right now you can request multiple lines for values for doing the get or set operations, but you can only request a single line at a time for events. And every time you request a line to monitor its events, you get a new file descriptor. And you can, of course, pull all these file descriptors at once. There is no issue with that, but you have to use a single file descriptor per monitored line. And there's actually no good reason to do that. So yeah, so we may be, even in LibGP IOD, we actually wrap it in a single object. So the user actually in using the GPID, the user doesn't see that we actually have multiple line descriptors or one file descriptor per line. So what will happen in V2, we will merge these two requests together, or rather we will still have two request types, but we will always use sets of lines and operate on them. And the edge detection or edge event detection will simply become another flag. And we're talking about flags. So right now we have a single 32-bit field for all the flags. And they all, a lot of them conflict with one another logically. And that makes parsing these options in the kernel and nightmare and it's also easy to get it wrong. So in V2, they will actually be Anom's and they will be enumerated types. And yeah, we will have a separate type for direction, a separate type for drive, separate type for bias and a separate type for edge detection. And this way it will be impossible to get it wrong because you will simply only choose one from each set. Yeah, so there's a lot of small implementation details in this which I won't be covering, but we now seem to have something much better and thoroughly designed. So right now I want to just talk a bit about LibGP IOD, which is the user space, wrapper around the row, IOQL calls, the goal of which is to make them easier to use, provide lots of convenience helpers and set of command line tools and also bindings to higher level languages. So yeah, just a bit of history in the beginning. So the reason for me to write LibGP IOD was the support for power switches on Belibra ACME probes. We have this power consumption measurement device. So these power switches were basically GPIOs exposed by GPIO expanders, but I tried to model them somehow initially. First as IO, industrial IO and in this kernel attributes, but this was shut down because and rightfully so because these GPIOs were not really part of any driver conceptually. So then I switched to the regulator framework and I tried to model them as regulators controlled from user space, but this was shut down by Mark Brown, the maintainer of the regulator subsystem. And again, this has been discussed in the past that regulators should not be controllable from user space. And finally, it turned out that using the GPIO character device is the way to go. So first version was released in January, 2017. This was at the time still a work in progress effort, but a year later, the 1.0 stable API was released. Right now we have a current stable version is 1.5.1. Actually the slide says it's the two, but I'm going to make this release this week. And this release needs kernel 5.5. I'm still supporting the 1.4 series because it's the version that works with Linux with Linux 5.4 long-term support version. And as soon as we get the version two IO sub stream, we'll start working on a new major version of libGPIOD because certain things will need to change in the API in order to adapt the new kernel interface. So these changes will not be backward compatible, hence the new major release that we're planning. So what's inside? So we have a C API of the core library. It's fully documented in Doxygen. We have a set of command line tools. So we have GPIO detect that lists the chips present in the system. We have GPIO info that prints information about chips and lines. We have GPIO set and GPIO get. These programs allow to set and read current values of GPIO lines. And finally we have GPIO find, which allows you to look up lines by name. And also we have GPIO mon, which allows you to monitor line events from command line. We have a custom test suite. This works together with the GPIO mock up kernel module and interrupt simulator in the kernel. And we have, again, fully documented bindings to C++ and Python. As far as I know, at least the Python bindings are pretty widely used. And new features. So in 1.5, we've added support for the bias flags and the set config IOctl. We also switched to using proper testing frameworks. So what we do now, we use the Jilip unit testing for the core library. We use catch two for C++ bindings. We use Python unit tests for Python bindings. And then for the command line tools, we use the bus, which is the bash automated testing suite, I think. What will change in the 2.0.x series? So we're gonna have a completely new API that will work with the new kernel interface. We will introduce Jilip bindings and a debas demon, which will be the first demon to hold the global state of GPIO lines. We'll implement GPIO watch, which will be a command line tool for watching state changes of GPIO lines. So basically, this will use the line watch IOctl. And also something that the slide doesn't mention is changes in how we look up lines by name, because it turned out that a significant thing that we missed in the GPIO is the fact that line names are not unique neither for a chip. Neither globally as well as for a chip. So basically multiple lines in the chip and have the same name, nothing stops us from doing that. Unfortunately, the current version of the GPIO is not aware of that. So we're gonna have to work on this as well. And yeah, so that's actually it. It went faster than I thought. So I'm gonna be here for you to answer the questions. I'd like to say thank you to Kent Gibson who wrote the V2, so who's working mostly on the V2 of GPIO UAPI. And I'll also like to say thank you to Herth Eiter Huffe, who's implemented the GPIO aggregator, which is my really, really, just really nice tool. And that's it. Okay, some open questions. So let me just skim through the questions that are already here. So Christopher is asking, I like the CISFS interface since it was possible to easily modify it from a script. Is this still possible with the new GPIO character device? Of course, it's not possible to do it with Iockel calls, but LibGPIOD as I said has a set of command line tools which makes it quite convenient, I think. I use it in script, so definitely you should check it out. You can even monitor events and adjust the format in which the events are printed to be easily parsable by script. So I have used Epo to jade, I think, it's a name. I have used Epo to attempt to cut GPIO edges in specific hardware, IMX6. Is that reliable and does it suffer from race conditions that you have solved with your current work? Unfortunately, I'm not the race conditions. You mean the ones that we fixed with the character device in regards to the CISFS? We should probably move that to Slack. I can be of help with that, but I need to understand the question better. So what is the overhead of using the GPIO aggregator as compared to direct access to the actual hardware GPIO chips? So I haven't measured it, but basically what happens is that the kernel GPIO aggregator module requests the lines from the real module. There is a bit of overhead, but not much. I will check it out and see you on Slack. And answer you on Slack. So Robert is asking, when a line is added to a virtual GPIO chip with GPIO aggregator, is it possible to access the line through both the real chip and the virtual chip? Or is the line only accessible through the virtual chip now? So you can read line info from the real chip, but you cannot request the line. You can only now request it from the GPIO aggregator because there can only be one user of a line and conceptually the GPIO aggregator becomes the user of this line of the line on the real chip. Drew is asking, is it a good idea to declare in device tree that I want to use certain GPIO lines from user space? It's a good question, I don't know. First, I don't think we have any means to do that. We probably can use the name for that that it's somehow used, marked to be used from user space, but right now, no, it's not a good idea because device tree is describing the hardware. I think that every time something like this pops up, Rob Herring is against it and says that device tree should focus on describing the hardware and not describing the implementation details. So I modify my answer, it's no. And Drew, again, on BigoBone, I can set the GPIO max mode and pin control device tree properties, but how can I specify that I want to use the corresponding GPIO lines with the GPIOD? Is it useful to specify? Yeah, so this is the same, this is related to that question above. In DT, this shouldn't be specified, basically. And Drew, one complaint I've gotten from users is that the GPIOD user API has no way to keep a line set after the process ends. They said this makes SysFS better for them. Any solution for that scenario? Yes. So this is something I mentioned. We, many users want something to store the global state of GPIO lines. I have, this has been in my to-do for far too long, but I will at some point implement a very simple demon written in C together with a client that will hold the global state and expose it to the client. And basically, as long as the demon will be active, you will be able to set a state of a line and keep it that way. Also something that is already, that I've been already working on for some time now is the divas demon and then set of divas bindings. This of course will keep the global state of lines, like the state of lines will be kept even after the client program exists, if it's so wishes. And yeah, otherwise this is how character devices work in Linux and this is by design. So yeah, the most we can do is implement a demon that will stay alive and be an intermediate between the GPIO character device and the client programs. So I think for now there are no more questions. I will stay on Slack of course, to answer any more questions. So if we have time, I can go back to the questions that we had before, answer it live. So I have used EPOL to attempt to catch the edges in specific. So this was the question from Jade. I will read it again. I have used EPOL to attempt to catch GPIO edges in specific hardware, is that reliable and does it suffer from race conditions that you have solved with your current work? So if you've done it using CISFS, yes, this is racy and it's not reliable. If you switch to the character device, it's now going to be more reliable. It's not 100% reliable because the buffer in the kernel can overflow, but it's gonna be much, much more reliable than CISFS. Another question from Drew. Any other bindings in C++ and Python plant? Yes, so Kent Gibson who wrote, who implemented the set conflict IOQL and the bias then support for bias flags works on his Go library. It's not binding, basically it's not related to the GPAD, it's a standalone library written in Go, but I know it's functional and Kent is actively supporting it. Google for the name because I don't even know. It's called GPIO simply. It's a native Go library for Raspberry Pi GPIO. Oh, actually it's for Resbia. Okay, so it's called GPIOD. It's a native Go library for Linux GPIO and uses the character device, yeah. Okay, so if there are no more questions. Okay, so there are no more questions. I think this is it and I'll see you on Slack then. Thank you very much.