 Hi, my name is Bartosz. I'm with Baidibre French-based Linux consultancy and today I'll be talking about why so many Linux plumbing layer components end up having multiple incompatible major versions and what can we do to avoid it. I'll be mostly talking about very specific things that make kernel or user space library interfaces impossible to improve without breaking compatibility contracts in general what mistakes one should try to avoid in order to keep libraries APIs in general extendable in the future. Baidibre is not so small now. We are around 50 people. We are a Linux consultancy based in southern France. We help clients launch hardware and software products and we help them through the entire process from concept to manufacturing. We work with the Linux community extensively. We upstream support for several hardware platforms and we are the founding members of the kernel CI project. Personally I have 12 years of experience now. I work all across the stack from kernel space and even Bermedal and bootloader's RTOS up to low-level user space and Linux build tools like Buildroot or Yocto. I am the author and maintainer of LibGPID and co-maintener of the GPIO kernel subsystem and I also contribute a lot to different open source projects including Yocto. This talk is based on lessons learned while working on two open source components. The GPIO character device exposed by Linux for every GPIO chip since 4.6 and the user space library and tools that we provide in order to wrap the kernel interfaces in an elegant and easy to use way. A bit of history first. The GPIO character device has been introduced in the kernel as a replacement for the deprecated CISFS interface and LibGPID was released in order to provide a C interface that is easier to use than role in a PsiOctol calls and that would be the base for the command line tools that could be used in scripts with the idea that this would make it easier for people used to CISFS to switch to character device. And if you look at the UAPI header for the GPIO character device today you'll see that the number of symbols is marked as GPIO underscore V2 and if you look at the mainline git repository for LibGPID at kernel.org you'll notice that there is a branch called next slash LibGPID V2 and that it is under active development at the moment. So what exactly happened? Why are we creating a new backward incompatible interface for GPIO only four years after the first one was released? So before I tell you I'd like to discuss what different types of compatibility we are exposed to in Linux. Some of those things I described may be obvious to many folks but there are unexpected caveats when it comes to binary compatibility that even experienced Linux hackers find surprising especially if you venture beyond C and go into the realm of C++ or Rust. And there are also a lot of kernel developers who rarely venture into the user land where issues of compatibility are more pronounced than what you deal with in the kernel. In the kernel it's quite simple, there's no stability of internal interfaces and indefinite stability of kernel user space interfaces but this is actually a bit more complicated than that in the user space. So let's take LibGPID as an example. LibGPID talks to the kernel using Ioctl system calls passing around a set of structures defined in the relevant user API, in the relevant user API of the exposed by kernel and on the other side it exposes a C header containing a bunch of symbols and a shared library called LibGPID. On top of that we've built the C++ and Python bindings and well Python bindings are written in C and C++ bindings are written in C++. So the important thing to note is that the C++ bindings expose another shared library called LibGPID CXX to make things complicated. But this is all pretty normal stuff, most system-level components are written in C and compiled as shared libraries and expose C headers to interface with the library and while the C standard does not define any ABI standard Linux has a de facto standardized ABI. So we're now concerned with two types of compatibility, source or API and binary or ABI. The general definitions of the two are simple. For the former two versions of a library are source compatible if any program using the exposed symbols can be recompiled with both without updating its source. And for the latter it's a bit stricter, a program linked against the library is guaranteed to work with both versions without rebuilding the program for the versions to be ABI compatible. And so it truly is like both ABI and ABI truly are just contracts and this contract can be honored, extended or broken between versions. And when considering the differences between two versions we can have three situations both for API as well as ABI compatibility. So we can go from A to B or one to two without any change in the interface, for example the only thing that changed is some internal implementation details. And we can go from A to B and add new interfaces, for example we add a new function that makes A compatible with B, so program using A will work with B but programs using B are not guaranteed to work with A. And finally we can go from A to B and remove a certain function or replace it with a different one or change the arguments or modify a visible structure or change in values and in that case programs, in that case we break all compatibility and programs using A will not work with B and vice versa. So API and ABI compatibility are independent from each other and one can be broken without affecting the other. For instance if you extend the size of a struct but don't change its previous layout or rename the numbers, you do break the ABI but you will still be source compatible. If you on the other hand rename some preprocessor defines in the header but keep the old values or rename enums but keep the old values, you do break the source compatibility but not the ABI because a program linked against two versions with two versions of libraries built with different symbol names that retain the values will work. And so back to live GPIOD and the character device. When the first version of the character device and subsequently the first version of live GPIOD was released it supported a set of features that we wanted to expose to the user space because obviously we knew what should be available to user land and with GPIO features should remain confined to the kernel except that those SQ users started coming up with annoyingly valid reasons for extending the interface with features that we didn't plan for and then we faced the reality of our interfaces both in the kernel as well as in live GPIOD being badly designed on several levels. So example in the structure passed between the kernel and the user space we left no reserved space for future use while we could potentially add new flags like new values for the flags it wouldn't be enough because users at some point started asking us to add for instance the debounce period setting because what is reading line interrupts in user space good for if you can't set up the debouncing. Another request for line events was sequence numbers because what good is bitbanging implementation in user space if you need that if you can't assure the right ordering of events in the queue. So already we are in a situation where we could introduce new feature we could not introduce new features to the interface that would not break the ABI contract but we were in the situation where we could introduce new features to the interface but that would but we basically locked ourselves out of that possibility because we didn't think about it in advance and we didn't simply leave any reserved space. So this is the lesson one that we learned always add reserved space to your exposed structures. It costs very little unless you go above and know that the memory page boundary it unless you do that it doesn't really cost anything and it allows you to extend your interfaces in the future without breaking the contract especially when designing eye octals because in user space we could we can do a better thing that I will discuss in a minute but lesson one learned in lib GPOD when sorry in the GPIO character device interface when we designed it we left some padding so that well right now it seems as if we are done that the API is complete we really don't want to expose any new features but I have a feeling that someone at some point will come up with an idea that we will agree to implement and for that we're already prepared every structure that we are exposing in the V2 interface has already some padding we were we should be good for most part. So lesson two is related to to the problem described before as you could see we have so if you look into the header for lib GPOD you're gonna notice that we have the same problem several structures are exposed as in their layout is defined in the header that means that the user can allocate the structure on the stack and their layout is exposed they are exposed because at the time this was my first serious library that I was designing I didn't know any better and I feared that making certain structures opaque would impose a penalty if they were allocated on the heap exclusively like for like for instance the mentioned line event structures they normally would need to be created a lot because there can be a lot of line events coming up in the queue and other concern was that they that if I made certain structures opaque they would be very complicated to use with C accessors and in this case I'm talking mostly about the config structure that is used to set up a line request a GPL and line request before translating it into the kernel version and passing it over by over the I have to call so I settled on having certain structures visible to the user and of course this was a bad idea in fact it's a well-known pattern in low level Linux libraries to make all objects opaque and this really is the right approach as not making any guarantees to the users about the layout of or the size of structures saves us all the all kinds of problems with the ABI but what if you know that certain structures will be allocated a lot and you don't want to impact the performance by calling into the allocator all the time do one of the ideas and this is what we used in libGPairD is using opaque containers so we have a container object that is allocated on the heap and then when we and then it stores static like it stores already pre-allocated structures that we can reuse all the time and they are not being reallocated again the lesson learned so this is maybe an example that gives you an idea of what we did so before above this is what we had we had a user who wanted to create a buffer for line events needed to allocate it on the crude allocated on the on the stack and then simply call a function that would read the events and try to translate it then from the kernel form into the user space for and put them in store them into into this visible buffer so what we do now is that the user now has to allocate a an edge event buffer and then read events into that buffer and then this is the only time that we actually allocate anything here and then when the events are are read they are stored in this buffer and the user can actually get handles to the opaque event structures and act on them without reallocating them all the time so this is obviously an improvement but what about other languages so we also support Python C++ for Python it's clear Python doesn't have API and also in Python every object is allocated on the heap so this this is not really a concern here but in C++ the class layout is visible and in this case we just went with the pimple idiom so that classes look like this they expose only their methods and use a smart pointer to store a pointer to the implementation class which is all which is actually hidden so yeah the the the the the class the public part of the class only contains a smart pointer and the implementation itself is hidden from from the from the user and what we do for edge events in this case because we wanted to replicate this very efficient way of allocating the event buffer we actually used a bit of polymorphism where the the edge event classes that are stored in the buffer are not managed but as soon as the user copies them as long as the user just reads their values that that's fine they're not being copied but as soon as the user copies them then they become managed with with a smart pointer so just a nice trick to have something efficient replicated sufficient behavior in C++ so lessons 3 make sure your expose structures work on 32 and 64-bit architectures so this I'm adding the asterisk here because while for most part you don't need to be concerned about the layout of structures as long as you're not sending them over the wire when you're defining them but even then you if you're sending them over the wire you should probably not really do it yourself you should use some standard way of marshalling the data but in general when you're defining a structure you're not really concerned with padding with alignment and all that this is done by the compiler except when okay except yeah except in certain surprising circumstances so let's take a look at this structure so what is wrong with that so it only has two fields and this is the structure exposed by the version one of the criminal interface for the GPIO character device this is the event data that would be actually read by the GPIO D from the kernel so what is wrong with it at first sight nothing is wrong with it on 32-bit architectures the structure will be aligned to 32 bits and this if we look just like this and on 64-bit architectures there would be additional padding at it so that the structure would be aligned to 64 bits and this isn't a problem if your kernel is 64 bits and the user space is 64 bits but we also need to support Linux that support compatibility mode for 32-bit user space that the kernel is 64 bits and the user space is compiled in as 32 bits so we actually encountered this situation where we would send events from 64-bit kernel to 32-bit user space and the user space would be completely confused because the kernel would send structures that would be 128 bits in size and it wouldn't expect it it would expect 96 bits so how do you fix that all again just make sure that you pad your structure to the correct size especially if this is the structure that is exposed by the kernel you have to think about the compatibility mode for v2 we made sure that all our structures especially those that can be retrieved with the read system call be correctly padded and work with this specific compatibility case next lesson some surprising details can break the ABI so it turns out that the structure layout is not the only thing that's considered to be part of the contract between the kernel and the user space so everyone knows one of the most famous Linux runs about how we must not suddenly change the error codes returned to user space in hindsight this is a no-brainer however we had a quite a similar situation over at the GPIO subsystem so the clock that we used to timestamp the line events in the kernel that would be then exported to user space was the real-time clock at some point it was determined that it uh oh it's always that a real clock is not a real-time clock is not is not guaranteed to not go back or to move back and forth in time and that it wouldn't really so users notice that we don't really convey the right information to the user space about the timestamps of the events or approximate moments in time when the events would occur so we decided to change it to the monotonic clock and soon after we had a user complaint about the change because they actually relied on the timestamps being derived from the real-time clock fortunately the user adjusted their code and didn't require us to go back but if they insisted obviously we would need to keep the interface stable and you would have to revert that change so fortunately we are still uh we still have uh we retained the monotonic clock but what we did for version two we simply added a switch so that the user can can decipher themselves what clock they want to use so right now we support the monotonic clock uh for event timestamps real-time clock and also we have an upcoming support for for the hardware timestamping engine in it's it's a new general thing in feature in the Linux kernel but we will support it in GPIO for our event timestamps so lesson five don't try to keep the user space data models the same as the one in the kernel so the goal of user space libraries working close to the kernel is to represent whatever the abstract data model kernel uses in a way best suited to user line and oftentimes the kernel representation is quite specific to the problems faced inside the kernel the problems may be different in user space and also user space accesses the data structures in a different way so as tempting as it may be to stick to the to a similar representation it's often better to spend some time on designing a bespoke model of the data structures for for the user space so what you see here is the representation of the GPIOs in the kernel so every chip like we have an example chip that has exports eight lines and what drivers do in the kernel they they call GPIOD get and they receive a GPIO descriptor which is a pointer to a structure defined in the lib GPIOD code and also users can can call a variant of this function called GPIOD get array and receive a container for multiple descriptors what happens now is that the GPIOD get function actually does request the line so that the lines become owned their ownership is passed to the using driver and then nobody else can can request them so this is this is what we do in the kernel and what I thought would be best for version one of lib GPIOD would be to simply follow that model except that there is a slight change so we have we still have a chip and this is the chip that we open with GPIOD chip open unlike the kernel in the kernel the users don't really need to open a chip but in this case we are opening a character device so we have another layer and then a user can call a function called GPIOD chip get line to retrieve an opaque pointer to to the line object and every line object represents a line and now this line is not yet requested this is unlike what the kernel does this line is not yet requested right now we only have a handle that allows us to read information about the line so so we can use we can read its state its direction its various flags and we also can retrieve multiple lines and then then they are stored in an object that we call the GPIOD line bulk and the same they are still not requested so now only now the user can call the request function GPIOD line request or request lines and only then they become owned by the user space process so what is wrong with this so the thing about the line information that can be retrieved from the GPIOD line object is that it's when the when the user space reads it from the kernel it's it really is only a snapshot so we when you read a single flag of the line a single setting of the line you're it's not guaranteed to retain it this value you know in in any time in the future so this is a problem because there is so until why why we do have a certain way of notifying the user space about changes in status of lines it's not very obvious from the API of the library and you would have to unless you set up an event loop if you just have a very simple program you would have to reread the status of the lines manually so we decided for version two to have a completely different model of of the like a completely different data structure model but we still have a chip we still open it with GPIOD chip open but what happens now there's there no longer exists a a a class of objects that would represent lines what we have instead is we have objects called GPIOD line info and it's very explicit that they don't contain up-to-date info that they contain a snapshot of information retrieved at the time when the call was made to GPIOD chip get line info so this is how this is this is an improvement over over the previous version because right now it's very clear that all the settings that the user can read all the flags that or various values that the user can read from this line info object are only a snapshot in time and every time so we still have the mechanism to inform users about changes in the status but right now they receive every time a status occurs they can read a new line infrastructure with up-to-date information so yeah this this is how it works right now for the line info and also the request line requests are made in a different way so we are no longer operating on separate lines because it no longer makes any sense we request entire sets of lines we have a function called GPIOD chip request lines it takes as argument it takes a like an elaborate config structure but right now we have we like we do the request for a set of offsets it can be a single line but we're never operating on single lines we're always doing request for multiple lines at once and this allows us to so now the lifetime of the GPIOD line request object is detached from from the chip so the chip can be closed and the request can be kept because it's actually associated with a different file descriptor so this is this is an idea on how we completely detached our user space data model from the one in the kernel so i i encourage everyone to just try to give it a try not to follow the kernel data structure model to the letter and so the things i described here are very specific to our project i i hope they're they're they're useful but they're pretty specific to this very very narrow set of interfaces but there is this project that i actually i i i sadly learned about it after i designed and released the the first version of the GPIOD but it actually i'm talking about the libabc project which is a a dummy library that simply shows the good practices of designing user space libraries for the plumbing layer in Linux so it contains some things about which i which i spoke so that the that the all data structures exposed by the library should be opaque and should be accessed with dedicated functions that there should be no global state in a library because this way it can be made not not thread safe but thread aware etc etc so this is a good read i i posted the links to it here and that's it for the presentation i'm open to discussion thank you very much i hope this was useful the things i i spoke about here are actually something that i learned the hard way um and i really hope that the second version of libgpav once we release it will be at least resistant to needing to have another incompatible version in the future thank you