 Cool right on time Right. Hi, everyone. My name is Fabio. I work for Google in the Chrome OS the EC team By my email my username on github is Fabio Baltieri Fabio W on discord I'm gonna talk about the input sub system that has been recently introduced in Zephyr 3.4 So yeah, it's open with the most important question where we doing this doesn't our thoughts really need An input subsystem is something that's been asked by real users working on something. So let's try to There's no review of the API we had for input devices in Zephyr up until a couple of months ago It was pretty much this it's something called case scan But the name implies it was designed for keyboard scanning API's We came perfectly well for what it was meant for which is a keyboard scanning device We throw column and key presses the API was essentially a callback that would give you like X Y and press coordinate So few shortcomings of the existing API for a start you can only register a single listener if you're using an RTOS You're probably want to do more than one thing at the same time. So that's a bit of a limitation if you need multiple this dinner Say for example, you want to dispatch events to a USB channel or a Bluetooth channel or some other management thing You need to do it at application level and there's been some solution. That's been developed. We'll see them later Not a limitation the the API has been designed for a keyboard. So it's Limited into anything that you can represent in grow column and presses if you need anything something more You can't use it if you need something less. Yeah, you can throw away some of the data At least that row and column are 32 bits. So you can have a pretty substantial keyboard Some of the API became redundant and as the device power management stuff were introduced So one thing it was using was the enable and disable functions are not really needed They can be implemented within a generic way at driver level So even what that could could be removed This is some example of all the keyboard scanning PI has been used by existing driver that are not necessarily Keyboard scanning matrix. So this is a cap sense Controller chip. He doesn't have rows don't have column. So yeah, the callback is just zero column pressed. Okay, not too bad This is a more interesting one it became popular when lbjl started to be introduced and there's touch screen and a touch screen produces XY and press events So, yeah, sure. He's not he's not a keyboard anymore. It's a touch screen now. This is the same API Then there's been some other cases where we didn't even try to reuse the existing case can API For example, when when we designed the GPIO keys like the driver for GPIO with debouncing It hasn't been even tried to conform to the existing API So what we started doing is defining new subsystem specific for a single driver Just to represent the specific type of device So if you go ahead in this way, we end up abusing the subsystem stuff And we end up with a proliferation of subsystem that are actually a single driver. So that was not good either It's one of my favorite example. So this is maybe harder to follow. But yeah, so top left This is the driver for the quadrature decoder driver on the nrf 52 now if you know anything about those chips They're normally used for implementing mice and keyboard So you can imagine that the quadrature decoder is used for scroller wheels now a scroller wheel Is a quadrature encoding device with the tent typically every two or four Steps of the thing so it's been implemented as a sensor But now the sensor API says that you need to have rotation in degrees So the driver gets defined with how many steps are in the whole wheel But we really want to know how many steps are in a detent. So at application level We read the sensor we take back the angle in degrees and we divided by the angle of detent To obtain back the step that we could specify directly on the device driver. So Stuff have been developed with what we had and people jump through hoops to try and Yeah, I make what they could with with what we had available There's been some previous attempt on Defining a new generic API to super seed case can so a couple of example This is something that has been proposed as a touch screen API because of some shortcoming of the existing one people who have like multiple finger Yeah, like gesture tabs things like that So the main comment is we can't have subsystem that are too device-specific because we would end up with Gigantic API that's only a few of the driver would use and On the on the opposite side of the spectrum we had proposal This was actually a bit closer to what we ended up getting the proposal that that were fairly application-specific in this case They on the core of the subsystem We were getting a request to implement stuff like long presses and the comment Is from the ZNK project container which can't know something about input devices if it has keyboard and it's like, yeah Like long press why not double tap triple tap and it makes sense So we need something extensible, but we can't really measure something that is clearly designed for a very specific thing And in this case you could guess it if you know what the guy was doing So we were ended up in this kind of deadlock situation where the the maintainer and the reviewer were complaining about Pull requests that were not making too much sense that really needed a subsystem that was generic and couldn't handle this situation But then we were pushing back requests for implementing thing saying that we need a subsystem and the user A user is implementing a driver is not too happy if we ask them to implement an entire subsystem That's not kind of a reasonable feedback for a normal pull request They just want to get their driver done and move on and do through application So a bit of a study of What we have in the in the in our environment well linux as a input API That's the worst if you have never used it. It's fairly simple So this is just some snippet of a driver. That's fairly easy to understand. So it's linux ton of memory So the first step is allocating Structure device whatever it is it's opaque. It's an opaque thing inside the kernel. It just does what it does We don't really care about this then then In linux device register a set of capabilities and attributes the stuff is used for dynamic discovery of devices So this is also something that we ended up not really doing at the end then at the end of it We do register the device This is what exposes it in the user space in the form of some slash dev node and notifying all the event system And then the only thing that the driver has to do whenever it has to report some event is to use the simple API that says report an event and whatever it is there's like an event type and Yeah, like two types and the national value and The important thing is that this is meant to represent every type of input device including device that have like composite Composite entities and that is done by implementing a sync type of event So in case of for example of a dash tree and you would report Not so the text absolute why and then a push event and and then sync later and the sync Tells the application level that the previews entities are Synchronized between each other and they can be used together. I'll have an example of that Yeah, so on down the user space the the API is pretty simple It's just a structure to have the event type code and value and yeah So this is an example of a mouse and what I was mentioning the second block is a diagonal movement You will get to you get two events for an X and Y and then a thing tell the system. Hey, these two came together So It's after implementation we basically decided to take the concept that we could reuse from Linux one and Simplify some part make them fit the Zephyr model a bit better So requirement we needed something that would support a many to many model We may have multiple devices in the system and then we want to we definitely want to have multiple listeners complex device You may have Yeah usb bluetooth Management endpoint like idle detection code, whatever Zephyr is modular. We want the stuff to be modular So we want to have many callback Listener registration at build time. It's something that is fairly important. It's a key concept of Zephyr in the modularity of the project the listener has to register and Get the callback inserted the build time so that we can exclude the file and then the whole functionality goes away We don't have to Change any other code up or from the specific one We had to be back where compatible I'm going out of order. We had to be back for the compatible with case can case can is a stable API We can just break stuff and kill it It needs to be generic for represent Generic enough to represent any type of input device and we get that by copying the Linux one because it already works And we need accessibility because there has been demand for the stuff like long presses double-fig So this is what we came up. It's heavily inspired by the Linux user space data structure We embedded sync just because in many times you don't need it and to keep it a bit more lightweight You can still send a dummy event was just to think if that's what you need We added a structure device Which is kind of the equivalent of opening up slash dev input Slash dev device know that it can be used to identify which device generated event for filtering On the right, so this is an example of a device driver is implemented There's no registration no capabilities for some bad device. I hopefully know what is going on there There's not even an API the normally Zephyr devices have a subset of function that device has registered we Didn't need to put anything in this point power management stuff is can be handled separately. So Yeah, that's what we did so advice can just Start enable itself and then start sending Report using those API which are pretty much a copy of the Linux one and They have to specify a device node if available And then yeah, the code the type and the value and then the synchronization bit is Include it's bound with any other event and the time of thing that can be used depending on how that Actual event processing is implemented. I'll talk about it in a bit So this is an example of a device that have to report multi that have to send multiple reports that are synchronized and this is specifically out that string can be implemented in this situation you send Couple of values for the X and Y touch and then for the third one you only for only on the last one You enable the sync and then the application level is going to see the sink is going to cash the previous value see the sink And when you send the sink it understand that the previous values are stable and they can be used The the the application API is very simple so you just register a callback and The the callback is registered with a macro and that is processed as build time as I said So there's nothing else to do the first parameter is an optional the stroke device pointer and that is the only type of filtering that we provide any other any more complicated filtering get to has to be done at the application level just to keep it simple and avoid having complex stuff in the In the callback data structure And yeah, no sub system API Another thing that linux does that we did not do in disaster one is linux as Duplicate duplicate event suppression if there is a driver and you keep sending the same event over and over again It doesn't get processed if It doesn't change. This is very expensive memory wisely in access dynamic memory. So that's easy for them, but we don't have that So there's two mode of operation right now. Well, we're gonna be more later So synchronous It's super light to it. Basically it it falls back to the previous case case constitution When the caller when the driver reports an event or any other type of code reports an event all the callback gets Called in the same context So, yeah, this is the the lighter weight mode of operation and the the thread In the thread mode the the system spawns a trade automatically and then all the events are put in a message queue and they are processed Asynchronously in a dedicated trace. So this can be used If the if some of the callback need to sleep or do some more complex operation So here here's the idea with this If you're dealing with a very simple application, you want a minimum overhead You don't really want to have a dedicated thread for this Maybe you just have a couple of buttons that you need to bouncing Well, just have a callback and typically these are executed in the in the work queue anyway So you can use the synchronous mode If you have an application that is doing something more more complex and may need to sleep or lose a bit more time Um In the callback, you can use the thread you don't have to instance your own thread But then the more complex type of application is where you probably have an event system that you want to Connect to and in this case You just want the the simplest way possible of in of taking event back from the input sub system and injecting them Into your own event Manager and that is how you would fall back using the synchronous mode So case can put compatibility. So this This api can represent case can devices just find there's like just there's only a row column indicator and And a push so we what we did to maintain compatibility was to add this case can input translator node And this is documentation in in the dts binding and it can just be added to an existing device node And then we can point to chosen keyboard scan Handle to the new one and existing application are gonna or are gonna keep working unmodified So what we're doing right now is going through the existing driver that should Will like touchscreen while all of them in practice moving them into the input subsystem Looking for all the board where these were instantiated And do all the magic for this thing to work So the samples work out of the box nothing should break if you have a downstream application You have to do this to for that to keep working um This is how the case can compatibility Implementation work and it shows the bit I mentioned about multiple entities synchronized So it is very simple like you take the event you save it and if you know that you have to process multiple ones You only trigger the downstream event when the synchronization bit is set So the previous one can come out in in in any order as long as the sink is at the end Uh extensibility Not in special in the end The the codes are just numbers So we just decided to reserve a range of those that can be used for vendor specific stuff and then you can put whatever you need in that Um filter so this was the the answer of that comment you saw previously one of the proposed implementation So there shouldn't be Applications specific features at subsystem level right that doesn't make any sense But we should have a solution for implementing them cleanly and in a reusable way because this is what we do Uh, so this was this was the proposal. Um, the just now in the stops you from taking events and Producing more events. So This is something that we upstream took over the specific case that was Requested in the previous implementation for having like long press events I mean it's kind of something that is useful But it was mostly provided as a demo of what you should do in your application if you Need to generate more complex event And any three usable code. So in your device tree you can instantiate one of This block and this is going to generate the code that is going to take the input device from a real device and then Generate more input events as a reaction and that's how you can you can implement long press in in a more complex way Are we gonna accept upstream on this front? I'm not sure we have to discuss So we're gonna get like a longer press quadruple press triple click probably not But it it's good to have it as a reference and it does testing as well It shows how to do Yeah, so that is it. It's a small parentheses on event system. This was The main topic of conversation during the review of the proposal So essentially the sub system is implementing its own mini lightweight event many too many event system The obvious question was why not using zebus directly and the zebus is the official zephyr message bus Uh has been introduced recently. There's like a review project that uses it. And yeah, it doesn't make sense We decided not to have direct integration mainly because there are application out in the wild that uses their own Event subsystem and it would mean a bit awkward to force everybody to another event system just so that they can Send stuff from an event system to another event system um, but This is our Event system for the project. So we're going we're probably going to have out for mode in addition to synchroneous and thread and we're going to make it easy we're going to First party support for sending input event through the zebus system and then the application can use it The other two that have been looked into is obviously the zenk one. So if you're not familiar, this is a fairly popular project for mechanical zenk case test for sensor zephyr mechanical keyboard. So Obviously here's something out on how to do keyboard and Yeah, we we looked into this interesting thing about this message system is that it used dynamic memory and what they do for input Management is that they take the event from case scan They put it in a queue and then they Send it down the line and the event system is a synchronous. So this should this is pretty much what the subsystem does Anyway, and it should be really easy to integrate and another detail. I forgot to talk about is The out of memory handling strategy or lack of the rough Yeah This happens every time you press a button It's fine any idea the other one The other system that we looked into is the Nordic connectors decay because again, this is very popular for for building my keyboard Yeah, same story a synchronous system Dynamic memory. Um, yeah, this one uses the free memory quickly strategy if something goes wrong Cis reboot Apart from that, there's no case scan integration because for some reason the example that they have for the Nordic desktop Does not use case scan the add drone code. But again, um, yeah, same story like If they want to integrate they could simply get a synchronous handler and inject events back Okay So what's next for the substance? Well, what do we have now? Um, yeah, the subsystem is upstream. It's usable it landed in 3.4 There is one example filter stir for long press. We ported over a bunch of drivers. I think there's a new one since a couple of days ago Yeah, we're working on it Once the driver have been ported. We are looking to zebus integration Maybe Rodrigo will send some patch for that and move at the same time I'm personally working on some common code for Keywords can in matrices it would be brilliant if you could use that for cmk as well down the road whenever they'll manage to up rev 234 or whatever Lvgl We have lvgl as a module right now. It takes case scan event. It works with the compatibility driver We wanted to take input event natively and then we could also not only use touch screen But also like encoder wheels and buttons lvgl says some fancy stuff that now we can support So we'll work on that and then ultimately we want to deprecate case scan um I don't own every Possible touch screen display evaluation board in the market. I'm trying really hard, but I don't so yeah if you can If you don't run and can help out Sending patches or testing out that would be it'll be cool And yeah, that's it if you have any question or or any input Yeah So Just well actually I have a question as well Just very briefly the the thread mode versus the synchronous mode and adding z bus is a great idea as a as an option And then one more suggestion if you want to get inspired as well in the bluetooth subsystem for events What we do is we also support providing your own Work queue because some some applications have their own work queue and you want to reuse that to process things Like bluetooth events or maybe uh input events and the other one is uh, why not also Also provide the the possibility of using the system work queue as well in some case that might work Yeah, absolutely. Well most I think all of the event that we have right now in in in the driver already start from the work queue But it would be it would be a great one especially because it would then it would mimic exactly what the mk does And uh, yeah, no good to know. Um, yeah, like I I'll I'll expect a pr on my desk by monday morning. Thank you. Thank you Uh, thanks for picking uh, linux as the as the source of inspiration. I think that's a good job That's the easy way of getting stuff approved. Oh, yeah, you're already doing it. So can't be that bad, right? I did I was some intimately familiar with some pieces of the linux inputs that doing drivers for graphics tablets you know drawing tables in the digiment project And uh, I'm curious like I didn't notice any way to discover the extent of an absolute positioning device like the ABS x max and x mean Like that that's what the capabilities are did I missed that? Yeah, let me let me let me try find it back um The bits where it says absolute x and y sorry, where's it? No, that's not right and It's right on the one that where it says absolute Uh, um, yeah input a set abs params that 0 to 55 are the minimum and maximum for the specific axis But we don't have it in zephyr anyway. This is uh rx This abs rx Yeah rx and r y are just that what x and y Okay, well, I see that it's handled and then the user is able to query that Yeah, when you when you when you open a device slash dev you get this one dumped into your file descriptor. Okay, cool Thanks, uh, just just one common frustration with the linux uh input stack on my side that would be nice to avoid Is that there is some protocols that are kind of out of band like pieces of the input stack are out of band Regarding that interface the even dev interface and the linux Where things like capabilities and how to interpret actually the event stream Discovered through various like use bid's And the yeah deboss all that stuff like Like and that's I understand what's done. Why is that it's done? But it would be nice to have these things documented. I'm sure You will come up with things that are like, you know Don't fit the existing protocol later So yeah, but the big part when you try to use that actually is trying to figure out How what what has happened and why this is working? Why this is not working? So yeah straight from the get go like I would ask to document those cases Yeah, but for zephyr you go with the assumption that the system is known at some like we we don't have Excuse me. We don't have capabilities in zephyr right so far at least But yeah, yeah, and linux is a bit of a different story, right? Because you can like Whatever can happen Yeah, yeah, just just saying that like this protocol probably going on last year a long time But once once you step out of that, that's better be decremented. Yeah, sure enough. Yeah, thanks Thanks for a comment Hey fabio If you work actually remember reviewing this, uh, but it's been so long now I can probably have to ask questions. There's a refresher but good So first of all, uh, just a comment on the you know, how you were mentioning the k malak situation inside of the macro Yeah, I would I would do there is just externalize it and have a generic allocator Passed in and then it's just like a functor or whatever you want to call it Um, and then you can use like, you know fixed size pool or whatever But the other one is like why why are we deprecating case can can you just remind me? I have no idea. Why not? I'm actually No, I think the the the conversation we were having during the proposal was why having two api that can do the same thing There's there's no reason for We don't have to kill it It would be awkward. We can just put a message say, hey, don't write new case can driver But no, we could keep it. It's a conversation we can have down the road I just thought I'd throw another one out there for an idea Just in case like you needed something to do because you have infinite bandwidth, um, of course Uh, and I got a reminder from Maxim. He's one of my medibates, uh Swiping swiping support Um, so not a single character at a time But like you do an fft on a pattern on a touchscreen and you get a phrase or some kind just uh Yeah, I think if you yeah, if you could do some some clever filtering And and make a point that it could be useful for multiple further projects That'd be a good selling point for sure Yeah, the inspiration for that was kind of the Input input library something something I can't remember jar send me the link and I thought this is cool Let's do something like that. Yeah. Nope. Yeah, or they could be a module for all of this stuff that would also work Thank you. Cheers. Thanks Cool. There's no other question