 OK, you can see my screen now, right? So yeah, this is the input handling update. Well, I'm going to talk about some things that will be coming up in Qt 6. You have the, yeah, my handle on IRC is eCloud, and I hang out on those couple of channels sometimes, although we're not using it as much in the Qt company anymore because they switched to Microsoft Teams, of all things, so I'm not in IRC as much as I used to be. I've been a Qt application developer since about 2004, joined the Qt company in Oslo in 2011. And since then, I've been, I knew just right away that there were some problems with our support for touchscreens and Wacom tablets, and so I've been kind of focused on that over the years, but also quite a few other things. Worked on Linux and macOS and Qt PDF and a bunch of Qt Quick stuff, and now I'm one of the Qt Quick maintainers. So the goals for the agenda for this session, I'm going to talk about the goals, what we wanted to change about input handling in Qt 6, and the API changes that seem to be necessary to make that possible, and how it relates to Qt Quick, and then there will be a kind of a demo, although nothing, yeah, well, you'll see. And then some remaining work, and I don't know if we have time for questions at the end, it depends how fast I go. So the goals, Qmouse event had a mouse event source, which we added in Qt 5.4, and we thought that that would help us figure out the cases when a mouse event is synthesized from touch, you should be able to find out somehow that that's what it is, right? When you're handling the event, but it turned out that that's not really enough. So in Qt 6, I wanted to have a whole device hierarchy and every input event should have the device that it came from. So you'll be able to get all the possible information about the device, and so you can tell if a mouse event really comes from a touchscreen, you can see which touchscreen it is. We want to keep widgets and quick stuff working the same as it did before as much as possible. We want to have the ability to write common event delivery code for all the pointer events in Qt Quick, and probably we're not gonna change widgets, but we could if we wanted to. So we wanna have the ability to write agnostic delivery code that it doesn't really care whether it's a mouse or touch, it knows what to do with it. Yeah, we had prototyped that in Qt Quick in Qt 5 by writing wrapper classes for the events. So we should be able to get rid of those now because we'll make the events just look just like them. There's always been some problems with FlickaBall in Qt Quick. It didn't really handle touch directly, it only handles mouse events. And so that means if you have set the press delay on a FlickaBall and then you want to have it do a replay later on, it's always gonna replay a mouse event because it never got the touch event, it got a synthesized mouse event. So that's kind of silly, we need to fix that. We want the event objects to be as lightweight as possible like they always have been or even lighter. And there's other bugs besides FlickaBall that we hope will be easier to fix after this stuff is done. Like I say, I always try to make the welcome tablet support a little bit better, but the main thing that's left is the proximity events. When you hold the stylus close to the tablet, but you're not touching yet, those need to be delivered fully and they're not yet, they're just delivered to the application, not to the window. So we'll try to fix that. And maybe we'll try to work towards having multi-seat support eventually. That's not gonna be something that's gonna make Qt 6.0, but we'll try to remove the obstacles so that as much as possible, so that maybe we can get there eventually in the six series. By which I mean Wayland support, you can have multiple users with multiple keyboards and mice working at the same time and stuff like that, and also on X11. So the initial conditions where we were with Qt Quick and Qt 5, we have these wrapper objects. So Qt Quick pointer event is a Q object wrapper for touch event, mouse event, and so forth, because we wanted to have the newer API and prototype how that was gonna work. Qt Quick event point is a wrapper for the Qt Touch event touch point, but every single point event, like a mouse event has one point as well. So you can always get a point and you can always see what the position is and so forth from that. And then we have Qt Quick event point because it's a long-lived object, a Q object, it can actually store state in between events. Like if you set the grabber, you're actually grabbing a point because touch events are split up and delivered according to where they're touching. So if a couple of your fingers are touching one item, then that item is gonna get a special event that has just those points. So when it grabs, it grabs just those two points if that's what it wants to do. And so the Qt Quick event point will store the state because it continues to live on even after that one event is delivered and the next time an event comes, then we match it up and we see that the grabber is still there. So we know where to deliver it. Qt Quick event point has a parent pointer. So it knows the event that it came from, which is a stable pointer because those event Q objects are also persisting between events. And so we can pass the Qt Quick event point all by itself to a function and it knows the event that it came from. This makes our delivery code, it's possible to make it simpler in Qt Quick window and we partially did that in Qt 5. And the wrappers were meant as a prototype for Qt 6. So if you look back at older versions, this is an old slide as well. The problem originally in early versions of Qt 5 was that touch events, that's the green series of arrows, you see how the delivery goes. And then the Q mouse event, that's the orange series of arrows. And you can see that it visits different functions depending on what the type of the event was. And then tablet events were not being delivered at all. They make it to Qt Quick window event and then they're simply thrown away. And I wanted to be able to support tablet events without having to completely write a whole another set of delivery logic. So that was why we started this refactoring. And you can see why this became a problem. The event hierarchy in Qt 5 looks like this. So the only common base class we have for all this stuff is input event. And that also includes key event. And then the different event objects have similar API but not exactly the same. Like pause is pretty much always there, but in touch point it's a Q point F and in mouse event it's Q point. And this isn't polymorphic. I mean, you have to know what the actual leaf event type is in order to handle it. So you can't just handle all input events the same way. So in Qt 5, like greater than 5.8, we made this new hierarchy of these wrapper objects. So the common interface now is that there's point and point count. We have a pointer event and regardless what type of device it came from, you can always ask how many points there are. If it's a mouse, it's gonna say there's one. If it's touch it can be more than one and then you can iterate over the points and deal with them somehow. And we also prototype this device object which we're gonna have in Qt 6 and Qt GUI. So how did the delivery work in Qt 5, 10 and greater actually, but pointer events were officially supported in 5.11. So that's why I said 5.11 and greater. We start always with a Q event loop which gets the original event from the operating system. And then that event, we send window system events, all of them to Qt window system interface and then into some dispatching. So it processes a window system event and let's say a mouse event came in. Now your platform plugin actually heap allocated an event. This, or window system interface did actually, this came in from the platform plugin, but then, well anyway, there's a heap allocated Q window system interface private mouse event. And that's because it can potentially be put onto an event Q. So we process that and notify and then in this process mouse event, this is where we create the original Q mouse event. And that one stack allocated because it stays in the same thread after that. It goes to the, in this case, the event function of the Q quick window and then we do dispatching there. And so we find out that it's a mouse press event. And then we start to make a pointer event out of that. This is where the wrapper comes into play. So we call pointer event instance, which will then call query pointer event instance, which will then try to find, do we have an existing Q object wrapper that's suitable for this event? Usually it finds one and then it calls the reset function to initialize this object to handle a new event that just came in. And then that has to initialize a pointer mouse event as well. That's the, no, I wanted to say it has to initialize the Q quick event point as well, which is the individual point that is inside the event. And at that point, this event has become a wrapper. And now we can start pointer event delivery in Q quick. So then we deliver the pointer event and we press this and releases are special because that's the case where we have to actually go and find the item that it needs to be delivered to. So we call this pointer targets function, which is a recursive function that generates a vector of all the items that are underneath the point because in a particular order, the top down order, because that's the order that we're gonna visit them. So we want that list to be stable. So we generate it upfront. And then for each of those items, we go through and ask, are there any handlers inside this item that want the event? And so then the item in turn iterates all of the handlers that are inside the item, that are children of the item, and asks each of those, do you want this event point? So let's say it says yes. In this example, it's a drag handler, we're doing a drag here. So it says yes, I want it, depress. So then it goes back and calls deliver matching points to item and that calls handle pointer event on the item private. And then that will in turn dispatch to all of the handlers. So we get back to our drag handler again and then it asks us again, do you want this pointer event? So it says yes. So then we call handle pointer event impul, which is the implementation. And that will realize that since you're, if it wants to find out that a drag is happening, then it needs to become a grabber. This is simplifying a little bit because actually it becomes a passive grabber first and then it becomes, but let's just say it becomes exclusive grabber right away. So it will call set grabber passing itself as the drag handler becomes the grabber. And then because it knows that this event doesn't need to go anywhere else, it will call set accepted on the point, which means that we're done doing delivery. So now in Qt six, we have an event hierarchy that looks a lot like the Qt quick event hierarchy and Qt five. So you have Qt event, Qt input event. Then we have a new class here, Qt pointer event, which has this familiar interface. You can ask how many points there are and you can get an individual point. And I also added this points, which returns a Q list by reference because that's how we're actually storing them. So if you want to write int based for loops, then you can use the old two functions, point and point count. If you want to write the range for loop, you can use this one, but this one is const. So the trouble is if you have an event point in a range for loop, then you can't necessarily call set accepted on that. So we've been trying to figure out what to do about that. But anyway, the pointer event has a pointing device, which is this, well, input event has a device, which is the input device. And then this pointing device is just a convenience method that will do the casting for you to convert it to a Q pointing device, which is what it actually is. So the Q input device has a seat name now for one thing. So that will be how you distinguish them. We're not really sure if we need to do anything more formal than that to handle seats on Wayland, but that seems like a good place to start to just say that the input device has a seat name. And so if your application actually cares about working with multiple seats, then it's kind of up to you to just realize that the seat names are gonna be different on different devices and handle them differently. You can get a list of all the known input devices and those get there because the platform plugin is calling this register input device on Q window system interface. And this is something we're not done with yet that all of the platform plugins need to be able to discover the connected devices and register them all. And I thought that that was gonna be possible, but it actually turns out on some operating systems that's easier than others. I mean, I started with X11, so there's a lot of details available there. And on Wayland, it works pretty well too, but then it turns out on macOS, eh, not so much. They pretty much, you won't know that there's a device there until you get an event from it. That's the trouble. And even then there's not much details about the device. It won't tell you whether, is this the trackpad built into my MacBook Pro or is it a Bluetooth connected trackpad or is it something else, a magic mouse? I don't really know. But it'd be nice if we can find the native APIs to register those devices, then we should do that. And so there's a fallback mechanism. If the device hasn't been registered, then it will just create one at runtime. Even if there's no devices known, you'll get a default one. So you can get a list of devices and that's nice for tools like QtDiag, for example, we've already updated to just list all the devices on your system. And then if you're delivering a keyboard event, for example, and you don't know where it came from, there's this primary keyboard accessor, which we'll figure that out for you and guarantees to give you back an input device. And likewise, there's one for primary pointing device, which on both systems is gonna return you a mouse. Okay, so we have pointer event with this common interface and then about setting grabbers, those were in the queue event point before because you're grabbing a point, right? But because of this problem that now we cannot, an event point cannot necessarily have a pointer back to its pointer event, which we'll talk about in a minute about the memory allocation. And so we basically move those functions up to queue pointer event. So when you receive an event, then you'll be able to take the exclusive grab by calling set exclusive grabber or take a passive grab by calling add passive grabber and you have to pass in the point instance that you want to grab and the object that is becoming the grabber. So then for subclasses, we have single point event, which is most of them, like mouse event only has one point. And then we have touch event and it has this legacy accessor touch points, which is exactly the same as the points accessor. So we just went ahead and deprecated that because now you can be agnostic, you get a pointer event, you can always handle multiple points. It just might turn out that there's only one. And then we had a bit of debate about whether to have a depointer or not in the event point. I didn't really want to, but Lars thinks that we should because that way we'll be free to change it, the usual reason, right? So we'll get into that in a minute. So the delivery to handlers in Qt6 looks like this now. We're still starting with our send window system events, process window system event and we have this heap allocated mouse event coming in. And now we ask the pointing device for the active point with the ID zero and it returns that. So that's because the pointing device has this event point data object, which then has an event point. And this is the place where we're storing the persistent event point now where it's gonna remember the grabbers except for the sake of making event point lighter, we move the grabbers out. So we have this event point data holding a list of passive grabbers and the one exclusive grabber and we just keep these instances around. So first it will ask for the active point, it gets the back and then it's ready to construct a mouse event with that active point, which in turn makes the point inside the mouse event because it has one. And then we update, this isn't really correct anymore. At some point there was an update from, but now that we're actually just caught, we're just taking the point that we already got and putting it in the mouse event, that's not necessary. So now we start our pointer event delivery a little bit earlier. We have the cue pointer event all the way in Qt GUI layer instead of just in Qt Quick. So we call process mouse event, you see it as a cue mouse event, but it's really a cue pointer event. And then we call event on Qt Quick window. So it gets a mouse press event and does the usual stuff. So it gets the list of targets, checks about handlers wanting them. And then when the handler wants to become the exclusive grabber, it's gonna actually call on the mouse event itself to ask for this point to become a grabber. And at that point, those events can be thrown away, but the points remain because we're actually doing reference counting. So I'll get to that in a minute. So initial conditions in Qt GUI, I think I mostly explained this already. This is the Qt 5 state of affairs that the touch point was heap allocated and has a pimple in spite of being temporary. And I thought, well, that's awfully heavy weight for something that's being created and thrown away in such a short time. QPA events and touch points are, duplicate classes, but they have similar API, but it's not close enough. The QPA events are being heap allocated in spite of being temporary. I wish we could change that, but it seems like we can't because of the queuing and because it's crossing threads. On the other hand, the Q input events are stack allocated in Q GUI application and they go out of scope after delivery. So that's nice and efficient. Q input event subclasses have duplicated a bit incompatible API, like mouse event looks like touch event kind of, but not really. And the touch event was the only one that could have multiple points. So design decisions, this is kind of a decision tree that we've been through so far. So starting from the initial conditions on the previous slide, we knew that we wanted to introduce Q pointer event because we did this prototype in Qt Quick and that worked really well. And we knew that every pointer event should have the multi point API, even though sometimes there's only one point because if we didn't do that, then we wouldn't be able to use this nice unified delivery logic. We'd have to write different logic for each type of event. And I don't wanna do that. So then we'd have to keep the wrappers and we wanna give it to those. So then the next question is, where should we put the storage? Single point event and touch event originally had virtual functions for the point count and the point accessor. And so that is nice because the single point event could be smaller because it only needs to store one point whereas the touch event has to have a Q list. But then the trouble is if we wanna have this API for the sake of range four to have a points accessor, it needs to return a Q list anyway. So we might as well just move the list up that makes things easier. And so Q pointer event currently doesn't have virtual functions. That's at least not for those accessors because it can, yeah, directly access the list. Okay, the next question was, should the Q event point have a pointer to the event? And well, the problem with that is events come and go, they're kind of disposable and they're also, they don't have D pointers. So they're stack allocated and therefore they're movable and copyable and all that kind of stuff. And so the pointer wouldn't be stable. I wanted to do that just for the sake of Qt Quick but then I realized it just takes too much bookkeeping. I have to find out when the event that the point is pointing to moves and change it and so forth. And it's just too much trouble. So no, we're gonna have to change some things in Qt Quick a little bit because of that but it's just private API. It just means everywhere we pass the point we have to pass the event on the point together. Yeah, should I keep going or? Okay. So if you want to do a Q&A, you could start now. We have one question for you on the shared notes. Maybe additional questions will pop up in the chat. Well, I've got quite a bit more to go through if that's okay or? Then go for it, yeah. Yeah, okay. So the next question was, should we hold the event point state in the pointing device, which conceptually makes sense because the touchscreen should keep track of your finger positions all the time. So let's go with that. The next time, should we keep the Q event point instances inside the events directly or should we have pointers inside the events? So we decided to keep them inside directly and instead we're just hoping that we'll be able to make the Q event point small enough that that's okay to hold them by value and the nice consequence of this is events are copyable and they can be saved and replayed, which is what people expect because it turned out that there were a lot of places in the code where somebody just constructs a Q mouse event and then wants to play it back later and they expect that to be an atomic thing that you construct a Q mouse event and it will just sit there and it won't change underneath you. Whereas if a Q event point was holding pointers to the points, rather, yeah, the event itself holds pointers to the points and the points could change underneath you because as soon as the next one comes in from the operating system, it might turn out that a point that you thought was being held is actually released now and then you try to deliver your replay and that doesn't work. So okay, they're gonna be held by value. So the next thing is, should they be Q gadgets? And the nice thing about that is that we can emit them by signals to QML. Gadgets can be emitted by value. Only Q objects can be emitted by pointer, basically. We don't want Q objects because that would be heavy. So if we make them Q gadgets, then we can emit signals. But on the other hand, when you emit by value, you're making a copy. So then if you call a function which changes the state, the state can be lost because it's a copy. So it doesn't seem like accepting and rejecting like the way mouse area does would work that way if you're emitting an event or a point by value and then changing it. Okay, so then we had this API review with Lars a couple of days ago and he basically wants to put back the D pointer in QEventPoint, which I was kind of against to an extent because I wanted to make them so small and I thought, ah, this is just, every accessor has an interaction. We're gonna be doing all this heap allocation. I didn't really wanna do that. But on the other hand, well, we have the usual freedom to change the event point private later on, which maybe is gonna be important. We don't really know. And then the QPA events are heap allocated anyway. So maybe later we can just do the heap allocation in the platform plugin and then those event points could come all the way from there up into the whole delivery logic. That might be a way to optimize it. And also, if we store the grabbers separately and we also store the accepted state separately, then it could be that since it's, or even if we don't store it separately, since we have a D pointer, that means that it's really the private that is the long lived object. And so the accepted state could be remembered long enough even if you're emitting by value to QML and you change the accepted state, you're actually modifying the private, which is still there. So you don't lose it when that copy is destroyed. The next thing was, should we move those instances to Q pointing device? And no, should we move the grabber information to the pointing device, to this event point data object? So then the nice thing is that this Q event point private becomes mostly read only data and it gets really small. And it's kind of ephemeral data. It just holds the positions and stuff. And then we throw it away when we're done with it, whereas the grabs tend to live on for longer. So this is where we ended up so far. And we'll see if we're still there by the time Q6 comes out, but that's how it looks. Okay, so here's a little snippet about what I meant about agnostic delivery. If you receive to Q event, you can call this is pointer event function, which is actually just checking a flag. We added a couple of flags in there so that it avoids dynamic cast and things like that. And so if you find out that it is a pointer event, you can static cast it. You can check whether it is a press event, regardless what device it comes from. It could be a finger being pressed or a mouse button being pressed. And then you can go through the points and handle them. So if you have a positive reaction to a press at a particular position, then maybe you want to go ahead and grab. And you can do all of this without caring what specific device it came from. That might be useful some of the time, but a lot of times you're gonna care about more details than that. Here's another snippet about. Sorry to interrupt you, but we're running slightly out of time. Do you want to quickly rub up or do you want to take a quick look at the two questions on the shared notes for you? Well, can I just answer the questions afterwards on the shared notes? Absolutely, yeah. Okay. So checking about whether a mouse event is synthesized or not, or oldest API was just to check whether it was spontaneous, which turned out to be pretty vague because most events are spontaneous actually. So we added this mouse event source in Qt 5.4 so you can check whether the source is not synthesized or synthesized by Qt or synthesized by the US, but then you don't really know. Is it from a touchscreen or a tablet or something else? So now we want to have API similar to what we had in Qt Quick that you can actually check the pointing device type and check specifically is it a touchscreen, a touchpad, a tablet or whatever. And we also have this pointer type, which was originally put in place just to be able to distinguish the ends of a Wacom stylus. You could have the pointy end for drawing or you could have the eraser, but then we filled it out with some other types too, just to kind of have it always make sense. But now it's starting to look like that might be a little bit redundant, like there's two ways to tell if it's from a touch device. It could be the device type is a touchscreen and the pointer type is a finger. So that's a little bit redundant. We're not sure if we want to keep it that way. Okay, so point handler, this is an old example, which I just updated a little bit to show off some of the Qt 6 stuff. So you see there's a seed name there.