 Thank you, Ingo. Yeah, GammaRay. Who of you has already heard about this? Who has already used it? I hope that will be a few more after this presentation. So what is GammaRay? GammaRay is a runtime introspection tool for Qt. If you might have encountered the tooling you find in current web browsers like the Web Inspector or Firebug, imagine something like that just for Qt applications. So it's something that at runtime allows you to look at what's going on inside your application, but at a much higher level than a conventional instruction level debugger. So you don't want to debug layouting issues by stepping through the Q layout implementation. You want to debug that by seeing the actual layout and fine-tuning the various layouting parameters until it actually matches what you have in mind. And that's the kind of stuff that we can do with GammaRay. But instead of doing all theory here, what I actually want to do in this session is primarily show you how to use GammaRay and show you a few real-world problems that we investigated using GammaRay. So to get started, let's look at how to actually open your application with GammaRay and look at what's going inside here. Is this half-ray readable or should I increase the font? It's okay? Good. So, larger? Okay. Better? Okay. So there's two ways you can connect to an application similar to how you would use a normal debugger, right? So you can either launch your application just by prefixing it with GammaRay. And that launches the application. This is just some dummy widget layouting example here. And then you get the GammaRay main window started as a second window. Is that half-ray readable? I'm abusing the high-DPI support in Qt here to scale it up. So there will be a few rendering glitches. I have no idea if the high-DPI team is here. So in the end, what we get with GammaRay here is the... This is actually not visible at all. So we have the tool selector on the left starting with the object tree. That's the most basic tool. That's what you have in the middle then. That's basically the entire hierarchy of all Q objects in your application. And then you have the property editor where you see the specific properties of that object. And that's kind of the foundation for everything else inside GammaRay. And that's 20 or so specialized tools that focus on one specific aspect of Qt. For example, widgets. There you get some layouting overlays and it limits your view to that specific kind of object. But we'll see that in much more detail later. So that's basic UI and how you start your application at the beginning. Alternatively, you can also attach at runtime. So start your application. Either you pass the process ID on a command line like you would do it for a debugger. Or you just use the GammaRay launcher UI that you get by just launching GammaRay without any additional arguments. You select the process you want to attach to. It does some magic in the background. It takes a while because like attaching a debugger, you need to load all the debug information. And then you get the same main window. As you notice, there's fewer objects in there. If you attach at runtime, GammaRay is unfortunately not able to find all the objects that have been created since application startup. But only those that can be found by watching out for events or by walking down the object hierarchy. So usually prefer the launching with GammaRay attaching is just if you see your application misbehave, it's very hard to start. Then attaching at runtime comes in handy. Right. So back to this. Yeah, there's a few things to keep in mind. The way this works is there's a small part that's called probe. That's basically a DLL that's loaded into your application. And that needs to match the Qt version you are using in your target application exactly. That means the Qt version, that means the compiler, and that means the compiler settings on Windows. So debug release needs to match. If that doesn't match, GammaRay will notify you and will not even allow you to launch the application. That's probably one of the most common problems you might run into. And as a side effect of that, you most of the time will need to compile GammaRay yourself for your specific Qt configuration to make this work. That's a bit inconvenient, but the implementation of this heavily relies on all kinds of non-public internal Qt API. So, and that keeps changing. So there is unfortunately no way around this. Right. But then the interesting question is what can you actually do with this? So what would I use it for? And for that I have a few examples that are all based on real-world problems. So stuff we actually encountered in customer projects or we run into in our own code. And where we found GammaRay particularly useful for debugging this. All the examples are massively simplified, which might make some of the cases a bit too obvious, but if you imagine that in millions of lines of code, you might not even know having the tool that points you right to the spot where you have the problem becomes really valuable. So let's start with the first one. Slots. So this is a tiny widget-based application that has two buttons and an LCD widget. One of the few remaining uses of the LCD widget. It has two basic functions. I can click on the emit button and the LCD widget increments. And I have another button that reconnects the signal from the emit button to the widget for incrementing the number. So you see that still works. Reconnect again. And suddenly our increment gets bigger. So let's try to fix that with another reconnect. And our increments keep getting bigger. So something is wrong there. Probably somehow related to signals and slots. If you try to diagnose that with a conventional debugger, you put a break point at the code. I can actually show you the code. Not in this one. So maybe you can spot the problem even without the debugger. Readable. So it's layout, two buttons. We have the reconnect signal slot. So that's the code that's executed when you click the reconnect button. And we have the signal emitted slot that's connected to the clicked signal of the first button. So if your number keeps incrementing somehow wrongly, what you would usually do is use your instruction level debugger. You put a break point on the increment of the LCD widget and you see what triggered it, right? So what you will see is it's the click signal from the push button. That's not really helping, right? That's what we expected. But it doesn't really tell us why we get that multiple times. So let's... I already launched it just gamma ray. So let me make this a bit bigger again. Works. Reconnect. Still works. Reconnect again. Increments get bigger. Reconnect a few times. Gets even bigger. So how do we find anything remotely related to this in gamma ray? So the good starting point would be the button that actually emits the click signal. So let's find the corresponding object. I mean, you have the ability to search in the object tree. But most people don't give their objects sensible names. You have the address in the class names. It's not really helpful. What you can do with widgets and also with quick applications is to in-app picking of graphical elements. So if I control shift left click on the button, the right button is automatically selected. And it's even opening the widget inspector because it knows I clicked on the widget. And then it gives me the properties and the methods and so on for that specific object. So our problem is the signals. So we can check in the methods tab. Signals are technically methods in QMeta object speak. So that's where we're probably going to find the clicked signal. And first of all, let's check if how often it's actually emitted. So I can right click on it and connect to it. And if I now trigger it, you see in the output per click it's emitted exactly once without argument. So that's kind of what we expected. So at least at that side, it's all fine. I can also do the other side. I can manually emit it. It doesn't have any arguments. I can just confirm this. And we see it has the same effect. So I have the ability to monitor individual signals and I have the ability to manually trigger them, which in this case already helps me to bisect the problem. So the problem is not the emitting side, it's probably the receiver side. On a slightly bigger scale, we also have the signal tool that plots all the signal emissions in the entire application. So for the one click case, that is overkill. So this is the clicks coming in. But if I have problems with signal emissions on a larger scale, this comes in handy too. There's a useful feature to navigate between the various views you have in GammaRay. That's also in the context menu. So it offers you to basically select the same object in whatever other tool that might be applicable. So let's go back to widgets. So if the problem isn't on the sender side, we can look at, well, it's obviously not at the receiver side either because that is a one-line piece of code that just increments. So it might be worth looking at the connections in between. And we also have an option for that here. So for each object, you can look at both the inbound and the outbound connections. And on that button here, we of course expect an outbound connection to the code actually incrementing our value. And the fact that we see some warning icons down here already is kind of suspicious. And we see there's a clicked signal connected to the signal emitted slot. And once more, and once more, and once more. So apparently, we are accumulating connections. And that of course would explain our problem. And we can just reconnect that another time to verify it. And now we are already at six inbound connections on the slot. You can also by a context menu jump between sender and receiver to observe both sides. So this, I mean, now we have an explanation for our problem. It doesn't actually answer the question, why there is this problem. But if you look at the code now for the connection handling, you see one is done string-based, one is done pointer-based. They aren't interchangeable, so you can't disconnect a string-based connection with a pointer-based disconnect or the other way around. But at least GammaRay guided us to that specific connection handling code that we needed to look at. Yeah. So that's the first example on primarily on signal and slot handling. And if you try that with an instruction level debugger, it's very hard to find out what is actually connected to a specific object there. I mean, you need to look into the key object private internals or you see what is a print object info function that dumps out the connection. But then you need to find the right object first in your debugger and trigger that. So that comes in quite handy to have a nice visual view of this. Okay. So let's look at something else, something QML-based for a change. Also still a rather simple problem. Let me launch it with GammaRay right away. That's quick event handling. So even simpler UI. Let me make this slightly bigger. Just a single button is very hard to do anything wrong with just a single button. I can click it. I get some debug outfit from that. As long as I click like on the left side, if I click on the right side, nothing happens. So something is apparently eating our events. So the approach for this is similar again, right? I launched with GammaRay. The tool of choice for QML is the quick scene view. This looks slightly broken. There we go. Just needed a repaint. So we saw that clicking on the left side works. So we would expect that the element at that point in the screen is our actual button. So I can verify that with the picking tool in the preview here. So I just click somewhere on the screen and it selects the corresponding object. So if I now click on the right-hand side where clicking doesn't work, I see that there's something else selected. So there's something on top of the button eating my events. And we see it's apparently a second button that is not really visible. So let's check where this is in code. So this is another nice feature you can directly jump to code. It navigates your editor into the right place. That already works for QML. We are also working on making that work for C++ if you have debug information around. So indeed our application has a second button, but that's supposed to be hidden. But somebody tried to hide it by making it fully transparent rather than by actually not making it visible. At least that would be the theory, right? So another thing that GammaRay enables you to do is to actually live edit the properties, which is very useful on verifying this kind of series, right? So if we would actually hide the button by setting it to invisible, there we go. Then clicking on the right-hand side should work, right? So view this and indeed this now works. So this doesn't change the code. So I still need to fix the code, but with the code navigation to the right place, that's fairly easy. But I can test this theory without even restarting the application. Right. So we have the code navigation in QML, but as I mentioned earlier, we also produce debug output whenever I click on it. And for debug output Qt also provides source code information. So the same navigation works here. In the debug output view, I can jump directly to the line that produced the debug information. That's also useful if you get random warnings, for example, from within Qt. And you want to find out what's actually triggering this and how to get rid of it. Right. So that's the QML case. And I mean, this looks like a trivial example, but imagine having that in an application with many, many screens, and there's suddenly the not completely invisible on-screen keyboard hiding half of your screen. And you can click on the upper half, but the lower half doesn't work. Identifying that one element in a big code base that is messing up your event handling is just so much easier if you can do this kind of just click on this place and show me everything that's there. Actually, let me show you one more thing. That's a rather new feature in it. If I just use the picker, I get the top-level element at this specific location, which is most of the time the element I actually want. I can also do control shift click on that location, and I get the complete set of all elements that stack on top of each other there. For event handling, that's sometimes helpful, but that also comes in very useful for painting artifacts or just seeing what actually is visible there on top of each other, because all of this has rendering costs. So you get a fairly detailed view on what's actually stacked up there. And the same is also possible for widget layouts. Okay. Now we have my favorite example, something slightly more advanced than the event handling. Let me do this without gamma ray first. So this is two pretty QML sliders. It consists of a few rectangles in a list view and slightly changes colors. So pretty basic stuff one would think. If we look at the code, that's actually all right. So it's a fairly simple program, but once you run this through, say, API trace, which you would usually not do for something that simple, but which you would do once somebody complains that your application using your fancy sliders is slightly too slow. Oops. That's all we need. We see that the two simple sliders need 740 GL calls. We dive down there. We see there's a lot of them are draw calls. So that's the expensive part in OpenGL. Usually, one would assume that this is maybe four or five draw calls. So something is clearly wrong there. And if we look anywhere in this, in what's actually happening, we see it's actually drawing rect angle by rect angle. And this is like the slowest possible way you can do OpenGL. And then you lose all the advantages of Qt Quick. Now, how to debug this? I mean, there's a tiny bit of background information needed for this. Let me just jump to the magic code for this from the batch renderer. So how the way Qt Quick renders its content is it tries to batch together as many visual elements as possible and then merge them together in a single draw call. And then a single draw call it tries to do as much as possible in parallel. And that's where the GPU is really efficient. So this is like the sweet spot for rendering performance. That's where you want to get to. And this is the like the core part of the logic that decides if these batches can be merged together. But this isn't really, I mean, getting from this to why do my sliders not batch correctly in itself isn't really useful. So we need something to actually look into the running application and see, okay, this is the transformations we are doing. This doesn't match the logic and so on. So let me keep this here and search back to you. So let me run this in gamma ray. So for some of this Qt Quick has built in diagnostics because I guess certain other people also had similar questions when debugging the scene graph renderer. If they implemented this with debugging categories, gamma ray would give you the option to enable them at one time. That comes in handy in particularly for the rendering code because that produces huge amount of output every frame. So you want to see, usually you want to see diagnostics for like one moment somewhere at one time where you actually have the problem and that's not necessarily right from application startup. So you want to navigate to the screen that is broken, then enable the diagnostics and then look at what's actually going on. And thanks to categorized logging in Qt, you can actually do this with gamma ray and it also shows you the kind of different categories that Qt provides and that you could enable. Rendering is particularly interesting and input handling as well. Unfortunately the scene graph batching code uses some approach with environment variables. That still needs to be ported to this. So that's not actually helping us in this case. But it's the place where we started to see if we can actually get some information. So we need to look a lot deeper into what's actually going on here. And for that we have the scene graph view. So this actually shows us the internal structure that you hardly get access to from the outside of what the quick renderer uses to... Actually not here. This is the view. What's happening inside. And if you remember the complex condition for merging, one of this in there is the transformation matrix save for 2D only transformations. You would assume that our non-transformed rectangles or seemingly non-transformed rectangles would be fine for this. There's no rotation, no affine transformation or anything like this in there. So it looks from the result. But if you look at the actual transformation matrix that is applied, you'll notice in the combined matrix for one of the small rectangles in our slider that there's a negative number on the diagonal. And then anything that's not like same numbers on the diagonal is somewhat like a rotation. And that's actually triggering the condition in the merger. So the merger thinks this is a tricky rotation that it can't merge. And yeah, that's giving us the problem we have. So we have one step further already. We know that something is introducing a transformation that breaks the merger. But now we need to find out where this is actually coming from. And if you look, there's two properties here per node. There's matrix and combined matrix. Matrix is the transformation that happens in this specific step. And this is harmless. This is a simple translation. And then combined matrix is basically everything along the hierarchy up to this point. So we can go up the hierarchy and identify the node. This one, for example, there you see the matrix one actually is the one introducing the minus on the diagonal. So we can switch back to items and it identifies the specific element introducing the rotation. And, yeah, we see that there's some place in the code that tries to be clever regarding list view layouting and just flips it around by 180 degrees. Looks in the result as if there would be no transformation involved. But that's enough already to break the batch merger. So this is one and not that much time. So I'll just skip finding the other one and take this out as well. Same approach. So if you remove the first one, you find the second one. Just quickly showing you that this is actually the problem. So, of course, now the sliders are the wrong way around and the handles are all on the same side. So I actually broke the application. But if I run this with API trace, right now we are down to 76 GL calls. So this is, yeah, what was the other one? 700 something. So about the factor of 10 faster rendering or less GL calls, which is usually comparable to faster rendering. That, of course, still leaves me with an application where my sliders are the wrong way around. But that just means I need to find a solution that does this without rotating. Again, navigating through the scene graph tree with the conventional instruction level debugger, trying to find the matrix, print out the matrix in a way that you can actually read it, and then identify the node that broke it, map that back to code. Fairly complicated unless you have tools that actually support you in doing this. Okay, so I think that's the most complicated example I have for this. Let's look at something from a somewhat different field, classical old-school widget layouting problem. So we have this complicated dialogue. It's somewhat resizable, but yeah, this is as small as it gets. So apparently something is blocking it from making it smaller in the vertical direction. If you look at that, you would assume it's those address group boxes at the side. The street field sounds like something that can use several lines of text. So maybe this is actually all correct, but if I need the dialogue smaller, then something needs to be rearranged. So GammaRay allows me to verify that at runtime without actually changing anything. So I again select the widget I want with Ctrl Shift, click in the application, or I can also use the preview down here and the picking mode in there. So let's just hide this into a box to see if this actually fixes our problem, and apparently it doesn't. So something else is keeping up the space. So this approach is rather useful for kind of bisecting the layouting problem. Usually you would do exactly the same with recompiled cycles. So you take out something, you recompile, and you see if that fixed the problem, then the problem is in the part you took out and so on. With GammaRay you can do this in the live application. Saves you a lot of round trips in particular when you are working on embedded targets with slow cross compilation and expensive deploy cycles. So how do we find the thing that prevents this from getting smaller? You already see this blue area that's basically highlighting the spacing areas and the elements in the individual layout. So I can just click through those and see what's actually in here. And that very quickly shows me that there's apparently one widget down here, a label which currently doesn't have a text, which I don't see, but by just selecting it on screen, GammaRay gives me the ability to identify the, yeah, invisible things, but things that still need space. And for that, I mean, we can now investigate why it's not resizable. And again, for this the tool of choice is the property view. I mean, there's usual suspects like size policy. So as you can see here, GammaRay supports even the more complicated types in the property editor, at least the queue types or basically anything that's a gadget or a queue object or a type that we manually edit. So you can look at the individual details and modify them. Size policy looks sane in this thing, but the minimum size here looks like it could be the problem. So we can again edit that life and see if that fixes our problem. So again, this doesn't actually fix your problem in code yet, but you have verified that you have found the reason for your problem. And now you can actually fix this in code without even having a single recompiled cycle. Okay, so that's a midget layout thing. Right, something else I can show you quickly is state machines. Let me show the code for this as well. So this is a simple example that implements a traffic light with a declarative state machine, which I think is the standard example for anything state machine related. But you see, this is already, like, doesn't fit on a single screen anymore. Finding anything in there that, why is this actually not running? Let me start that. Finding any logic errors in a state machine just by looking at the code or stepping through the implementation is fairly tedious. And if you look at the flow of the traffic light, you notice that now it's going from green to red without going through the yellow state, but from looking at the code, finding the right place there for something that has four or five states you will manage, but in reality you have something with hundreds of states. Finding that in code is fairly messy. So what we offer for that is a graphical display of the state machine layout and a live view of the configuration. And there we immediately see that apparently we put one transition in the wrong place and have the green to red state dangling unreachable on the top. And you already see it's live updating and if I put the pause button on, so all the various state types in QDA supported by this. Right, and then one more. That's a brand new one that isn't even completely enabled yet in the released version. And I think then we are running out of time. That's the Q3D inspector. So here we have a rotating cylinder and apparently something is messed up there. We notice that the light reflections are somewhat often most importantly there's apparently some part missing. So if we look at that in the geometry view, and never mind the black thing that is a high DPI scaling issue, so in a normal version you wouldn't see that. We can look at the geometry of the elements we have in the Q3D scene. Middle mouse button doesn't work. If we look at the bottom of this, we see that apparently something is missing there. That's actually a fairly common problem in 3D geometry. If you render the triangles in the wrong order, you don't see them because you're looking at them back. Here we have a very useful diagnostic feature where we just disable that backface culling and render the stuff you're looking at from the wrong side in red. So we see that our geometry is actually correct. We are just looking at it from the wrong side. So it's just a matter of changing the index order. But again, the difference from you don't see anything to you can at least verify that your vertices are in the right place already. You just index order is wrong. It helps a lot to diagnose this problem. So that at least explains the missing bit in our cylinder. And the other feature we have in here is showing the normal vectors. And that explains the weird light issue we have. I'm not sure how well this is visible on the big screen. You see these tiny blue hairs? That's the normal vectors per vertex. So if we look at this somewhat from the side, there you see it, you see that they are somewhat bent towards the center rather than being perfectly straight off the surface. So that's again something that is that has some visual impact. But it's very hard to diagnose this without actually seeing what's happening. If you want to see more details, we can of course also look at like the raw buffer data that's sent there. But this is much harder to read. This is mostly if you're generating the geometry and want to compare if that's actually what you put in there. Or to find memory leaks. If you suddenly see bizarre numbers in there and far more than you expected and apparently something else is wrong. But the visualization is that is what really makes the thing useful, I think. Right. So I don't think we have time for the remaining demo. So let me switch back to the slide deck. All the examples I showed here and also those that I couldn't show here are all part of the gamma ray manual, including a description on basically what I showed you here and a few more examples that I didn't have time for. That leaves us with the question on where you can actually get gamma ray. The tool is available on GitHub. So it's available as GPL and also under commercial license. And it's also part of the Qt Automotive Suite where it also comes with a creator integration to make it even easier to use. If you want to see it in action or play with it at our table downstairs, there's a demo connected to an embedded device. So while everything I showed you here is desktop only, it also works remotely with an embedded target. Yeah. I think that's all I have. Thank you. Hello. How hard is a gamma ray to maintain? If it's all using all this private API, I can see just now what you showed. I probably broke something that was there in 5.8. So both how easy is it to maintain and how do you maintain it between different versions of Qt? Yeah. That's indeed a problem given that we support anything back to Qt 4.8. And we use a lot of private API that you guys keep breaking. Well, the best thing we can do is large CI coverage and adapt it for every Qt version. So I mean, if you look at the code, there's a lot of iftaves, there's a lot of special cases for different Qt versions. A lot of features are only available in certain Qt versions as there's new stuff being added. So we try to release with every Qt version and try to keep up. I don't think there's a way to make this easier. I mean, we already added a few not entirely public but somewhat stable APIs to Qt to make this easier, like the hooks for object creation and object destruction. Before that, it was a lot more messier. But yeah, it's work. I don't think there's much you can do to simplify this. Thank you. Can you use gamma ray on an embedded device like debug or edit properties, whatever? Do you have a server running on the embedded device? Right. The probe part, the part running in the application actually opens a socket and that's what the client connects to. So this works locally as well as remotely. We have that downstairs if you look at our table there, it's connected to an embedded device. Thank you. And it supports Linux, QNX, Android on the embedded target and Linux, Windows and OS X on the host. Any more questions? Thank you, Fokar. Thank you.