 All right, I guess I will get started. Welcome, everybody. I'm Mark James. And I'm an engineer at Intel working with the Mesa team. And here to talk to you about why Mesa is increasingly a more compelling option for use on your embedded platforms. So just a little bit of background about me. I've been working in user space Linux for over 10 years. And I got my start on embedded Linux platforms. So at the time, Linux was really exploding in the embedded space. And coming to this conference was actually a really important part of my career growth. So I'm really glad that Tim Bird has been putting it on for all these years, so thanks to him. It's been super satisfying to be a part of the transition away from proprietary platforms. And I think it's been great for the products and the customers, but especially for the engineers. It's so much easier to work with these platforms. I've been working on Mesa for about two years, implementing some performance tools for their driver team and also automating a lot of the processes they use to guarantee quality of the driver. So I'd like to thank Intel for sponsoring that work. It's been a real career goal of mine to work on an open source project full-time. So that's great. And I'd like to thank them for sponsoring my talk here as well. Okay, so just to give some background about Mesa, for those of you who are not really well versed in Linux graphics. Mesa is the open source driver that supports a wide variety of GPUs. It's developed completely in the open under a community model. It's supported commercially by a number of companies that make GPUs. And the companies collaborate with each other and with game developers and users to find bugs and fix and implement features for their hardware platforms. So it's got a development model that's very similar to the Linux kernel where patches come in over email are reviewed and they can be nacked to keep bad design out of the driver. It has a completely public repository that anyone can access. The developers are typically available over IRC for anyone to ask questions of and they communicate with each other over that mechanism as well. There's a public bugzilla that you can go and look to see any issues with the Mesa driver reported by our own quality assurance engineers or by end users as well. So there's a degree of transparency to this project that you won't get from a lot of other driver stacks. One way that it's different from the Linux kernel is that it has an MIT license. So if you want to change the driver to expose some extension or do any way to repackage it, I think probably there's parts of Mesa that are incorporated into other proprietary driver stacks and they don't have any obligation to release their patches. Okay, so this is just a slide to point you at the various links that you would want to go look at if you wanted to investigate Mesa. I wanted to point out specifically though the third link, that's a different project that's related to the graphics driver. It's an automated unit test project that all of the Mesa developers write tests for. So this is an example where you have an older, old school open source development model that's also implementing test driven development style programming where you have to have a test for a feature, for any feature that exercises your code before you can get your patch into Mesa to enable that feature. And that project has actually been crucial for I think all of the Silicon vendors who've been implementing drivers in Mesa to make sure that their code works properly. Okay, so I just would like to mention the various hardware platforms that Mesa supports. I work for Intel. So Intel is a huge sponsor of Mesa development. They have a very competent team. The other thing that Intel does is they have a large test lab with about 100 test units of Silicon platforms going all the way back to 2007 when Intel integrated graphics were first released. And we test every single commit. A lot of times before they're sent to the mailing list to verify that it doesn't break any existing or old hardware. So there's a significant investment that Intel makes that the whole community benefits from. Another big sponsor of Mesa development is AMD. They implement open drivers for their GPUs as well. Broadcom is sponsoring Mesa development. Erica Enhol has implemented a driver for the VC4 hardware and that's what you find in the Raspberry Pi. So he's here if you'd like to talk to him. I think that work is really passing the other options you have for running the graphics part on Raspberry systems. So those are three different companies that all collaborate for this project and they all leverage the same driver infrastructure. Another corporate sponsor is VMware which isn't really an embedded platform but they do in addition to enabling GPU acceleration in their virtual machines via Mesa they also have written some critical tools that everyone benefits from and I'll show you one of those later. There are other initiatives to write drivers for GPUs which aren't sponsored directly by companies and instead have been implemented by volunteers or just people who are interested in enabling a piece of hardware. So FreeDreno is an example of that type of project. It enables the Qualcomm Adreno chip and so if you are running that system it's definitely worth your time to go and see if it implements the features that you need. There's also the Etnavive driver which was recently upstreamed into Mesa and it works with Avante parts. I think that it's a new driver in Mesa so you'd have to go and see what features are supported by that project but Pengatronics is a company that's for hire that's been reverse engineering the hardware and if you have a problem with it you can talk to them and I'm sure they can work with you to solve your problems. Another really important project within Mesa is the Nubow project and it has a particularly large and passionate set of volunteers who've been reverse engineering NVIDIA GPUs to get free drivers working for them. They have a whole bunch of challenges ahead of them and they work really hard to figure out how the hardware works but NVIDIA I think sometimes puts barriers in their way that are hard to overcome. The last I heard there's some power management firmware on NVIDIA chips that there's really no way to operate if NVIDIA doesn't give you the details to work with them so it's a great project but I think that's one of the hardware platforms that you'll have limited support. So in talking about why you would choose Mesa for your embedded product, I think first you have to understand that almost every single GPU vendor will provide you a proprietary driver stack where it will operate the equipment but it won't give you access to the source necessarily or you might get access to the source through an NDA but it's not an open source project and you can't share that source with other people or you don't have access to the history of that implementation. So typically you have a choice, right? Do I take what the vendor's giving me with my board or do I go and use the community alternative or the open source alternative? So in many cases I think one thing that you need to know about the proprietary implementations is they are often just a port of the Windows OpenGL driver which they made to enable OpenGL and perhaps it might share a back end with their Windows DX driver. So the implementation is really focused on another platform and then they've ported it to run on Linux and because 3D drivers are often tightly coupled with kernel, they've had to put kernel patches into the Linux kernel to give it more of a Microsoft interface or a Windows interface so that those drivers will be easier to port. So that's kind of a problem if you wanna have flexibility in the versions you choose because those patches to the Linux kernel can't get upstreamed in the Linux kernel unless they have a open source user land that's calling them. So unless they release their driver as open source, those patches aren't gonna get in. So that's been a barrier and it means that for proprietary drivers, often you're limited to specific kernel versions with out of tree patches. So if you do wanna update, maybe you need a security patch in the latest kernel or maybe you want, they're giving you a new kernel but you wanna have the GPU work on an older kernel, you don't have as much flexibility in those choices. So another thing that you have really good support for in Mesa is Valgrind and I'm not really sure how the proprietary drivers handle this but when you're dealing with GPU memory, obviously you have to instrument calls to allocate and deallocate those memory regions so that Valgrind doesn't think it's a leak or it's a problem, right? So if you compile Mesa with Valgrind installed, Mesa and libdrm, you're gonna get an instrumented implementation of Mesa that Valgrind supports really well. If you have a memory error in your program, maybe in your OpenGL program, you're gonna have an easier time finding it because Valgrind works really well. So I think probably all of us know how critical Valgrind is for finding your memory errors. So another thing that you can do with Mesa that is really helpful is you can debug from your application right into the driver stack and locate what's happening within the driver and particularly for OpenGL where it's managing a lot of driver state for you to give you a simplified programming interface. It's sometimes really helpful to understand what's going on behind the scenes. Beyond just being able to debug into the source, I mean, if you have a source drop under NDA of your proprietary driver, you may also be able to debug in. But what you won't have is you won't be able to get blame. So if your call is eventually hitting a slow path or it's hitting some assertion saying this texture format isn't supported, you can use get to really figure out why what's the comment associated with this code that somebody put in. You'll have an email, you might even ask a question like, hey, I thought this would be fast. What do I have to do to fix it? So that's something that's unmatched by other driver stacks. And as I mentioned before, since you have the source and it's MIT licensed, if you have some kind of exotic requirement that you need to fulfill with your graphic stack, you can write your own extensions to the GL interface and use them in your product. And there are definitely examples where vendors of Linux platforms have gone and done this and I'll talk about that later. Also, since it's all in Git, if you have written custom extensions or maybe you've modified the driver in some way, when if you do wanna update your versions, you have Git rebase there for you and it really facilitates keeping your own modifications usable as the driver moves forward. So if you put all these things together, I think you'll see that with Mesa, you're uniquely enabled to resolve your own problems that you might have with the driver stack because of the transparency and how it's developed in the tools that you have. I think an example from my own work history with proprietary implementations, if you end up having problems with some binary drop, you're really kind of on the hook or you're really hopeful that your vendor is just gonna go fix the problem for you. But as an example, a company I used to work for purchased a proprietary communication stack and we deployed it in our product, which was relatively straightforward, but with embedded products particularly, this is what happens, after shipping the product out to the marketplace, customers began to deploy it in different environments and that revealed a whole bunch of bugs in that driver and that stack. So you may have a support contract to say, hey, help me fix my problems, but your upstream vendor is not enabled to go even reproduce those things because they don't have the hardware that you've been making. So particularly in an embedded scenario, you really need to have all the tools to go and figure out what's going wrong and how do I fix it. The company I was at, my friend had this job to go find all these problems and it was a year's long effort of constantly finding bugs in someone else's implementations. I mean, he was QA for this other company and he was also the help desk for all of the end users who were trying to basically get their problems solved. So it can generate a lot of work for you to have this kind of situation. The problem gets exacerbated at the point where your upstream vendor doesn't want to fix bugs on their old version anymore because they've shipped a new version and that forces you to have to update, right? And so now you're the very first user for their brand new release and you're now having to find bugs in that release but also all the work you did to integrate that old release might not be reusable. I mean, the extent to which they've re-architected their implementation means your patches are no longer valid and you have to go fix everything again. So I think this is the part of the engineering cycle which is really living hell, right? So I mean, my friend, I remember coming in multiple times and finding him sleeping at his desk in the morning, right? Because he'd been up all night, so. So Mesa's better. Mesa lets you sleep in your bed at night, so. All right. So why are there so many proprietary solutions, right? I mean, why do I have to tell this story? Everybody, maybe if people have been in the industry long enough, they've actually lived through something like that and that's why we're at the embedded Linux conference, right? So the truth is that that shift from proprietary implementations to open implementations has happened in most areas except for drivers, GPU drivers specifically. So we still have all these handsets going out, phones or whatever, and they all have proprietary GPU drivers, so why is that? The truth is, Mesa has been behind a lot of these proprietary solutions for a long time. It's been a small team. That process of porting Windows features into Linux has basically given them a big, given the proprietary implementations, a big upfront lead in getting all the features aboard. But over the past couple of years, Mesa's been on a tear and as of a month ago, we're fully certified compliant on all of the most recent Kronos specifications and I think this is, Mesa hasn't been in this situation for 10 years at least and so it's taken a lot of hard work and the team is extremely proud. We get these great sweatshirts to advertise, like we wear them everywhere because we're really thrilled. So from an end user perspective, the environment has changed for you now, right? You have open source drivers which make your life simpler but they also implement all the things you want. So it's time to go and look again at your products and see, hey, should I be using Mesa at this point? So there's a, so Mesa 17.0, which has all of these features for Intel drivers and I think for the Radeon driver, OpenGL 4.5 is exposed as well and there's also an open source implementation of Vulkan. I do, I want to mention Vulkan though, Mesa historically has been an implementation of the OpenGL APIs but the next generation of those programming models is Vulkan. It's a much lower level way to control the workload on your GPU and it gives you explicit control over everything that's happening on the hardware. So with OpenGL, it tries to simplify the programming model for you and as CPUs got turned into multi-core systems, it ended up not getting the performance out of the system that was possible. So Vulkan is a lower level interface and you have to do more work as someone writing a Vulkan program but it lets you have really tight control over what's happening. So if you have latency deadlines that you need to meet reliably, you might think about Vulkan for doing that. If you're interested in Vulkan, Jason Xtrend is gonna give a talk soon after this about that API and what it has to offer embedded platforms. So if you wanna check other driver stacks besides Intel, there's this website Mesa Matrix and it's in columns by hardware. So one second. So I mean you can see Intel is gonna have a green column for every single extension that's composed of the various features and AMD also advertises support for most of the extensions. If you look at Fredrino, it's not gonna support a lot of the desktop Linux or the desktop OpenGL extensions but it will do a competent job of OpenGL ES or other that kind of API because that's really more targeted to the, yeah, here's the OpenGL GLES extensions. So that's a webpage you can look at to see where Mesa's at for your platform. Or I mean, some of you may be interested in software rendering depending on what kind of hardware you're dealing with. There are three different software renders within Mesa that you can use to basically get pixels out. Yeah, so, but I think that the progress has really been noticeable. I mean, the amount, number of commits going into Mesa is huge and there's been press about it. And so that's resulted in a lot of progress. Okay, so beyond actually feature support in Mesa, another reason why you might consider Mesa instead of a proprietary driver for your platform is the implementation size. So the libraries that Mesa produces for stripped down compilation where you're just supporting Wayland and Weston can be around seven megabytes. So that's because Mesa really has to compile on a wide variety of scenarios and it limits the dependencies that it uses and it has a fairly tight implementation. Just looking at different proprietary drivers that you can download and unpack and see the size of the libraries, that's comparable to 40 megabytes. And that ratio is the same for Radeon from when I investigated it. So one of the reasons this might be is that if you were making a proprietary driver, a binary, and you're shipping the binary, you need to make it runnable on a lot of different desktops. And so if you have a dependency, you need to statically link it in so that it won't fail to run time on Red Hat Enterprise old version. So that may be one reason why the size is that different. But for embedded platforms, this matters, right? You don't wanna have a huge RAM footprint. So another great motivating factor for using Mesa and embedded targets is the ease of integration with your particular platform. So there's quite a variety of examples of how you can initialize a display and get an EGL context that you can render into. And if you're building a custom solution, these are all open source and you can go look at it. So just quickly, I'll show you KMS cube. So KMS cube is basically, KMS cube is basically a sample program written by Rob Clark that completely initializes the graphics hardware using the kernel mode setting driver. So you can go from a console, all the way to an EGL context and the total size of this is about 700 lines of code. So if you're doing this for the first time, like, hey, how do I get graphics? You can look at this and you don't need anything else. So right now, I don't know if you guys can see, this is just a terminal, a terminal logon on a dual platform that I'm running. And so it goes straight to a full screen rendering of a spinning cube. So that's probably the simplest solution for you if you just wanna get graphics out. If you've got a full screen app, maybe it's in, I don't know, a car display or something and you don't have to worry about windows or anything, you can map input events directly into that context. Okay, so if you have a more complicated product, maybe you have multiple applications that render 3D graphics and maybe they're in windows and they take input, then you need a compositor, basically, to manage those graphics context for you. So Weston and Wayland are, sorry, Weston is a compositor built on Wayland and there are lots of distributions that are built on top of these. So you can look at the Octo project or Tizen. Tizen is shipping on tons of Samsung TVs. I mean, these are distributions that work in the marketplace and you can go see, okay, well, how do I get graphics going for my platform based on these? Another example, a completely different example of how you can set up graphics is Android. Android has different requirements and typically has a modified kernel to enable the interfaces that Android needs. So at Intel, we've been working on a project called Android IA. What it does is it implements the Android APIs like Gralek, Hardware Composer on top of the upstream Linux interfaces and it lets you run Android on top of Mesa on top of an upstream kernel without having to take these out of tree patches. So it's another open source project you can look at to see a completely different variety of embedded graphics. So another graphic stack that you could look at is Chrome OS. So Chrome OS is not considered an embedded distribution, it runs on laptops, but it certainly is another example of some company that's written their own compositing solution in Freon. One thing that's interesting about Chrome OS is that that team has hacked Mesa quite a bit or used Mesa quite a bit to do some novel things. In Chrome OS right now, you can enable an Android container that runs the entire Android user land on top of the same Linux kernel in your Chromebook. And they'll pass frame buffers rendered by Android applications through I think a datagram socket over to the Chrome OS side of the container for display within Chrome OS. So this is something that having an open source driver really enabled and as a result, you can run the entire Android app store on Linux. So they got Photoshop working on Linux, right? I mean, I've been waiting for at least 15 years to have Photoshop working on Linux and now it's here. So hopefully someone will say, this is great, we should do this. We should implement this for Debian, right? And then we'll have Photoshop on Debian. We don't have to, you know, this seems like something that anybody could implement, right? Getting an Android true rendering graphics and enabling the app store. So it's very exciting. And, you know, the difficulty of implementing proprietary applications on Linux is something that Linus Torvalds has complained about, right? Like who are you gonna compile for if you wanna ship a proprietary application, right? That's why all the apps on Linux are open source because they have to be recompiled against whatever C library you have, right? So this is an example where, in addition to having a container, you also have a developer SDK that the entire world has already ported their applications to, right? So I think it's a big win for Linux and enabled by an open driver, so. Okay, so yeah, so I'd like to basically show you some of the tools that we've been working on. Now that we have feature parity in our driver, Intel is really kind of turning our focus to try to improve the performance so that we're getting the most out of the hardware. And so some of the things we're doing is authoring tools, we're working on benchmarks, trying to analyze the work. And so if you are using Mesa for your platform, I'd like to show you what you would do to go try to figure out why is my program slower than I thought it would be. So the first place you would go to analyze your workload is to the environment variables that Mesa supports. So there's a whole bunch of settings you can make that change the way Mesa sends your workload down to the driver, and some of them make it quite a bit noisier. So if some debug flags so that if you get a GL error, it'll immediately send that to standard error so you can find it. If you go on down, here's the Intel environment variables that you might set. So if you set Intel debug equal to perf, it's gonna tell you when it thinks you're making a mistake that causes the driver to be slower than expected. So just a quick example of that. All right, so this is a login to this embedded board and got some binaries here or some benchmarks here. So I'm gonna launch the car chase demo on the board with the Intel debug equals perf setting. And you'll see there's some output and you can see the right now the driver's compiling the shaders for execution. And we'll let it run for a bit. So we generate some output as it compiles the shaders and here's the workload running. It's kind of rendering a car chase. All right, so if we go through and look at the output, one of the things you might see is here's a warning. So it's telling you for your shader that it couldn't enable 7016 in the shader compilation. And so you're paying a performance penalty because in this particular shader, there's so many variables that they won't fit into the register space, right? So that's an example of how you might say, okay, well, let's improve that shader. So another example is, let's see. So in this case, the driver's emitting a message that's saying, hey, your fragment shader had to be recompiled. So this is an example where you already compiled a fragment shader for rendering, but then you made a state change. So you changed the number of colored buffers that you were rendering to. And so at that point, the driver had to stop and generated a completely new shader binary and that introduces a lot of latency to you. That's the kind of thing that can't happen with Vulkan. So another example is, so all right, so here's a message about buffer sub data. And so it's telling you that the last player cache doesn't support unsynchronized buffer sub data. That honestly is basically a driver feature that we have on this platform, that we haven't implemented on this platform. So this platform is gonna get faster when our driver team goes, I mean, it needs kernel features and it needs something in Mesa to basically operate the hardware more effectively. So hopefully in the next release of Mesa, this is on our short list of performance features that we have to go and fix to make it faster. There's another example of this, which is so using a blit copy to avoid stalling on GL buffer sub data. So this can happen when the driver thinks that you're trying to write to a portion of memory that is still busy on the GPU. So this might be a bug in your application you might be able to use GL draw range elements more appropriately to help the driver figure out what part of memory is really gonna be in use and what part of memory you can access. So this is an example of performance features, performance messages that you can use to analyze your app on your embedded target. So going back. So the next tool that I'd like to show you is API trace. This is a tool that VMware has implemented for all of us. It's a cross platform tool. What it does is it serializes all calls into the GL and then allows you to go and replay them exactly bit for bit. So this is a really powerful tool for us when we're having bugs reported to us. They can say, hey, this is what my program's doing. Why did it crash? It also works for Windows DX APIs. If I mean it's probably not interesting to anybody here, but it kind of shows how flexible the tool is. So just to show you that. So what I'm doing is I'm running API trace and telling it to trace and I send the output to a file in flash temp. At this point, you can see the embedded board is loading yet another benchmark. This is the Manhattan benchmark. So it has to go through the usual gymnastics of compiling all the shaders and there's a scene rendering and eventually some city will be destroyed. So interrupting that, if we go now and look in slash temp. Sorry, one second. I have to look in slash temp on the embedded board, not on my own system. All right, there's the trace file. It's a nice big file with all the things that you'd want to look at. And I'll go back to my log in. And to show you how you retrace pretty easy, you just call the GL retrace tool within API trace and you'll see that the whole application is going to run as far as we recorded on the system. So that's pretty useful. Another thing, oh, there's the scene rendering. Okay, so something else that you might want to do with API trace is you may be interested in seeing the calls themselves. So API trace dump will basically convert that binary file to a text file. Pretty big text file, but you can see all of the GL calls going through the workload listed out. So this lets you get insight into your application that might be harder to get just with a debugger. All right, so there's also some other things you can do in API trace, if you use a help function, you can make it chatter, you can increase debugging checks, those types of things, so that might help you take a look at your workload. All right, so the next tool is GPU top. So it's just recently that the Intel Mesa team has gotten access to the performance counters that are on the GPU. So this is a critical part of the hardware. When you're looking at a workload on the CPU, you have a lot of tools like Perf, right, that tell you where's your hotspot and your code. With 3D workloads, it's always running asynchronously on another piece of hardware. So you don't have access to exactly where the time is going a lot. Most silicon vendors have specific hardware built in that gives you counts that might give you information about that, and that wasn't something that was openly documented until recently for Intel platforms. So we've been implementing, we've been exposing those counters and making them available in our tools. This helps us figure out what the cost is, where the cost is in a workload. So let's just run GPU top. So when you run GPU top, it opens a port on your embedded device. And then you can go to a local host at the same port and type in the IP address of your target. And so you'll see there's metrics over here. And so these are the GPU counters that are available. So you can see how many core clocks are currently running. I mean, you can highlight a metric to see a description of exactly what it's doing, right? So that's kind of interesting. And if I go, I'll enable GPU busy. It's kind of hard to see. Here it is. Let's put GPU, that's the GPU busy. We'll put that in the graph. So the GPU is not busy right now, but if I run a benchmark on the system. So this is a T-Rex workload. You'll see it's compiling shaders right now. Now we've got rendering and you can see that the different counters of the hardware are climbing. Awesome, demo worked. And the GPU is obviously 100% busy, right? So this is an Intel GPU, right? Now there are similar counters for definitely for AMD. Nvidia has them too. I mean, we don't have access to expose those, but they do on Windows. So I mean, I have no idea what the proprietary driver exposes, right? There's always an extension in the GL to look at these counters. So Intel has written a tool for Windows which exposes Nvidia, AMD and Intel counters for analysis. And they're roughly similar because they're all direct rendering engines. GPU like, that's maybe a deferred renderer. It may be, you'll still have hardware counters, but they'll be used in different ways. I think imagination technologies has PBR tune which exposes some similar counters, but they also have zero open driver efforts. All right, so, all right. So yeah, I mean, that implementation was quite a lot of work for Robert Bragg really implemented that. And so he did a really good job there, so, okay. Okay, so another tool which works on top of these counters is Framery Trace. This is a tool I've been working on. This is a branch of API Trace. It works off of API Trace files and it exposes those same GPU counters for a single frame unless you kind of look in deep detail exactly what's going on in your rendering. So to run that, you have to go over to your embedded target and run a little server application that listens on a port for the UI to connect to it. And then on your host, you run this UI basically. And so this is a API Trace file. We're gonna look at frame 20. And all these tools work on local hosts too if you wanna look at the GPU, but when you're rendering pixels to the same display that you're also analyzing then it can confuse the output. So, sorry for me to see if I typed that correctly. Yeah, okay, so you can see here's the workload. It sent the Trace file over to the embedded platform. Told it to play up to frame 20. It's currently stuck on frame five or something, which basically says that it's loading. Oh, then we get the frame playing. All right, so this is a single frame which is displayed on the target and it's being instrumented. And so if you look down here, you've got all the same metrics that were exposed. It's kind of a long list, but that wasn't me, I swear. So you would look at GPU clocks basically. So these are the ticks basically that the GPU consumed for every single render in the frame. This is the most expensive call in this particular frame. There's a whole bunch of calls and you'll see that each one of these does draw some aspect of the frame. That's kind of expensive. For this frame, you can see it displays the vertex shader and the fragment shader. There's also the intermittent representation and the binary that's actually sent down. So if you're a real, if you know a lot about the hardware, you can go and see exactly what's going on. You can also edit the shader and compile it and if you wanna figure out, okay, what's going on with my shader. Sorry, I'm trying to see here. Okay, this is the render target at that part of the frame. It plays through to the last render for that target, but if you stop at the highlighted selection, it'll tell you. And if you're like, what's this actually doing? Highlight selected render, it just replaces the fragment shader that's been linked in with something that draws pink on the pixels. So this kind of tells you, these are all the pixels basically that were affected by that render. That explains why it's that expensive. If you look over at smaller renders like here and you highlight what do I have, it's kind of less pixels perhaps, right? So, but you might find, hey, some of these are really expensive. One workflow for this tool is to sort of graph horizontally GPU time versus clocks and that really just makes the expensive renders pop out. That's where you want to go and investigate your shader to figure out why is it slow. But if you want to kind of isolate for pixels, you might map it against, oh, why am I typing? Oh, sorry, thank you. Yeah, rasterized pixels, right? This is great. It tells you exactly how many pixels were painted by your shader, right? This is something you couldn't, you can't get any other way. And then in this case, you'd look for these the tall skinny bars. Say, oh, you know what? Maybe that one might be a little more, more expensive for what it is and what is it actually doing? Okay. This is definitely interesting, right? Super tall, super thin. What's going on with that? You can show the API calls for, I'm not doing this this way. All right, you can show the API calls for this particular render and this explains why there's no pixels. This is a compute load. If you look at the shaders, you'll see there's no fragment shader. There's a compute shader, right? So that explains that particular aspect of the frame. Yeah, so yeah, and if you wanted to go edit that, you can compile it. And what this'll do is it'll change the shader, relink it, rerun it within the frame and then give you new metrics for what you did. So that feature we definitely used on the driver team where accessing a constant uniform inside of a loop was causing us to go and access the constant data through memory every single time, even though it was constant. And so moving that access outside of the loop caused the memory bytes metrics to fall and that eventually was enabled in a global code motion optimization within the driver. So this is a way, I think it's kind of a, just new for Linux anyways, the ability to look this closely or workload and exactly what's going on with hardware. There's nothing, right now we're working on Intel parts, obviously. Like I said, the same counters are available for Radeon and so if, I think certainly possible to expose the same extensions in that driver and then enable them in the same tool. This is an open source tool, so. All right, so hopefully that's interesting. For those of you who are looking at graphics workloads. And again, there's an embedded board here running. I mean, you're not gonna be fighting Godzilla on your, you know, speedometer, hopefully, but it maybe demonstrates the graphics power that's available to you in these small flip prints now. All right, let's close that. How are we doing on time? Oh, we're three minutes over. Okay, sorry guys. So, just before I stop completely, there are a couple of other tools that you wanna look at. There's another prototype called GraphIps which provides similar features to GPU top but also lets you instrument your workload and gives you live analysis of the metrics as your workload runs. That's helpful for you to figure out if you're GPU bound or CPU bound and whether you wanna be looking with GPU tools or CPU tools or maybe if you might be able to figure out the system bottleneck using that tool. RenderDoc is another tool that's been around for a long time looking at Windows DX APIs, but the author has done a lot of work to enable looking at Vulkan. So, if you wanna go and debug your Vulkan frame, you should look at RenderDoc. At the moment, I think the RenderDoc UI is really Windows oriented, so you'll need a Windows host to do that, but I know that Valve is paying to have them develop a QT QML UI that'll run on any host. Okay, thanks for listening. I hope this was new information for you guys, at least some of it. So, do you have any questions? Yes. That's right. Yeah, I had that in caps in my notes and I went right over it. The Mesa team at Intel is hiring, so if you are enjoying your visit to Portland and you'd like to stay and work on open source graphics, please send email, I'll have it on the slide. My email is on the first slide and you can send email to me and I will connect you to the perfect person for you. So, specifically if you're interested, if you know a lot about graphics workloads, that's wonderful, but anyone who's really competent and fluent in the Linux ecosystems, I mean, and can work on Linux tools, like we're working on things that are just user space Linux, so definitely we're interested. Okay.