 Hello everyone. My name is Michael Treta and I will be telling you about the Linux Graphics Deck and give you an introduction to the Linux Graphics Deck, especially on embedded devices. About me, as I said, I'm Michael Treta. I work as an embedded Linux developer at Pengotronics. Pengotronics is an embedded Linux consulting and support company, so we are doing support for Linux. And there I work in the graphics team. So it's all about graphics, DRM, video for Linux, so cameras, encoders, and graphics output to display and GPUs. The Linux Graphics Deck can be really confusing to us data. There are many acronyms that you can find. For example, Wayland, DRM, which already sounds the same, but it's different. KMS, FB, EGL, OpenGL. So once you've started, you have all these acronyms and it's really difficult to bring these together and to understand what's hidden behind these. And I will try to help you to understand how these connect together and how you can build your model in your mind about the system to understand it better. The focus of this talk will be the Linux APIs. So I will look into the software components that are in Linux and example applications, how you can use DRM and graphics and how you can remove this. And which example applications you can use to look into the graphics deck debug it and see if something is working or if it isn't. Most of this talk will be platform independent. There are, by the nature of the hardware, some platform-dependent things like display controllers, GPUs. I will point that out, but I will try to keep it as generic across different platforms so that it works the same for a Rockchip or IMX6 or Intel or whatever you encounter. The agenda of the talk will be three steps. In the first step, we will be trying to bring a pixel buffer to the display. That means we have some buffer that's filled with color data and bring it out to the display. As a second step, we will be drawing pixels into this color buffer, into this buffer. So we have some nice image we can bring to the display. And in the third step, we will be composing different pixel buffers into one single pixel buffer that can be sent to the display. So I wanted to show this on Rockchip hardware with an RK3568. It's not booting right now, so I have some backup of what I wanted to show, but unfortunately we won't be looking at a live system because demos never work. This is a nice platform. So yesterday in the showcase, we presented the eval kit based on the same SOC. This one is Ratsar ROG3, which you can actually buy, which is cheaper than the eval kit, and you can run a mainline kernel with graphics output and GPU acceleration on that. So if I go into a hardware specific stuff, this will be based on this Rockchip platform. Step one, bring a pixel buffer onto the display. So what usually happens here to bring pixels to the display is that the application first draws pixels into some buffer, so it's some memory, and it puts them as RGB or NV12. So in some specified format, it writes the color of each pixel. For example, it may just go through each line of this buffer and put the color at once there. Then this pixel buffer must be sent to the display controller, which is responsible for driving the display, and the display must be configured to correctly interpret the data in this buffer for the display. So the color format, the size, the stride, which means how long are the lines of this buffer, all of this has to be configured in the display controller to correctly interpret the otherwise meaningless data. First, acronyms. So we already saw some on the introduction slide. DI means direct rendering infrastructure. DI is the overall thing and infrastructure of the Linux graphics stack. So this is everything, the infrastructure. The direct rendering manager is the kernel component of DRI, so DRM is part of DRI. KMS is the kernel mode setting. This is part of DRM, which is especially responsible for driving a display. So mode setting is exactly configuring the display for the data in the pixel buffers. So this is done with KMS. And as a last term, FB, that means frame buffer. So this is the correct name for the pixel buffer. So if I use frame buffer or pixel buffer, this is usually exchangeable. Note that when I use FB or frame buffer, this is not def FB or the FB def subsystem, which is an old and deprecated interface to graphic, to displays in the Linux kernel. If you want to know more about that and DRM, I may guide you to talk by GERD on Friday. So again, the display stack, we see the display on the bottom. DRM, as I said, is the kernel component. On top of that, we have libDRM, which provides C helpers for interacting with the kernel. And on top of that, I will show you output from mode test, which allows you to look into the DRM driver for driving the display, especially KMS and what it provides to you. In DRM, we have, so as I said, the display and the display controller drive to display are usually part of the SOC. So these are platform dependent. The display controller, and you have some part in the DRM driver, which is also responsible for driving exactly this display controller, so it has to be platform dependent as well. More exactly, it depends on the display controller, so you might have the same display controller on different SOCs, but consider it as platform specific. And KMS, as I said, is the interface to the user space for driving the display in DRM. So let's see if I can show you something here. So as I said, mode test is a tool to set a mode via KMS on the display, but it also provides the information about the display controller, the connected display, et cetera. It has to provide it because otherwise it wouldn't be able to set valid modes. The output of mode test may look like this. Can you see the output? So is it readable? Okay, perfect. So what you see here is more or less the output of the command on the left, the mode test minus M rock chip, which says we are running on a rock chip platform. Let's go to the next slide. On top we see the encoders. You can also see it here. So this is some outdated construct which is exposed to user space, so we cannot get rid of that, but we will ignore it. That's everything to encode us. The second part here is connector. So here we see the output of an HDMI connector, and it says it's connected. So some display is connected to HDMI, and it read the edit of the display to find out which modes are supported. So we can see the display I connected while doing this as some 1920 times 1200 pixel size or full HD with different frame rates. So that's information you can retrieve with mode test. So you see a lot of modes. Here you see the blob of the edit. Some further information. That's what you can see about encoders and connectors with mode test. The more interesting part from a software perspective are the CRTCs. The CRTC is an abstraction for the display controller itself. So display controllers may compose different planes and build the final buffer that's sent to the display, and you configure the CRTC to build the buffer that's sent out. So on the top we see that we configured a mode already. So the one with the highest resolution. And we have various planes that are composed by the CRTC or the display controller with different pixel formats. So here you have to configure the planes and the CRTC according to the data you want to show on your display. And as a final step to actually put data there, you have to allocate these frame buffers with FBs. And in your configuration you tell your display controller, okay, take this frame buffer, put it on this plane and show it. Exactly. That's what I wanted to tell you about. As said, DRM is a kernel module. So you have further debugging information about the kernel module available via the SysFS and the debug FS. So you find in the path that are shown here more information about the connection status. You can read the edit there as well. So this is the information exposed by the kernel without going through mode test or some user space component. And for debugging the kernel, you have some specialty in DRM. So you have a parameter which is called debug to enable different flags for debugging DRM driver. There I enable all the flags, but check the DRM driver in the kernel for the meaning of each flag so you can have a more fine grained configuration of that. Exactly. So that's how we bring the pixels to the display. As a next step, we want to draw pixels into the buffer that we sent out to the display. Doing this with the CPU and going by each pixel can be really slow. And for running at higher frame rates, we have deadlines for each buffer when it has to be finished. So this might work, but for higher resolutions, it usually doesn't. Also, coping frames by pixels on the CPU can be slow. It's more or less the same as drawing because you have to take it from one buffer, put it to the other, write it through all your caches, and this might take some time. So that's usually why there are hard accelerators for writing pixels to a buffer. So all this writing and drawing of a picture is hardware accelerated. Again, some more acronyms. The API to applications for using hardware accelerators, from now on I'm using GPUs as a term for that. So it's graphics processing units. These are usually the hardware accelerators. First, OpenGL is the API to applications. So applications use OpenGL to do some rendering. That's currently the case. There is OpenGL is more or less deprecated and superseded by Vulkan, but it's not that broadly used yet. So we will be focusing on OpenGL. A GLSL is the shading language. So you write some shaders for what the GPU should do and you write these shader programs in GLSL. And the third is EGL, which is the interface between OpenGL and your native platform graphics interface. What this means we will see in a few seconds. Here we see again a diagram of the graphics deck now including OpenGL. On the left-hand side we have KMS. So we still have to put something on the display with KMS. This is everything we saw before just boiled down to this is the kernel interface. The application then has to get the buffer that's put on the display on some way. That's GBM. So it can take a frame buffer that's allocated by KMS and wrap it into a GBM object. That's just some more or less internal representation. This GBM object further goes through some DRM component. So everything of that is implemented in MESA. It goes through some DRM component, which is especially for interacting with KMS or DRM drivers. And some interface that's called DRI2 not to be confused with direct rendering interface, which is the overall thing, but it's the same just an implementation. It's difficult with the naming, but this is an implementation of an internal interface. And on top of that you have EGL. So you get the frame buffer via KMS, put it through GBM, DRM, DRI, EGL, and now you have an EGL image that's a high level of representation of your frame buffer that you will put on the display. And this EGL image you can put to OpenGL and render to it with OpenGL. OpenGL, as I said, abstracts the GPU. The GPU is platform specific. Again, GPU specific, but it might differ between your platforms. On the Rockchip platform that I have here, the GPU is supported by the Panfrost driver. It's a Mali GPU. So this implementation of OpenGL is, again, platform specific. All of that over here is Linux interfaces and works the same on all platforms. As an example application, we used KMS, or I used KMS cube, which was a really simple application to access KMS and use OpenGL to render something. So not much is happening there. It's just an example application for debugging the generic MESA implementation. You can use environment variables. So for debugging what's going on in MESA, you might set, for example, EGL log level so it logs what's going on in EGL, or you might dump the GLSL shaders that the application uses. So you can look into what's OpenGL shaders, what OpenGL shaders look like that the application is using. You can look further down into MESA in the platform specific part. So the shaders have to be compiled to some GPU specific machine language. For looking into what your driver is doing, you can use other environment variables. They are usually prefixed with a driver name and MESA debug. Then you can see raw shaders or see what optimizations the driver is doing to your shaders for running in other GPU. Again, if you have a different platform, the variables name might be different. Check the MESA source. Usually the options and the environment variables are documented there. Now we have a problem that KMS and the display can be only driven by a single application. That's by design. Only one application can drive the display. That's called DRM master. So if you're running a second application that tries to access DRM because you want to use a second application on a different plane, for example, you might receive permission denied error when starting the application and that's usually caused by the second application not being the DRM master. Furthermore, toolkits like GTK, Qt, don't support direct interaction with KMS and applications as well. That's because most of the time it doesn't do what you actually want and if you have multiple applications, it just doesn't work. Therefore, we need a window compositor that takes the different applications, puts them together into a single buffer and sends it out to the display. That's where Wayland comes in. So we have one. We have here the KMS. We already saw this on the previous slides. We have some Wayland compositor running on top of KMS and multiple applications, the Wayland clients on top of the Wayland compositor. The Wayland clients and the compositor interact with each other via Wayland protocols which are specified. Yeah, various protocols for especially specified for this interaction. In my example, I will use Western which is the reference implementation of a Wayland compositor. It's usable but not really usable for desktop use cases. So you have different Wayland compositors if you look into your machines. So if you use GNOME, you might have MATA as a compositor with KDE. You have KWIN or there are various compositors based on WL routes like Sway and Cage. So these are all implementations of Wayland compositors. Here we see different representation of the diagram we saw before. We have DRM and KMS in the bottom like with KMS cube. We have the Wayland compositor Western running on KMS. Some Wayland client, the simple EGL application running as well and interacting with the Western compositor with the Wayland protocol. So here Wayland is actually a Wayland protocol. So Western can be run by just typing Western on the command line. And it should just fire up. It prints some debugging output. I won't go into details right now. And Western supports debug protocols for debugging. So you can enable the debug protocols with a dash, dash debug parameter. And then you can use Western debug with different debugging streams to see what Western is actually doing. For example, one debugging stream is back in DRM, which is exactly the implementation or the component of Western that uses KMS. So if you look at this stream, you will see what Western does with the planes, what color of pixel formats it sets on the planes and in general how Western interacts with KMS. So if there is something strange on the display and you're expecting that Western is doing something wrong, look at this debugging stream. There are further tools for debugging Wayland. Wayland Info is a client application that connects to your Wayland compositor and shows you which protocols are supported in which versions. So you can use this to get information about your Wayland compositor and what can be used in your application of the compositor here. As said, you have the protocols. These are quite important because the applications are using it to talk to the compositor to look into what the application sends to the compositor and receives as responses. You might say this Wayland debug environment variable. So you get all the messages that are sent between the client and the server and see if something is surprising there. And that's for debugging the client side, so what the client is seeing. You have as well a compositor. The compositor has to interact with clients as well via the protocols for debugging the server, the compositor perspective of the Wayland protocol. You might use Western debug again with a protostream which shows you all the protocol messages that the compositor is seeing. So that's useful for debugging the interaction between a Wayland client and a Wayland server. This might look familiar. We want applications to draw into pixel buffers as well. These are not frame buffers that are directly sent to the display but are sent to the Wayland compositor for compositing. So we have different pixel buffers that are later composited. Now I will start on top. So we have a simply GL which uses OpenGL. That's the same as with KMS cube. And the difference, yeah, buffers are abstracted as EGL images and DRI. But here we don't have the DRM platform but now we have a Wayland platform on the client side. And this Wayland platform, instead of using frame buffers from KMS, it uses buffers that are sent via Wayland protocols to the Wayland compositor. But for the application, it almost looks the same. So everything happens down here in Mesa which detects, oh, I'm not running on KMS but I'm running as a Wayland client. So as a summary, now we are connecting the terms that we saw in the beginning. So we have, again, OpenGL which is the abstraction for using GPUs. We have EGL which is used to provide buffers to OpenGL into which we can render. All of this is part of the direct rendering infrastructure which is the overall term. We have Wayland for passing buffers that are allocated by the clients. To some Wayland compositor for composing it into a final buffer. We have DRM as the kernel component that is our driver for, is the kernel component of the DRI infrastructure. We have KMS which is the API that drives the display as part of DRM. And we have frame buffers which are the buffers that we can pass through DRM to a display and have something we can show on the display. That's about my talk and there are two further talks about DRM. One right in the afternoon and one later in the afternoon. If you want a different perspective on DRM, you might extend there as well. You can reach me via my mail address. I will be around here. Thank you and if I get a demo running, we might as well look into the commands that I showed and see it on a live system. If you're interested in that, maybe catch me up and let's see if we can do something there. Thank you. Any questions? Comments? Disagreements? Yes? So the question was here, for example, that there are multiple lines to EGL and OpenGL and if the application has to use or has to directly talk to EGL and OpenGL, the answer is yes, the application is responsible, for example, for allocating the EGL buffer, EGL images. So it has to talk EGL and it has to talk OpenGL and in the previous example with GBM, it also has to explicitly use GBM there as well. So it's not that it can use all of them, it actually has to use them. I don't see any hands anymore. So the question is here, we see the Panfrost driver as part of MESA and the question is are other GPUs supported as MESA as well and do vendors upstream their drivers to MESA? And there are other drivers, so usually MESA is the umbrella project for all the GPU drivers in Linux and upstream drivers usually go there. If vendors provide upstream drivers, they should use MESA as well, so that's the way to go. You get all the goodies around that, the EGL abstraction, OpenGL optimizers, some optimizers there as well. So if you want to upstream your driver, go to MESA. Usually vendors don't do that, they have their own user space blob for that and also the kernel driver is usually not DRM or not in the DRM infrastructure. Thank you. Back there. So the question is here, if there is a direct path from the GPU to the display, so from a software perspective it's not, but in the kernel you have DMA buff which is exactly an abstraction to pass a buffer that's used by the GPU to the display driver. So in the software stack you don't pass it directly, but it's actually pass directly underneath the layers and not copied. Does it answer your question? Yes. It's called DMA buff. It's a concept in the kernel and it's exposed via file descriptor, but it's handled by MESA so you don't actually do it yourself in your application. The question is about the fastest way from the GPU to the main memory. I think you mean if you have some specific video memory on the GPU for a screen share, that depends on your hardware. That's all I can say about that. So that's really completely hardware dependent and use case dependent. So for streaming you would use some NV12 format for encoding, for example, so if you do H264 encoding for your streaming, you want to use it, have it in NV12 which is a planar format. This is not optimal for displaying it on most of the GPUs. Some have support for that, that again depends on the display controller. If you can display such a buffer, usually you cannot write with your GPU into such a buffer. So that really depends on your use case, your limitations and what your hardware can do. I think I saw a question of a historical reason, but that happened before I was involved there. I think we have to stop now. Thanks again for all your questions and see you around. Bye.