 Okay, hello everyone. In the program it states from where we are going to talk about regular general blender development onboarding. That was the plan that Tom had and he appointed me and I said, there is already an onboarding speak on Thursday so I'm going to do the next step. And that is basically onboarding on the GPU module. My name is Jeroen Bakker. I'm a developer at Blender since 2008. Since 2008 I was a community developer, did some many small things in the note editor, bigger things in the compositor. And in 2018 when we were working on the code quest, so Blender 2.8 development Tom mentioned talk to me and said, we have one project that we don't have any developers for, can you do that? And I said, what's the project? Yeah, we don't have a developer for the viewport. We have one for EV, but not for the viewport. So I don't know anything about GPU development or modern GPU development, but I can learn. So I had some great guys back then, Clemore, Brecht, Seaguy, teaching me a bit from how they think about GPU development. And then it starts rolling and then it was more from, okay, I'm in the GPU development module and still there and I really enjoy it. The target audience of this speak can be anything. I'm not going that deep into the details. I'm going to show you a bit how a GPU works, but mainly about structures and ways how we structure the code inside Blender so you can use GPU. I'm going to start with, okay, if you have Blender and from frame to frame, what's actually happening in high level things. Then I'm going to talk about the GPU module, which is the source Blender GPU directory in the source code and then the draw manager, source Blender draw folder in the source code. And what I want to do is afterwards, like you can't debug a GPU. That's not possible. You can't set a breakpoint in your source code at least at this point and then step through your shaders or your application and see what's going on. But I will show you from, okay, how do we do that? And what kind of tools are we using? And that can be interesting also from a triaging perspective or regular Blender development perspective. Sometimes I get an email from, oh yeah, it went wrong in the GPU module. Here is the bug report and I have to start over again. But having just more knowledge about from, okay, how does the GPU development work or the GPU module, that might improve the quality of the bug reports that we have. And that would be an awesome win for this talk. So what happens between frames? Well, basically Blender has a main function. And a lot of those starting developers will set the breakpoint in the main method of Blender and then starts debugging and step into and sees thousands of line of initializing code and test of stuff. The Blender main function is not the voice main parameters which starts an executable. Blender has its own W main, WM main, which is actually the main loop of Blender. I will show you shortly. The first thing that that that main loop does is it will grab some events from your operating system. It does that via an abstraction layer so it can run on any hardware platform. And then processes that event. It can be from, okay, I'm pressing G. Okay, for G I have to start. For G in the viewport, I have to start the grab of the or the transform operation. Okay, I'm starting the transform operation. For the transform operation, the change in the object, you change the position of the object and then in the operator, you also set from, okay, this object has been changed. So basically, you're only working on scene data there. But you're tagging the things that are important that have been changed. Then we have a second step. That is the processing of the windows event. And with the windows event, we are talking about resizing the window, resizing areas, that kind of things. I'm not going to dive into that. And afterwards modifiers will or handlers will be invoked. And those handlers will listen to the actual text that has been set via the operator. And then it will then say from, okay, this object has changed the position of the object has changed. So I need to update the viewport tag viewport has to be updated. And oh yeah, the property panels also shows the position of the object tag. The property panels has to be rerun. After that, the view will be rerun. I'm going to go into a bit deeper of that, of course, because that's the whole part of this talk. But the first one to show you from, okay, this is actually the actual code inside Blender showing that main loop. It's there. It's described. The comments are the same. It's just copy paste from the source code. So I didn't change anything. This is the main loop. It's really important for new developers to perhaps not look at the main function, but look at this main function. Well, an editor has been tagged to be redrawn. At that point, editors, how are they structured? You have the 3d viewport. The 3d viewport has separate areas, which is the main area where the 3d scene is displayed. Then you have the top bar. You have some sidebar, a menu bar. And those are called regions. And every region has its own draw function, which is a callback which will be called when the area has been tagged. And inside that draw function, UI components are being called. And eventually, GUI, the GPU module is being called, or the draw manager is being called. I'm going to step in more detail of the last two things. The UI components are in this talk a bit out of scope. Internally, they just call the GPU module immediate mode. But that part I will go into later on. Just a step back, creating a new space type, which is actually an editor. And I want to show you from where due to the callbacks of the draw functions are created. This is also actual code. It's nothing propriety. And here we define the view 3d space type, which is the view 3d editor. At front, we just define we're going to create an editor. And at the end, we're going to register that this is an editor. During when blender starts, this function is being called to register. Okay, this is how the 3d viewport works. The 3d viewport has multiple regions. And yeah, of course, the main region, the main window region is the part where the scene is shown. And that is also being created set in that data structure. And per region, you can set different callbacks. And for this talk, the draw callback is actually the one we are talking about. And inside that draw function, you can use the GPU module to draw something. And what's the GPU module? The GPU module is a common way how to draw something using the GPU, but being independent of the actual platform it's running on. Like OpenGL, when you run Blender, it starts and runs on OpenGL. So there's one back end. But in the near future, we will be having the metal back end and the Vulkan back end. And perhaps who knows a DirectX back end? You never know. And so the GPU module, if you use that, it will automatically run on all those back ends that we have created. It is based on the core profile. For people who are not familiar, I'm going to dive deeper into the core profile. And it has a compatibility layer for all the OpenGL draw calls. What do we mean with core profile? With core profile, we actually mean it's a shader-based approach of drawing to your screen. So we're talking about a vertex shader or a fragment shader. If you know the website shader toy, that kind of stuff. I'm going to show you also from how the immediate mode works, but that's later on. The core profile has been introduced or stabilized in OpenGL 3.3 and to deprecate the immediate mode. To deprecate, it still exists because some software companies with a lot of money said, no, we're not going to migrate to the new API. Games did want to do it because it has a lot of performance benefits. So we have now two standards. Inside the shader-based approach, when you want to draw something on the screen, you have to select a render target. Your render target is a frame buffer. It's also called that, where the actual pixel will land. And you have a program. A program is a combination of several of one vertex shader and a fragment shader. I'm simplifying it of course. There's much more going on. Linked as a single program, which does the operations that needs to be done to draw something to that frame buffer. It also needs some index buffer and vertex buffers geometry data. And perhaps you want to render a texture or have some additional buffers like weight painting buffers you want to draw aside. You add that to separate buffers and textures. But there are also some parameters you can set. And we call them, they were called uniforms or in modern Vulcan-based languages, it's renamed to global uniforms due to some restructures that Vulcan gets. And at the end, when you have loaded the program, loaded all the buffers, attached them together, you just say, okay, draw. And based on that, it will then draw what you have pre-prepared. It's a different programming model than in CPU, where you can tweak or just do a step, then take back, load more data in between because you know now for sure that you need the data. In shader-based approach, you actually need to load all the data that you can do to perform the whole draw command. In such a draw command, a lot of stages will happen as from vertex shader, triangle, assembly, vesselization. I'm going to do it from step by step to show you what they are. Let's first start with the vertex buffer. Here's the vertex buffer of the default cube. But I also added some color variations to it to make it more exciting. This is actually the data that Blender then stores on the GPU. Then we have a vertex shader. What does the vertex shader do? The vertex buffer that was uploaded now with the in we define that we want to read the position and the color from the vertex buffer. We also have a parameter and that parameter is a transformation matrix. That is the world matrix, including the view transform and the camera transform all together inside a single matrix. Then we can multiply the matrix with in position and store that in the GL position, which will be given to the rest of the shader program. And we copy the in color to the out color, so we pass that also along. So we now have eight dots on the screen and we have the actual coordinates on the screen and the colors that we also had in the input vertex buffer. Then we get to the index buffer and the index buffer that connects the vertices into actual geometry. Now a hidden step happens. Actually, it's the rasterization step and in the rasterization step, every triangle primitive of the previous step is being split in horizontal lines. For every horizontal screen line, the leftmost pixel and color is determined and the rightmost pixel and color is determined. And then for each pixel in between of the leftmost and the rightmost, the fragment shader will be invoked. This is an example of the fragment shader. Okay, I'm getting a color, the position of the screen that's already there because fragment shader result is stored in that position, so we don't have to read it. We get us our color, we want to store it in the output color and we just copy it. And by doing that, we got a beautiful colored cube and this, all these steps that happen in a single draw command. But then having multiple GPU backends like metal, Vulkan, OpenGL, there are some differences and one of those differences is that the GLSL, like Vulkan has a GLSL, which is the shading language, OpenGL has a GLSL calls the same shading language, but they're not 100% compatible. The actual code, the logic, that's quite the same. But there are differences in from, okay, how do you bind or you define the bindings to load the data inside the shaders. For that, createInfo was being created, createInfo. CreateInfo is a data structure where we can define those bindings and how the shader is built up so we can generate those differences. How does such a createInfo look like? Well, here's createInfo where we defined, here we're defining a createInfo with the name shader3dsmootColor, where we say from, okay, from the vertex buffer, we will read an imposition and the in color. We also need a parameter, which is called push constants, because we're moving towards a Vulkan-based way of working where the global uniforms or where we're going to use push constants instead of uniforms. And for the source code, it's somewhere in an file called colorvert.glsl. The vertex shader will pass some data around to the next stage, to the fragment shader in this case. We also defined that, but we created an interface info for. In the interface info is then the container, the structure, the data structure containing the attributes that will be passed from the vertex shader to the fragment shader. And then we have the definition of the fragment shader where, okay, this is the source code. And we're going to write to something that's called the fragment color, the fact color. The last line, do static compilation, is that we also have a mechanism to create inheritance of these shaders. And that is something because we are generating the actual.glsl code that from the definition, we can already use in sort of inheritance to reduce the code duplication. Otherwise, you had to do this for a great example is the workbench engine. The workbench has 100 shaders and actually it's only doing one thing. But as the parameters are different because you have the solid shading or the, or the x-ray shading, we just change that per parameter. And then we can actually use the rest of the definition. With the static compilation, we mark that shader as from, okay, this is actually a shader that we want to use. And we want to compile. It contains all the data. And this is also a step from, okay, we can do validation on this. And this is then how you create a shader. Just having that definition, the glsl. And when you want to use it, create a shader from a specific info, pass the name, and you will get that shader back. Because a lot of code is generated, this is the actual shaders that code that you actually need to write in the glsl. All the binding information and the stage information and the parameters, which is part of the create info, will be added automatically for you. Because we know which shader is static, we can also do validation on top of it. And with validation, we mean that from, okay, you're working on your glsl and you're working on your system and your system has OpenGL and you're working on those shader. If you change something, will it still work on metal or will it still work on Vulkan, you want to validate that. You can start Blender, OpenGL, then go to that functionality that uses that shader, see if it crashes, or you can use compile time shader validation. This is a tool that is inside Blender, which you can enable in the compiler options, which during compilation time will already compile those shaders and give you an error if something fails. Since last week, if you're on an apple, it will also do the metal validation and the OpenGL validation at the same time. And eventually when we have Vulkan, we will be using the moltenvk. So if you have an apple development station, it will run all the validation of the three GPU backends that we support. Immediate mode. A lot of, like for 3D environments, such a shader-based approach is really nice to work with. But if you want to draw buttons or lines or whatever simple UI components, it's hard to set up, because it is just a lot of changes you need to make in different places. That's not really helpful. So we made an incompatibility layer in the GPU module, which looks like the old immediate mode, but internally uses shaders to actually draw that. So internally it still uses the core profile, which we want to use. And how does that look like? Well, here we define our vertex format, so how does the vertex I'm going to send to the GPU are structured. For this example, I'm only using a position that's defined here. Then I'm going to select a shader. And Blender for the user interface has a lot of building shaders, which you can just use so you don't have to write in them yourself. And you can set some parameters there as well. And the uniform color parameter of the old OpenGL immediate mode is directly transformed to a specific uniform. Then you want to send some data. Like here, I'm going to draw a box, a rectangle. And I'm going to say from I'm going to draw some lines. And those lines are connected. And these are the points that I want to use. And when we're at the last sentence, I am and it will then create the vertex buffer that's needed, the index buffer that's needed, and try to and send that to a buffer, which will then do the actual drawing. Internally, it will multiple of those draw calls will be grouped together for performance reasons. For example, if you have a lot of data that you want to, there's always a consideration to make from if you have a large amount of data you want to use, I would prefer or it's better to use the shader based approach, because the immediate base approach would upload the vertex buffer or create the vertex buffer every time that you're going to to draw something. And with the shader based approach, you can catch that on the GPU. So next topic, the draw manager. The draw manager was introduced in Blender 2.8, and with the ideas of from, okay, we have EV coming. And EV needs to be real time, high, high performance. We have the workbench engine, which, which was high, high performance, best performance beat for an animator. And we also had some, some other additional challenges, as for example, in Blender 2.7 series, when you were in rendered mode, and you could go into edit mode and still move the vertices around, but you couldn't see them. You couldn't see them because it was not part of the cycle's code base. And what, what we introduced was an overlay engine, and that overlay engine could render the, all the edit mode stuff on top of any render engine that you used. Grease pencil came along. Currently, we are working on a viewport compositor that's all done inside the draw manager. And eventually, we also used the same technology in the image editor and to draw the backdrop of the compositor. So we took also for performance reasons, but also for correct colonists. I'm not going too deep into from, okay, how does EV work and the workbench engine works? Because that's, I think I can talk about that for ages and still not understanding it. And there is some person in the room who understands it. But I'm going over from, okay, how does such a draw engine work on API level? So I'm going to quickly go over a simple draw, draw engine that, that I created, going over the inner stage, sinking stage, and actual drawing stage. The inner stage is when you're working in shader based approach, your shader does not, doesn't have to draw the final pixel that's on the screen. You can use intermediate buffers. And every, and those intermediate steps that you take, we organize in passes. And here I define two passes, a pre pass and a shading pass. And this is typically done in, in modern game, game engine to where the shading is so, is so complex that you don't want to shade on, on every pixel that, that will be drawn only on pixels that are visible in the screen. So you first do a pre pass to select the pixels that you want that, that are visible. And then you do a shading pass or deferred pass to actually do the calculation for, for the, for the lighting. It's, this is still in development, but this, this is something where we want to move to. And that is that every draw, if, if, if you want to draw something currently in Blender 3.3, everything is locked in a, in a, in a global locking mechanism. So only one thing can use the draw manager at a time. We want to separate that. So every draw engine will get a reference to a draw manager. So we can have multiple draw manager at the same time running. If you look in Eevee, when you press F, F, F12 for rendering, you have a status bar that, that goes up. But it doesn't go up when the image editor is visible. The reason for that is the image editor is also using the draw manager. But if you hide the image editor and every 3D viewport and every note editor, it will work. And then we have some, some, some cache where, where we store the shader that we want to use. And then we want to sync the scene data that, that, that we have to the render passes that we create. The syncing process is syncing. It doesn't do the drawing at, at all. It is a way how to structure the GPU calls. So the, the drawing performance in the end will lead to less context switches on the GPU, which will then render faster than if you do it in order. We first get the, the, the, the current draw manager and store that in the local variable. We clear the, the, the passes that we, we have, because that could still be filled with data from the previous pass, from the previous time that you draw. We get our shader and we set our shader. Inside the pass, the actual command set the shader is being readjusted. It's not being executed. It's only being readjusted. We bind the texture, which is also only readjusted, not being executed. Then how do you want to apply your final pixel to the, to the render target that you're selecting? We want to write to the depth buffer and only when the depth is greater than the actual pixel depth. And we clear the, the depth. We do the same for the sim, similar things for the, for the, for the shading pass. And then, and then the, for each object in the scene, this function is being called. And during this function, we first get the geometry of that object. If it is a mesh, you will receive the geometry. If it is a light or a camera or another object, you won't get any geometry because it doesn't have geometry. If it doesn't have geometry, you're finished. And if, if, if it has, we are going to continue. There's a lot of performance improvements that we're currently working on to reduce the number of draw calls. And we're going to use some instance-based drawing to, to draw multiple objects at the same time in a single draw call. So in the, every object has its own transform matrix. So you want to store that some, some, somewhere. And we first select from, okay, in which index of a buffer will be then the, the transform matrix for that specific object. And then in the pre-pass, we will add that geometry with that resource handle. And then finally, we're going to draw. We select the current view, which contains some information about how to render, how large is the screen, that kind of information. And then we submit the shape, the pre-pass and then the shading pass. And this way of working, this is really a simplistic example. I tried to, to, to use our basic engine, which is already much more complicated that I couldn't show, but it gives you an overview how all those draw engines work. But then you have multiple draw engines that work at the same time on the screen. And how do we do that? And basically, we have two render targets that we can draw into. One is, is, is, is a linear buffer, which is being used by all the actual draw engines, like EV, like workbench, like grease pencils. They all draw in scene linear space. And then the overlay will render in, in screen space, which is sRGB. And when copying the, those two images to the final framework buffer that will be visible to the user, we apply the correct color management and apply them together and copy them to the screen. So basically workbench could generate something, some image like this. The overlay engine can do create something like that. And eventually this is what the user sees. So I'm going to, let's see how much time I have. I don't have that much time. I'm going to skip this one. It was theoretically. I'm going to do a small demonstration about how we debug or, or code because it is a bit different than what, if, if you're not used to doing actual GPU development. For that, we use a tool which, which is called RenderDoc. RenderDoc is an open source tool, which you can RenderDoc.org, you can find, you can download. If it fails for, for, for, for you download the previous version. It's, it's, it's a common form in the AMD implementation is, is better, but the NVIDIA implementation can fail. And then you just have to revert one version or two versions back, and then it will work. When you start RenderDoc, it sees like this. And you can point to your executable and you can say from, okay, I'm going to launch. And it starts launching Blender. And in the small, you will see some, some text here. Don't mention it. It says press F12 to capture, but F12 is also Render. So it's a bit what, what, what I normally use is, is this, this frame, this page that you, that you, that you get. This is the Blender process I'm, I'm working on. And I can actually capture a full frame and every co co co co command that will be given to the GPU. And I can say from, okay, capture a frame and it captured the frame. The frames you're, you have been captured are here. And if you open this, this frame, it looks really difficult to see what's happening. But if you use the minus, minus debug GPU startup parameter and you're launching Blender, not going to save this Blender will now add more metadata to the draw calls. It will generate names. So you can see it, but it will also mention from, I'm now busy on this task, and it can then group those commands together. I think it was this one. Yeah, it captured one. And now you see exactly during that frame, but what it did. The top bar was not changed. So there was no draw calls beneath it. The status bar was changed because the text changed. I rotated to get the camera. The action for some reason changed. Not sure why, but could be optimized perhaps. But it's more about the space at all. Let's see. Nothing changed. That's that's the point. The action editor changed because my mouse was over it. And the 3D viewport did not change. So it doesn't didn't record any anything. Happens often. But you can also say from, okay, record something in two seconds. So I capture something within two seconds. I'm going here. I hope it captured something. It captured something. That's still the action. I'm going to, if that happens, I'm going just to capture multiple frames. And now you see that the 3D space has been doing a lot of drawing. And you can actually see from, okay, what did the workbench do? And you see the different passes, random passes that the workbench did. And in the texture viewer, you will see something nice. It will, the prepass draws, we don't clear texture if it is not needed. And for the for the shading pass, we checked the depth buffer, which is only used. And only when the depth buffer has value, we will read from the color buffer. So you see this, which is not a cube, but it is a rotated cube in multiple frames. The same with the normal buffer. I forgot which buffer that was. The object ID buffer. And then we have the depth buffer. Let's see if I can, there it is. So the depth buffer contains the data and that will then be read by the shading pass. And for each draw call, you can also go into all the different stages that we had. So the vertex shader, what happens before, what happens afterwards and see all the data. It's more interesting to do that in the, in this pass. So we have a position in the definition of our normal buffers. You can look into it, but you can also view it. You can view the data when, before the vertex buffer, the vertex shader was applied. And you can view it when it was executed. So afterwards. And in the fragment shader, for example, you can check, you can actually see the whole fragment shader, which is 1000 lines of code. But the main function is reasonable clear. And actually, you can change your code here, compile it and run it. This is actually what I want to present for today. It's a lot of information. Please look back on it on the video stream. These are currently the projects we're working on. We're doing a lot of refactoring at the moment. And if you want to join us, meet us at the EV Viewport module on Blender Chat. Thank you.