 It is 2.30 p.m. UTC plus one and that means it's super charge time. Hi, I'm surma and I'm Matt and we are going to do some webGL So buckle in let's go. All right time to do some webGL. I Think you have never done live coding like this before no not like this You have done it alone though and maybe some people who here today will have seen what you did You have done it's possible. You have done a gift decoder job script Quite interesting because if you haven't done it the gift format is a bit weird But today, I mean I was supposed to say we're gonna do something less weird, but then again, we're doing webGL which Not less weird. No, no not at all. So Let's talk about webGL a bit before we get into coding straight away because I feel like There's a lot of misconceptions out there about webGL and openGL and how it works and what it does Mm-hmm, and I don't know help me. So the first thing to realize Something that confuses a lot of people is that webGL is OpenGL the API is basically the same. Yes webGL one is openGL ES2 Which may not mean anything to you But like if you find tutorials on the web for how to do openGL pretty much directly applies to webGL Because I think what I the one of things that I do know is that is pretty much one-to-one The C API put into the JavaScript world more or less Which is not necessarily a good thing because if you're used to web APIs, this is going to be a whole lot different So there's a lot of weird stuff that just it's stuff that just looks weird because it doesn't look quite JavaScript Yeah, the next thing to understand is so people think of webGL as a way to do 3d graphics and that is what a lot of people use it for but When you're doing 3d graphics with webGL or openGL you have to do a lot of that work yourself Really all the openGL does is draw triangles on the screen Really fast. Yeah, because it doesn't in most of the processing in parallel, right? graphic cards are Basically cards with process on them that have up to like 10,000 100,000 cores something 16,000 is something Ridiculously high and you can run code on them in parallel. So I think we should start by you explaining what you're going to do and how it's actually going to work But before that I want to say hi to everyone who's watching hi as always I'm going to be in the chat So if at some point you find a bug you have a question or don't understand something you can use the hashtag super chat on Twitter or use the chat on YouTube on the live stream and I'll be monitoring that and will throw you like a little curveball every now and then To see if I can trip them up or not And that's on my side. Let's go. Okay. So what we're going to be doing so As I said the webGL is for drawing triangles and you can do whatever you like with that So what we're going to do is we're actually going to do some 2d image processing. We're going to take an image And draw it into a canvas and we're going to do some manipulation of it change the brightness or you know Maybe make it change brightness across the width of the image filter. We'll think of something and Yes, we're just going to process an image and turn it into something slightly different So why now we're just using the normal canvas but using a webGL context for that. Yeah So it's going to be the same HTML canvas element But you can access it in a webGL way. Yeah, we'll show in a sec But yeah, there's the context the way that you talk to This piece of screen estate. Yeah. Yeah, so instead of getting a 2d context, we'll get a webGL context Okay, so Let's talk about how webGL works. Yeah, because I don't think it's something we can Expect everyone to know about not everybody has done game development or open GL tutorials necessarily and it is quite a bit different from I want to say normal APIs, but the APIs usually come across with when doing web development. Yeah Okay, so on my table, I'm going to show you roughly what What webGL does so with webGL you give it a bunch of information which Has to include the positions of some points on the screen. So you define some points Define some points and the You write a small program that you send to the cover's card called a vertex shader That takes in whatever data that you that you give it I think I was wondering it's called a shader But nowadays it is just an arbitrary code program in the graphics card pipeline It doesn't only do shading it can do a lot more, but originally I guess they were used for Creating shadows and that's what they call shaders But now they've moved on evolved into more complex and powerful beasts. So your Vertex shader, it's the first of your two webGL shaders what that does it takes some input that you give it You can give it whatever you like and It has to spit out the numeric data and it has to spit out the coordinates on screen of a point So on the table here, we've got three points. Let's say we have some more and Then what happens is that once every three points that webGL gets it assembles those into a triangle So tip first three points and make one triangle tip the next three points make a new triangle So what this vertex shader does is convert your arbitrary data which in the game Context would be I guess the points of your 3d models that are somewhere in the 3d world and Would convert them to on-screen positions So in the vertex shape is where usually the whole perspective distortion thing happens Which we're not going to do today because we're doing 2d image processing Okay, I think you Okay, so you run the vertical your vertex shader runs it gives you the points the graphics card goes away and turns those into Triangles it then does Rasterization which is that it takes each triangle and it works out Which pixels on the screen Intersect with the triangle So here's my beautiful as of now it was just screen coordinates which doesn't necessarily correspond to individual pixels It's just floating point numbers on screen coordinates And then it figures out for you if these three points form a triangle Which pixels would be affected by that triangle right and then for every pixel that is part of that triangle It then runs the second shader called the fragment shader and this basically It runs it can have some information that's been passed through from the vertex shader or from elsewhere But it basically say has to output what color is this pixel? so the vertex shader says Where are the corner? Where is yeah? Where is this point on screen? Yeah, and the fragment shader says what color is this pixel? That's all that they do there's a distinction between fragment and pixel, but I don't think it's worth getting into Yeah, I still haven't understood it myself So let's just let's just say the fragment shader colors pixels and we can work with that in other in other graphics libraries Like the indirect text is actually called a pixel shader Exposure I think okay cool. So that's the basic process We need to provide We need to create two shaders a vertex shader fragment shader one which outputs positions one which outputs colors And then we need to bundle up some data to pass to the vertex shader so that it knows how to get those positions and then and then it should draw our thing on the screen as the basic process, so let's try that I would say okay, so let me We'll come back to the sec so code so html file It's literally just a body with a script tank to load our script We're gonna do much else do we just gonna do everything in JavaScript in this case So our app.js here First thing we're gonna do is we're gonna create a canvas It's where all the magic starts So again canvas you probably know it It's this DOM element that you can get access to with a drawing context and can start drawing on it But you can also get a different context So usually you use the 2d context to have line drawings or ellipses Some form of visualization or animation But you can also get a webgl context which Turns this context object into this very thin wrapper around the open GL API Exactly So here I'm gonna get that context. I'm gonna put it in a variable called GL mostly for Consistency with everyone else historical reasons that context gets called GL. That's true I think any every tutorial calls it GL so so oops we get a web. Yeah, yeah, we're GL context Then we can do some you know basic stuff with that we want to say hang on so The webgl context can draw on just part of the canvas. So one of the first things we're going to do is set the viewport Viewport so we're gonna say it goes from zero zero to the GL drawing width GL drawing height This is saying I want the webgl part to output the whole canvas want to fill the canvas So by default, I think the canvas has like 300 by 150 pixels or something like that Yeah, we can change that but we're gonna change it later Maybe but this basically just ensures that GL knows what kind of what size of a canvas we're working on Yeah, and to make it so that we can actually see where it is We're gonna set a clear color for the for the canvas which could be anything I'm gonna make this one sort of orangey Can't type and think at the same time It's an overrided skill for software engineering anyway, and then we're gonna say that the canvas should be cleared Okay, so over here Can refresh here? Right here. Okay So when you clear the canvas There's multiple things that can be cleared. Yeah Don't worry about that But we need to specify that we want to clear the color buffer, so we use the color buffer bit So one thing that even if you haven't done 3D program You might have heard of that you have the buffer for the image that you're rendering But also a separate depth buffer where you store how far away an object from the camera So you know if another object should be painted above it or not be painted at all So that's something you can clear separately here We'll only care about the color buffer bit because yeah, we don't want to do any 3D once again There is at least one more of a buffer that we haven't mentioned, but like yeah as I say buffers We're clearing the colors. We're clearing the colors. That's the one that's being visualized and we clear it Okay, so there is our canvas in the corner orangey as you promised great awesome great success Done see you next week Okay, so we've we know that our Our canvas has been created and that where we're working with it live brilliant Not what we were here for though. So I said that we need to create some shaders So we're going to take Some source code that we write and compile it into a shader the the language that you used to create shaders is GL SL It's kind of see like kind of look it's supposed to look like see but it does kind of work differently There's some things that are nice about it some things that are not nice about it So they're nice things because it's to do with vectors and matrices and things like working with those is super easy But it's very unforgiving of syntax Yeah, I remember earlier in the day It used to be even harder to write shaders because there was no error messaging whatsoever You would it would just not work you get a black screen and you would sit there and look for time This is in my colon. Did I mistype a variable name nowadays? You do get some error messages to make it a little bit easier to figure out what you did wrong. Yeah Also because I haven't mentioned it yet. This is not production ready code Oh, I mean it's not even good I would probably make sense to in the end wrap this up in a custom element But for now we're just gonna because web GL is is quite the complex API because with being powerful there also comes a lot of Abilities to do things wrong and a lot of things to do in general We're gonna have a bit of setup work. So we thought we would use as little distraction as possible. Yes So you're not even putting the shaders in a separate file You're just gonna inline the code right away because it's just yep. Yeah might as well I agree. Just go straight into a template string. So here I've written The part of my vertex shader and it's gonna set this GL position Variable, which is as we said before this shader is all about outputting a position And we're gonna make it. It's it's a 4d vector. It's like a four element array There's x y z and w W is like magic for perspective stuff. You don't have to worry about it We don't do perspective today, so we can just ignore it. Yeah more or less. It just needs to be one Then it works the Z we're gonna set to zero because again, we have no depth if you're curious about the fourth coordinate Look up homogeneous coordinates on Wikipedia. It's quite the ride So we actually have a first example here is something that GL SL is good for which is that I'm gonna I'm creating a vector for I'm gonna use a vector to which I haven't defined yet And then just pass in the extra two parameters to make it for elements long So here we're gonna say that this shader needs an attribute which is a Vec 2 that's a two element array basically that we're gonna call position So this is the data that we're gonna pass into the shader, which we'll show how to do later We're gonna take that position, which is going to be a 2d coordinate And then we're gonna expand it using this Vec 4 Yeah constrictor into a 4d vector, which is what GL position needs to be That shader is now done. So I just turned the two dimensional position into a four-dimensional vector So GL is happy, but that's all that's happening really. There's no arithmetic going on or anything. So precisely our Our fragment shader, which I'm gonna shorten to frag shader the the source is Similar it has a main method. It's gonna get called Here the output is a color, which is the variable GL frag color This is also a vector for so it's The red green blue and alpha Unlike when you're doing like CSS colors and things all of these are float very float floating point variables from zero to one So what color should this one? Let's go for something a bit more purpley Whatever. So that basically tells us if Any pixel that is covered by our triangle will now get this color assigned. Exactly. Okay And there's just a couple of extra things. Well one extra thing I have to do here for some reason WebGL doesn't like defaults for certain things. So we actually have to say Precision high P float. We have to say what precision all of the floating point numbers in the shader are Is it ever? Not high P. A lot of people use medium P. I guess the P stands for precision. Yeah I guess it makes it faster. Yeah, if you use medium P, but less precise Well, we only have so many pixels, right? All right, cool So now I could have used so now we are basically we're just getting data in we're using it as a position straight away and Then we're just saying everything that is covered by triangles is now Purple purple Yeah, okay. So that's our source code. Let's make it into let's make it actually now It's just a string. We still need to tell so the thing is it's a language, right? And usually would think a Sealike language needs to be compiled. Yes so First of all, we need to create some objects on the graphics card So WebGL has this great love of creating an object On the graphics card and then like making a pointer to it. So We create fairly old API. Yeah, so we create We create a shader and we say what type it is this one is a Vertex shader. I said that there's two shader types. There's a vertex shader and the fragment shader I think in newer versions. There is the geometry shader, which we're not going to get into today because Crohazy and you in fact have Our newer versions of OpenGL not WebGL you have the compute shader as well for doing just arbitrary computation We don't have that We don't need that for a little bit of 2D image processing No, so we create our shader objects, but these have nothing nothing in there. It's just like a no the strings are strong JavaScript. Yeah, it's just a place on the graphics card that we can refer to Literally just an idea this we have a handle on them. I think it's called the vertex shader and frag header variables are handles for these two shaders and now we got a Like push our stuff into there. Exactly. So we say that for the vertex shader handle we're going to attach the vertex shader source the string the source and We're going to do the same for frag shader So we've attached the source and now we can compile I was about to ask if that was already the compile step and apparently it's not No, no, it is not. So I'm not compile source compile shader. So we're going to compile the vertex shader and And Jill compile so for all the people who've joined us in the meantime because you know not everybody can get to a notification that we go Live straight away. Welcome. This is super charged. This is my colleague Matt and today We're going to do some 2D image processing using WebGL and what that means is We are getting in the deep end. We are writing shaders and stuff So so far we have set up a little canvas and we have turned that canvas into a little OpenGL context and now we can use the OpenGL API from the web hence where it's called WebGL And I'm just about to upload our shaders to the graphics card compile them So that we from now on can determine how the positions are turned into pixels and how these pixels are then colorized Exactly. So we've got a we've got our shaders and they could be used independently We need to say that they work together in a thing that because if the one is supposed to pass data from one to the next Because the vertical outputs the position And it's trying to triangles and I send use in the fragment shader. They need to be compatible, right? Yeah, so at the moment like they don't interact but later on we will add a way in which they interact and they will Have to be compatible so we can have We could have hundreds of different shaders and can combine them in many different ways And we combine them is in a program So we create a program again. This is just an object on the graphics card and we get an ID back a handle to it Then we attach our shaders Yeah, attach shader so we use our program and we use our Vert shader and then Same again with frag shader. So we attach the two shaders to our program We then link them so we call link program so we have compiled the shaders They're now as like programs on the graphics card now. We link them to one program telling that program that they need to Work together. Yes. So at this point it validates that they are compatible, okay? Does the plumbing in the graphics card to make it into a single single runtime? Yeah, whatever it does all graphics cards are different and they tend to be secret Exactly what it does. You just see the API Black boxes always good Now as I said, we could have hundreds of shaders. We can also have hundreds of programs Linked together, but only one program is used at any given time. So we finally have to say That we're going to use so use program and we're going to pass in oops The actual program so you can have multiple shaders linked to multiple programs, but You only can have one that the data that you are that you have on the cut is being passed into so you can switch programs But it would be separate runs of the graphics part. So this week So for more complicated things you have different shaders that do all sorts of different things and during one frame you might Set things up use one shader Draw that set things up for another shader draw that you use multiple shaders in a frame So you multiple programs in a frame and those programs can be convert composed of whatever shaders you like So we've got our shaders set up linked them into a program. Cool Now we're ready. I'm as good to see that the chat is coming alive with people. Hey, everyone As always ask questions ask away. I'm gonna forward them if appropriate You can also use the hashtag supercharged on Twitter, which I'm monitoring If you want to like actually understand how we draw things because we're close to actually having something We actually had something on screen already. We had a yellow ish rectangle, which is an achievement. It's super exciting That's kind of purple Okay, so First of all Tablet again literally back to a drawing board. Yes So Oh Here's what I made earlier Which is a thing so Blue Peter reference there for all you UK kids Okay, so we want to draw an image which is a rectangle And we can only draw triangles, but luckily we can because I would open gel does best draw rectangles and do it in a highly paralyzed fashion So we have our rectangle So that's the fun thing about Thinking about open GL web GL that we are gonna more or less run the pro the shader program that we wrote in Parallel in a way that each pixel has its own thread more or less More or less so because graphics card have like 16,000 cores. I think you said yeah, we can do that So yeah, each coral probably used for like a hundred pixels or something. Yeah to make it the whole image Madness, okay, so we split up our our rectangle for our image into two triangles Ingenious So the reason why we have to do this is because if there was no triangle our fragment shader wouldn't get run and we couldn't change The color of that pixel, which is what we want to do exactly so we're just gonna say We know our Frank vertex shader says where the corners the triangles are it works out and then we'll find the Pixel colors within that triangle with no triangle. There's no cause So we need to say where these corners are to make two triangles on the screen Now in web GL in our shader when we set those output coordinates Those coordinates for various reasons go from minus one to one. Yeah minus one is the left or the Bottom of the screen and one is the top and right. Why it goes up X goes right on the screen I know this is merit free. So I should be doing this Y goes up X goes right But it goes from minus one. It's a left to the left border of the screen or the area and plus one is The right border. Yeah, I mean totally that's a way of So let's draw in our axes here whoops, it's a little little wonky So origin here zero. So this is Minus one as pen is a little Settle a little thick, but we'll get by it's readable. Yeah, that's not a complex number. So it's just ah See what I did there. I didn't even intend to do that So the coordinates of our triangles it's over here. We have let's look well Let's label the the corners around like this I don't know why I did them in this order. I just seem natural. It's not even the mathematical order. No this one minus one So it's an interesting question like why don't we just use regular JS with canvas? Because this would be so much faster or it's gonna be so much faster if we're gonna do proper image processing where Each pixel will have a considerable amount of math and maybe even dependent on neighboring pixels That is doable with a normal canvas and JavaScript But it's gonna be slow because you will have to run in a for loop and you would run this Single thread over the entire image and if you do that on full screen resolution On a regular display, you're gonna have a good time with GL It can do every pixel in a separate thread and it's gonna be snappy fast arguably 60 FPS So you can I would be amazed if it wasn't yeah I mean unless you do really really like I think like really like blurring filters at high convolution rates We'll probably not make it but Yeah, that's all we're gonna do today. Okay, so we've got a we've got our four corners We're gonna make them into three triangles where we're gonna do that as we're gonna have We're actually gonna make our two triangles. There's gonna be triangle, which is a B C and another one, which is a CD yeah, so when we have a vertex shader It takes It takes some positions. Yeah, we need to pass in Those positions in the correct order So we just figured out the letters of the triangles now We can replace the letters of the actual coordinates because we also have them in the diagram Yeah, so let's so let's write in our vertices now when we're passing data to the graphics card Generally, you want to use an array buffer Well, you kind of have to use an array buffer because you have to specify the The format of the data so that it fits nice is a binary transfer the graphics card works binary so yeah So we're gonna create a new Float 32 array We're gonna create it from some data, which is oops Which is our coordinates, which I believe were minus one minus one minus one That is the bottom left. Yep. That is the top left One one top right. So you have our first triangle. These are the two Dots start from the bottom left again Then we move to the top right and then we move to the bottom right Interesting question. What is an array buffer? I guess it's something that you don't necessarily know if you don't haven't gotten in contact with this kind of thing an array buffer is JavaScript's take on having raw memory where you actually have a chunk of sequential memory And you can have arbitrary data in there and then you can figure out how do you want to interpret these? Sequence of bytes because you can look them in chunks of four and say these are 32 bit floats Which is what we are doing you could take the same data and interpret it as integers or Something else and and this is the way to make sure that we are actually Transferring a certain memory chunk to the graphics card. It has the correct format if we just took an array It would technically be up to JavaScript to figure out is this integers is this floats and this way we can force a Representation so we can transfer it Yep Okay, so we've got our data now. It's to like Send it to the graphics right now. It's still only in JavaScript. So again, we need to create a WebGL object and this one is going to be This is going to be the vertex buffer. It's going to be a a buffer Very simple object. So again, we just get an ID back and now we're gonna say what to actually do with that So buffer just again chunk of memory. Yes this time on the graphics card though So now we have a chunk of memory with data in it in JavaScript a Chunk of memory on the graphics card with nothing in it. Guess what we're going to do next Well, that is that yeah almost first of all one of the things that's a little weird with WebGL buffers is that you can't just say create a buffer and then store data in it That'll be mad. That'll be almost simple. No one expects that So what instead you have to do is you you have like named types of buffer. So what we're going to do is we're going to say We're going to bind Use the bind buffer method to say that the named buffer array buffer points at Our vertex buffer. So if I understand correctly, you can have as many buffers as you like But only one of them tend to take the role of being the array buffer at any point in time So it's like a slot you say like this buffer is now the array buffer and you do your work and then for the next Draw you can take that out and take another array buffer. You are now the array buffer. Exactly if I wanted to Populate a hundred buffers. I would have to do this bind buffer a hundred times. I see you buffers. Yeah, okay Yeah, now that we've bound that we can then Send some data we can use the buffer buffer data We can buffer some data So we say what the target is and this is that array buffer. So as I say, it is not vertex buffer. It is array buffer sure So you don't have a reference it by the vertex buffer variable by just by The buffer that has the current role array buffer. Yes We have like this level of indirection in there, which is a theme you will find a lot in OpenGL by the way In this case, I could have put the create buffer call right in here because it's literally the only time I use it This is nice. This is allows us to explain that there is two different concepts but you know okay, so Buffer data, so we're we say what the data is it is our vertices vertices And then we say we're going to give a hint to the graphics card about how this is used and the usage here is gl Static draw There are a bunch of options here, but this one basically says This data doesn't change so in theory the graphics card can optimize for this Some graphics cards do some don't some optimize badly Some you actually want to use the wrong kind because otherwise it makes it slow But we're going to be we're going to be good citizens and say this is static because we are never going to change this buffer data Okay We've got the data now we need to hook it up to our shader Now in our shader, we've just got this We've got an attribute and we've given it a name. So this is yeah Through this variable our data will come in. Yes. This is how it will come into the shader So in order to Say how our data matches up we need to get what we need to get an id for this variable um, so we're going to use we're going to say that the Position location, which is a wonderful Little linguistic It's the location in the shader of the position variable Oh, so it's like It's in this location in your shader script so that you can basically then in everyone inject a different value Exactly. So in in the shader when it compiles it assigns an index to every variable So you can from outside say i'm putting this data into index zero And it will be Whatever variable is has index zero. It's that you're not referring to the names. You're not passing strings around Okay, again a level of indirection. Yeah, cool. So we get an id doesn't that juggling all of this in your head Exactly. So this is uh, sorry get Atrib location Uh, so we're looking inside our program For the attribute called position Ah someone asks so these c programs and that person is referring to the shaders. Yeah can Receive user input somehow by bypassing the browser No, no But we as the developer can set them to values And that means the run changes. So we that is our hook into our aesthetically compiled program To make our scene look different. So one example would be you could say this is our camera And then the next one we don't have to recompile the program but can just change That bit and now the camera will be in a different position and the scene will render differently So it's basically like if you want to think about it like c programs. It's like command line flags Yeah, in a way maybe. Yeah Um Yeah, the the web gels it has no access to anything that you don't pass through from the JavaScript side It's completely I said unless you as the developer give data to it it being the engine It'll be a pretty serious flaw in the web sandbox. Yeah, that wasn't the case Um, okay, so we've got the location So now we can say that um We can point the data, uh, correct. So we say vertex Atrib pointer. So we're going to say That the index which is the position location um The we're going to send through each time it's going to take two values So the size of uh the data for each run of the shader So, yeah, it has a chunk of memory. There's some data, but it doesn't know what formula data has We know it's pairs of two because our screen coordinates and they're six in total Yeah, but we need to say that we need to tell the engine that that's how you're supposed to go through this chunk of memory Exactly. So on each on each run of the shader take two values out of the array Um, the the type of those values is float Uh, normalized is false and I believe it is specced that just normalized is false I don't know what that does Stride this is how far is it from the beginning of one set of data to the beginning of the next set of data? So if you had so you've got like 10 bytes of information or 10 floats of information per Vertex in your array, but you're only passing two through at a time might say that the size is two But the stride is 10 so that you you move 10 forwards to get to the next It's not relevant. So you can skip over that for every year. It's sort of yeah, it allows you to skip data Uh, we're going to set that to zero, which means just use the size Yeah Which is handy and offset how far into the data does it's into the array does the data start it starts at zero So that tells the shader how to look up the data. Yeah Almost there We need to enable vertex atrib array This is position location So we need to enable this, um So I'm guessing you could have Um If you say you can skip it you can probably say the first two bytes of my 10 bytes are a position um, the next two bytes are The color of this vertex or the normal or whatever what have you and so that's why you need to say these You can enable multiple of these vertex attribute pointers. Yeah. Yeah, exactly Um, so in this case, we've just got one as two bytes on so here we say That we want to actually enable This location. Yeah, so that it will every time we run it take that data Because we could say that the shader doesn't care this particular shader in this run. It doesn't care We don't actually want it to take that data here. We do Um, so we've enabled that Now we can finally do some drawing Ooh exciting. We're getting a champagne Um, so we call the draw arrays method because we are drawing based on On arrays of data Our mode is triangles Because here from the the autocomplete that there's triangle fan triangle strip. There's point and line. Oh, okay. We're doing triangles Because it is the most useful thing to do. Yeah, we yes, um So we're going to draw some triangles. Uh, we're going to start with triangle zero Uh, and we're going to draw Sorry, it's not triangle zero. We're going to make it from vertices So we're going to start with vertex zero and we're going to go We're going to draw six. Oh, that's confusing a bit. I mean you say i'm drawing triangles But then you tell the number of vertices and the number of triangles Exactly because that means what what if I say seven? Uh, then it will try to read beyond the buffer Uh, I don't actually know everything will explode because that doesn't make a triangle That's why I would say I would expect it to be a number of triangles because then it can't go wrong unless you go negative or something weird, but If a triangle fan and triangle strip, oh, they can yes, and small anyway, not today Um, so let me just check that I got all the parameters there. Yep. Just three Deep breath. Okay So the moment of trip is happening. We set up the canvas We compiled and linked our shaders. We got some vertex data We assigned it. We we sent the data to the graphics card. We assigned it to a location in the in the shader Will it will it draw will it draw? Let's let's check our yellow triangle and it turns purple first try. I'm legitimately impressed This should not have worked on first try. That's just like writing reg axis that are correct on the first try so, I mean Proof that it's still that it's actually the yellow is still behind this because we have cleared up with yellow And drew drawn two purple rectangles on top So what we're going to do is I'm going to move in uh move one of the the corners in a bit So that the triangle you should have like a little yellow if we move this one the minus one one Minus one one. Let's say make the Five and not point five There we go. So you can see the corner has moved in Um and the orange is still behind it. I believe you know All right, so that means we have now reached the point where our entire canvas is I want to say under our control It means for every pixel on that canvas our virtual fragment shader is going to be called. Yes, so Let's do something cooler than pink. I guess so, um I'm gonna add in One step at a time It's slightly different to What I told you I was going to be doing so don't freak out So one of the things you can do is you can calculate information in your Vertex shader and send it to the fragment shader and um A cool thing happens because obviously in your vertex shader, you only only runs three times per triangle But your fragment shader gets run hundreds of times depending on how many pixels are Is that it will interpolate the values so if So let's say that the uh top corner Let's go back to the diagram here. So let's say that this corner up here This corner here is red So this corner up here starts red and uh This corner down here. Oops is Also red so nothing changes Is yellow then what will happen is that you will get a gradient So as it goes from here to here, it will go from red to yellow So it will so we assign data to our vertices saying this vertex top left has the additional data The color red the other one has the color yellow and for every pixel in between That doesn't actually belong to a vertex. It's going to just interpolate by saying Okay, this is like 50 percent in the middle. So it's going to be 50 percent red 50 percent yellow means some kind of orange. Yeah, and so we get a gradient for free So, yeah, it's not really uh, so I said colors, but it's just some data So, uh, we already showed that a color is a a four value array of a vector four Yeah, um, so I could say that the vector four for the b corner is A color that represents red and the one for the a corner is a color represents yellow And the individual numbers in those vectors will gradually Go from one to the other as it goes as it goes. My explanation was a little bit high level. I admit I was just saying it's going to mix your colors. It's actually just the numbers the raw data It's one of those things that bears saying twice just because it's uh, like a lot of it's very helpful I guess something I can really come into advantage. Yeah And uh a bit mind-bending sometimes. Uh, yeah, so that's what we're going to do is um, so Um, I'm going to set it into through what I'm actually going to send through is the coordinates So the the coordinate of the corner of of each corner. I'm going to send that through from the Vertex shader to the fragment shader so that we have a value Which is our actual coordinate within the rectangle Oh, okay. So yeah, basically in the coordinate system from minus one to one You're going to change that to be uh, yeah, so Uh, yeah, so I will change that to be from zero to one because we need that later when we're looking at textures Okay, but that is uh, yeah so Uh, first of all, how do we communicate between the the shaders? uh, so we have a Thing called a varying which means that it varies over the the coordinates Um, it's going to be we're going to use a vector two And I'm going to call it text chords. This is the coordinates for the texture as as I discussed Um, and we need to set that Uh, same thing up in both shaders so that they can be linked together and just by the nature that they have the same name The gl engine is ultimately going to figure out that whatever I put Out in the first vertex shader is going to be put into the same variable in the second fragment shader Exactly it will automatically link these things together so magic so in our vertex shader will set what that is so the text chords it's going to be Now there's a bit of magic here. So it's going to be our position which is already a two So our position is the one that goes from minus one to one. It's our screen position, right? The thing we drew earlier minus one to one minus one to one so for each coordinate what we need to do is we need to um Multiply it by sorry add one to get it from It's from minus one to one to be zero to two Yes, and then divide by two to get it to zero to one There was a lot of zero ones. I think I believe you say we we shift the entire thing to the right I'm going to try to make it from the view so we have your minus one one We add one so it goes from zero to two. We just shifted the entire thing and then divide by two We'll just scale it zero Um So one of the things you might think you have to do is okay, so text chords dot x equals position dot x Is text chords something that is a predefined thing by gl or something a name that you have chosen It's a name that I have chosen so we could call it buildable. Yep We're not going to because that seems like a bad variable name. Okay. Well, it was worth a try I can always do that but you know, yeah, so you you might think you have to do this You have to set the x and then the y but Because shaders are designed for doing things with vectors and matrices You can just do it to the whole to the whole vector We'll have to do that So what this will do is it will take each element of the position and add one to it and then each element of that vector and divide it by two magic That's pretty neat. Yeah. So now when we look up text chords in our fragment shader It will have the interpolated value of that. So we can actually use that to get the color Also, you can just visualize it basically. Yeah, so here. So like I used Up here. I used our 2d vector to make a vector. Yeah to complete the vector. Okay, we're gonna Do the same here. So this is going to be our text Okay, so let me try to predict this. We have now for each each vertex It's gonna output Its own position and text core but instead of using the minus one to one coordinate system It's used going to use the zero to one coordinate system correct And then we're gonna these are going to be interpolated for the pixels in between And be used as the red and yellow parts of the color. So that would mean That we go from having blue in the top left corner to white in the bottom right corner and the other two colors on the other corners I just can't figure out red and green red plus blue So turquoise and purple. Well, let me know no because those are the zeros when it's zero green Oh, plus the blue. Yeah, you're right. You're right Well, I uh, you're almost right. I think you got the corners the wrong way around. Oh, no Oh, yeah, I don't know why it took a while to uh, it's All blue. I expected flipped Yes, it is it is flipped because Zero zero is now the bottom left corner So it's gonna be white and oh, yeah, right the coordinate system is yeah I was usually in on the web The coordinate system starts top left. Yeah and gel does not it it grows towards top and right Yeah, so it doesn't help that in our explanation left and right are backwards We're using web gel top and bottom are backwards. So when we're explaining it on camera with our hands, oh god So it's usually wrong. Uh, that doesn't see that does seem a little wrong. I feel like I've done something wrong, but um Looks good to me. There's no there's no red or green. Where would that be? You have always blue or sorry There's no Does it go from purple to cyan? I'm actually colorblind. This is a terrible demonstration for me It looks correct Yeah To everyone who's joined late welcome. Glad you joined anyway because the more the merrier we are doing Some web gl and what we're going to try to do with web gl is something that has nothing to do with 3d We're just going to use web gl as a really really fast Triangle drawing angel and do some image processing. So if you think of any kind of filter you usually apply to images We're going to do that in real time With web gl because we can because it's actually quite interesting So, um We can mess around with colors like that. Let's do something more like what we said we were going to do at the beginning We're going to use an image Ah, so considering we're using we're working in web gl that probably means textures Exactly Uh texture It's just some data, but it happens that uh, you treat it as an image and you can put an image in it So it's it's an image. I see a recurring theme It's just like the graphic cards take some random data and then you teach the graphic card how to interpret that data Yeah, so we're going to create an image Uh, this is just standard dom I think she wanted new there. Oh, yeah. Yeah. Now I can this I actually know this is javascript APIs. I know those Uh, the source is going to be uh, I called it image.jpeg Somebody just said time to build the game of life and web gl I think it has been done with somebody implemented the web the game of life in just shaders Which makes it really really fast It's out there somewhere. It's good fun Uh, so when our image is loaded and is ready to be used, let's do All of the rest of our stuff Oh, so we're not going to do anything until the image loads because we need it to be ready to upload to the to the credit card Graphics card graphics card. Yeah, don't upload anything to your credit card. It does have a chip on it But I don't think it could hold on to this. I mean, um, okay, so, uh Obviously in production code, you know, you might be using request animation frame to say when you draw things Um, I'm just going to say once the image is loaded. We're ready to go So how do we use the image we need to create a texture object? So we so yeah Basically saying again, I need a chunk of memory to put image data in instead of random vertex data Now we have used it couldn't guess it is create texture just like we had create buffer create program create shader Um and create everything else. Um, that is literally we call it create everything else So for our textures, um First of all, uh, we need to do the binding. Um, but there's actually two levels of binding here um So You need to say which unit the the texture is going in because in the shader you can have up to well It's graphics card dependent But let's say you have eight textures you can refer to in a single shader So there's a limit you just kind of have a certain number of graphics cards on the graphics Textures on the graphics card Sorry, you can have you can have lots of textures on the graphics card Okay, but when you're in a in a shader, you can only refer to Up to a smaller limit refer to a limited amount in a single run of the shader um So we we want to say that we are using active texture is um We're just going to say texture zero. It's a nice number. Um, because it that's it's the Zero is the default for when we access the shader. Okay texture in the shader Uh, and then we actually need to uh bind the texture Uh, the name that we're using here. So before we have the name array buffer, this is going to be texture 2d The thing we're binding is our texture then we um Actually sets some set an image into that so text image to oh gl GL dot text image 2d So our target which was texture 2d as I said Yeah We are uh, so we actually want the other autocomplete here. So, uh I'm really excited to see that that's the last parameter of this function as you can see from the visual code autocomplete Is the pixels which can be an image, but it can also be Uh, oh, sorry, but it can also be A bitmap an html image element or an html video element. Yes, isn't that exciting? Lots of things can be drawn into a canvas, which is awesome. Um, so, uh, so what I've said is the internal format is rgba, it's going to have bright green blue and alpha the format of the image that we're getting is rgba because all image Uh image elements the DOM elements come through that way the actual type is So each of those elements red green blue alpha is going to be an unsigned Bite in the in the file. Obviously we get there's a float. Yeah the shader and the thing we're actually passing through is our image Okay, the thing we loaded at the very top that we waited on on load for yes I'm just going to pass that in because I guess that is the very slight Additional convenience that web gl has over open gl where it's like, you know You're on the web at the least I can do is help you convert Web image formats to open gl image formats. Yes. Now we have to set some parameters on On our texture basically the The graphics card likes to work with textures that are Power of two in size so that they are say 512 by 512 and move 2048 by 2048 Our image is just an arbitrary size. So we have to say how Yeah, we have to deal with that set some parameters that tells us how to deal with it. Um, so we're going to set um the Uh as it's uh the texture Wrap texture wrap s So wrapping uh can happen horizontally or vertically For some reason those coordinates are s and t I guess what it is for is that if you at the in the end you're going to pick Pixels from the texture and ask what color is this pixel What should happen if you actually quest a pixel is outside the image? That is where wrapping comes in should it wrap around and you just keep going in circles Should it just repeat the last pixel at the edge? These are the parameters we are setting here. Um, because for some reason it makes Open gl work with images on arbitrary sizes I think you said it before so it is basically A magic incantation to make it work Don't think about it too much. You don't have to do this with web gel 2 but with web gel 1 you do um, so we also set the the min if uh, um, yeah, what to do if The min filter and mag filter is if we're if we're blowing up the the image So that one pixel of the texture is going to take up more than one pixel of the screen What to do and vice versa if more than one pixel of the texture Is going to be used in one pixel on the screen. What to do here? We're using nearest neighbor filtering. Yeah, because that's um, you know what? Let's use linear. It will look better Basically averaging out if multiple pixels are squished into one screen pixel All the other way around it's going to interpolate between them So we can now access this texture in the shader. We've set it to be texture zero Which is the default one in the shader. We don't actually have to hook up anything there Cool. So here we need um, a kind of variable that's known as a uniform Which is a thing that is the same for every run. Yeah So for all the pixels that are having that are running this program in parallel A uniform is going to have the same value in contrast to a varying that can vary across parallel running Instances of your shader. Yes. That's what the two differences are The names are a little bit confusing because a uniform can change between runs When we draw the next version of our image next time we call drawer arrays We could have set the uniform to something else. So here the type is a sampler 2d, which is a thing for sampling a texture Sampling meaning what I said before Asking what is that pixels color? Exactly. That's taking a sample In gl speak So we're going to call the texture 2d function. We're going to pass it in Our sampler to say where we're going to get it from. Okay. I'm going to pass in our text chords To say where in the texture it fits so well As if we had planned this. So now our fragment color Should be the color of the image for every part of the image. Hopefully Let's see how that Boom it is upside down, but we have a picture Um, I mean the the flipping around I think it's something we can fix pretty quickly I would assume. Yeah, so one of the first things I'm going to do here is I'm just gonna Quickly magic up some style. Oh, yeah, make it big up So the canvas is going to be what how big shall I make it? I don't know what the aspect ratio is. We're just going to go Whatever I've done something different beautiful and it is vertically flipped because Being something I did not write down and maybe should have done So for the web There is actually an extra feature in web gel that is not in open gel Which says all my textures are back upside down Oh, that's convenient because when you get it out of an image element It is upside down from what it would be in Most open gel libraries Well, we're just going to do it ourselves because you know you're writing trainers anyway might as well Yeah, so in our case I'm just going to say that in our text chords. I'm going to say that text chords dot y Because one might one minus. Oops. I can't type Yeah, so that's basically what used to be one is no zero. What used to be zero is now one It's a flip of the y-coordinate and it should be the right way around So now suddenly the picture of Australia has become a picture I don't know Italy really So one of the things you might notice here is that I've made it bigger But it it's now very pixely. I don't know if you can see that on the stream. It's very pixely. I mean That is from even from a JavaScript side I know that's because the canvas still only has its default number of pixels which is 300 by 150 so we should probably Increase that as well. Yeah, so what I'm going to do is I'm going to say the canvas dot width can be the image dot Natural width so we're going to say whatever the natural width of the image is So the number of pixels in our canvas regardless of how it looks on the screen number of pixels in our canvas is going to be Whatever is in the original image. So if we stretch the canvas around whatever we like, it will always be just as good as the original image There we go much. So we're going to use this situation at this this moment to make a very short pause We're going to try to restart the stream real quick to fix the audio that's apparently going on We have been out of sync a little bit. So I'll wait for Apparently we're already fixed. I was quick All right, you may continue Thank you So now that we've got our image going Um, I guess I could set the uh No, right. So now we said image processing. So let's talk about how we might do some processing One of the things we can do do in our fragment shader Uh, we can do a little bit of calculation with this color perhaps So this is where we finally fulfill the title of today's episode Because we're not going to just draw an image. We're going to do make some image processing happening So we're going to take an image. We're going to process it to look differently in some way so One of the things we can do is There are various filters that exist. Um, if you use image editing stuff, you get all sorts of filters Let's change the warmth of the image um So I'm going to set up a um A variable I'm just going to make it a uh float And I'll call it warmth I'm going to set it for the moment. I'm just I'm just going to pick a value out of out of uh thinner um, now this This filter when I saw how this was done. I don't know if this is how it's done in every image app But um, it seemed magic So warmth filter you probably know is usually in this image app You can make a picture look cooler or warmer which gives it like a blue Light bluish tint or like a nice orange tint on beach pictures. A warmth filter is always nice. Yeah, so we're gonna Um, instead of sending this straight to the frag color. I'm going to pull it out into a variable. All right Uh, it's that I can manipulate it. You could still manipulate gl frag color. I'm pretty sure But this is And then What you do is you say the color dot r. Oh, so uh, our, um All of the vectors So vect 2 vect 3 vect 4 All the vector types you can refer to them with x y z w if you're talking about them as coordinates Or r g b a if you're talking that there's a color. It's just like an alias So here where I say color dot r. I'm saying it's just the first the first No, probably maybe I think it's just the the two so Uh color r minus equals warmth Uh, no, sorry plus equals because we want positive values to make it feel warmer and color dot b minus equals warmth Which if you know what that's doing color wise that just seems mad It's just going to bump up the blueness Sorry for when you make it warm, it's going to bump up the redness and bump down the blueness. Okay, and vice versa Um Yeah, uh, now what will happen here is that um potentially our warmth Our color values could go below zero or above one, but we'll just be clamped 201 is something that we're up to you on as far as just for free Uh, so it will happen in the graphics card when it actually gets sent as a pixel Yeah, like it will when it works out what the actual color set from the float value It will do it then. Okay, we can so it won't happen immediately. So color dot r So it's going to store really be negative Which sometimes is what you want if you do some further processing after and you actually want to be aware of like this overshoot or Under shoot whatever the right charm is Um, it's because the values are lost right away, but you can be sure that the stuff on screen Is going to be correct data and this looks much warmer now make it cooler Uh dead simple just make that negative Much cooler So again, you can do things like um, so brightness Um That oops, there are lots of there are lots of different ways of doing these filters that are um Like people have studied the psychology of perception and the the physiology of your eyes And they work out these cool mathematical models and then other people come along. Let's just say, uh, it's approximately just Add some numbers. So brightness is the same sort of thing. We're just going to say, um, uh Well, okay, we're gonna do it Color dot rgb We don't want to add to the opacity. Yeah. Uh, so this is uh, this is known as swizzling for reasons I'm not actually a hundred percent sure of where you take a vector and just use a part of it and we're just gonna say Add on the brightness All right, so now I have gonna have a cooler but brighter picture And it is in fact brighter. Yeah Amazing. Um, and because this is happening in the shade, we just built photoshop It's uh, yeah, I think it's a little way to go before we get there But I mean, this is the principle for a lot of these filters. And the thing is we could actually animate these parameters And it would run at 60 fps and I'm willing to bet that it wouldn't be as smooth if you choose javascript canvas If you had to do these operations on every pixel like add some values to the rg and b value It would a be painful and b be rather slow. So whenever you want to do like per pixel animations webgl is the way to go um I think we're pretty much On time now, so I'm really happy that we just were able to show the actual processing bit to happen Um, as always I'm going to ask you to give me the code afterwards So I can put it up on our github repository, which is in a new location. Actually, it's now in the google chrome labs organization Uh, it's the ui element samples the code is going to be in there. We're going to add some comments Um, so you can play around you can implement your own processing you can Switch the r and the b around to make the red part be the blue part of the blue will be the red part and destroy your image Why wouldn't you? Thanks everyone for watching Great to see again that there's always people showing up in the chat and asking questions and being involved in what we do Thank you very much for doing this. No worries. Thank you lots learned um And I'll see you all On next time around. Bye