 Cool, um, so yeah, my name is Nicholas. I want it today. I'm gonna try to Kind of show you everything that's related to WebGL for graphics and data visualization So first question is who here is familiar with WebGL? Nice awesome. Okay, so we're gonna go Through like a bunch of like use cases interesting use cases for WebGL in in specifically like data visualization then we're gonna go through a bit of like explaining how the WebGL works behind the scenes so rendering pipeline and stuff like that and then We'll go through an actual example And we'll have a bit of a look into the code So yeah before that I used to work at Twitter. I worked there for three years Leading the the public facing data visualization work recently move over to Uber where I'm leading a new data visualization team and We're hiring. So if you're interested, please Come find me. So, yeah, this is the agenda Hopefully you won't get lost around the third to fourth point Well, let's start with WebGL. What is WebGL? So WebGL is is just like a JavaScript API to app to access the GPU And the GPU is is pretty fast So this is one example I built when I was at Twitter Showing a number of retweets for the Obama to the former year's tweet. So this is just like showing 800,000 retweets and which you know reached like a millions of of viewers on Twitter And these sort of thing you can build with with WebGL pretty easily actually So in terms of the standard this was developed by the Chronos group Chronos group is the same organization that actually built the open GL wrote the open GL standard in the early 90s For this sort of like desktop really old computers and then it moved over to the open GL ES standard the embedded system standard The one dot one dot zero version was out in the early 2000s. This is why I'm using that fat like iPhone And then it evolved over to WebGL WebGL one dot zero is the actual same API then open GL ES 2.0 and WebGL done 2.0, which will be out probably this year May mix the open GL ES 3.0 API The biggest change between open GL and open GL ES at some point was the fact of adding a programmable pipeline I'm going to go through that afterwards, but it basically lets you code some actual GL SL code which is a specific domain specific language for graphics to do some neat stuff on the GPU So what can WebGL be used for so I have a bunch of examples Not all of them are data visualization related, but hopefully will cover everything So maybe you are already familiar with you know 2D canvas or SVG and D3 and you're like, okay Why would I be using WebGL for like? You know, I don't usually made maybe I don't usually visualize like a hundred thousand elements and maybe doesn't even make sense We should aggregate those But you know here I'll show a bunch of examples that probably are within your domain of work and and you know, hopefully you'll get inspired and want to learn WebGL So we're going to go through exploratory visualizations You have a lot of data how to kind of like filter and manipulate that data in a way that the rendering Is not the bottleneck of the processing. We're going to go through real-time data analysis and other another example in which You know using other web standards is definitely slower than using WebGL We're going to go through a bit of storytelling Scientific visualization and then data art or data illustration So this first example see if it loads Yeah, so this example shows loads all the cities that have an airport I think or all the main cities that have an airport and then it loads all the different airlines and you can Basically browse through each one of the airlines and see All the different routes So this data is actually a bunch of years old already But the interesting thing is that you know here you can kind of explore the centroid of the different routes And so you can infer where the airline is coming from like Air France centroid is France US Airways the US and Tam for example is in South America Qantas for example might be interesting in Australia and as you can see we can still you know add routes and you know We're definitely adding hundreds to thousands of routes And the rendering is like not complaining at all. So it's working pretty well You can also interact with the 3d shape is pretty smooth So in terms of like you know interaction or like being able to kind of like load a lot of data These are like primitives are you know lines and circles and so on It's WebGL the rendering is not the bottleneck of the application It's mostly all that is related to computations in JavaScript But not the actual rendering at least for these sort of examples On the exploratory visualization side we have other types of examples Maybe less flashy or but more like useful in a sense. So I built for example this example on Trying to visualize the mobility flow in France So we've been we've seen a bunch of maps that sort of like visualize mobility flow like people flowing from one state to the other of the US and we've seen remixes of these maps and You know in most cases like these maps use arrows And this is a visualization question actually I don't really know why they would use arrows if you can actually call out code everything the arrows actually are occluding You know other states and so on so what I decided is okay I'm going to build my own mobility map and I want to see what's the mobility like in France So mobility means you know that within the five years one person moved from one state in France to another And here I'm color coding so each state So this means that each state if it's more red means that more people are getting into that state If it's blue means that more people are actually exiting that state And so what you can do is you can hover here for example say Paris and Ask yourself well, where do people go from from Paris? And so you'll see that the suburbs are the first places people go to actually nobody goes to Paris From this chart, but everybody goes as elsewhere So this this might tell you something about mobility flow in Paris if you kind of hover the suburbs Then you'll see that people go or even further out and so on and so you can you can keep hovering these other states And you'll see the mobility patterns, but you know WebGL It's not you know the definitive tool to visualize these sort of things I won't even further and I try to zoom in and be able to visualize each one of the counties. So Counties is definitely like 40,000 or so on Counties so that this starts being like kind of like a pretty big data set And so this is the case where I feel that you know WebGL is definitely Maybe one step ahead in the sense that you can you know You can use it to kind of like browse through all these all these big data set And and you won't see any lag A similar example of like exploring a data set is this one in this one. I'm visualizing wind patterns So for each one of the of the stations recording this sort of like wind speed direction and temperature I took the all the data for the last 72 hours and I actually packed that into a binary array buffer Which is a binary data structure instead of just having a text as a JSON because it's pretty big and I loaded that And I built this sort of like tool in which you can You can explore across the last 72 hours how how weather evolved and it's interesting that suddenly you kind of like see this this outlier here Right and you can you can replay this On every hour and see how kind of like Mount Washington gets every time like bigger and bigger So in this case, it's it's kind of like the same situation you have a lot of data You're still aggregating the data. Well, that's gonna explode But you know, it's interesting for you to just like dump the data in and be able to explore it Be without any lag you can change the visual marks over to circles or lines if you just want to focus on the direction of the wind Another example I really like Which is also a pretty straightforward to build with WebGL is surface explorer So this is more for like people who like math, but I feel it's it's still interesting You can load any any parametric equation here and You can display it for example as a grid or just as a surface and you can change the time parameter Let's say we're gonna use two and then make it loop Let's say five and then we can change this You can actually change the equation and it gets recompiled Let's see times five Yeah, it makes kind of weird things So, you know, this is as a kind of like interactive way to just like see what happens with different equations and you know instead of instead of like trying to imagine and what all these like Long line of you know sinus and cosinus things mean you can just like directly visualize it and edit yourself these sort of equations another one I used I used what GL for was this temperature anomalies visualization that is Taking a lot to load a lot of time to load actually Yep So this one took takes NASA imagery of different changes in temperature across the years from 1880 to 2004 and so what blue color coding means is that the temperature went down from you know from that period from the 1880s to 1884 Temperature in those regions went down and red means that temperature went up and so this kind of like makes a good case for maybe Climate change people so this means you know between 1890s and 1894 temperature was going down But if you go over to 1990s you see that it starts going up pretty rapidly and over 2000s The earth is on fire So that's in for exploratory database So you've we've seen a bunch of examples most of them are using geo data But you could definitely like you know consider using you know Exploratory data visualizations of like could be anything a scatterplot and so on so actually now We're going to look at real-time data analysis and the data structure that we're going to be using is Something that takes abstract data and and not like you know scientific data like position and lat-longs So this one is mostly to showcase Kind of like the the real-time Analytics that one can build on this this does this have sound you have sound please Yes, no, maybe Anyway Yeah, it's plugged in it's a video it's playing some really good music really sad that Okay, no problem So what this is doing is actually taking every frame of this video this looping is a vine and It's decomposing the image into different colors So it's creating basically a color histogram of each one of the pixels So if you have you know an image that's fully red then you'll have this coral cube that shows it a lot of red on the bottom right there your GB cube is a Cubing which each one of the components in the 3d space is a different color right and you can change this sort of Representation right so if you use HSL or HSV you're changing the cube the RGB cubing to a different shape so Here you can change the video and see that different the compositions for For different images actually I used this straight trailer to kind of test it out from the beginning And you can change the the color scheme so if you change to the bicone for example You can see how color the composition happens in that specific Horrors team if you change it over to the to the one time you can see that So this is the sort of thing that would be really Taxing or expensive to do into the canvas because of the rendering bottleneck that you have of like trying to visualize all these dots all the same time And we're not just rendering dots. We're rendering actually 3D structures In it Other things like you know some of the vines actually make up like some interesting Sculptors like this one for example that has a bunch of like nested White dots we use like a transparent shape so that we don't include the other the other 3d shapes So this is for real-time data analysis You could imagine like try to do analysis on top of sound on top of like other types of images do image Processing and stuff like that and all that would happen in real time And you would not see any any legs if you kind of code it well Another option is storytelling and actually build this when I was at Twitter this went out on can lion Can So this is the mentions of Ian McKellen and Sir Patrick Stewart doing time on tweets And the yellow dots show actually mentions of the of the two people in the same tweet So this this visualization is trying to show that you know There's a growth of interactions between Sir Patrick Stewart and and Ian McKellen through time Those can be related to a specific Timing which like they actually traveled together to New York and went to this play together and other things can be related actually to you Know a bigger friendship They have or also can be related to the fact that they're actually promoted their last X-Men movie This is not particularly super insightful, but it's definitely a nice a nice way to tell a story It this was used by the execs at Twitter during can and this was shown to Sir Patrick Stewart He was on stage and he was like wow, this is amazing So it's it's definitely like a good way to to do some something that's eye-catching and keeps the attention over to the to the screen Another thing that's actually very useful But I don't use that much is scientific visualization so a lot of people use Some pretty interesting algorithms for example to visualize flow and you've probably seen the wind map or or earth and And and these these kind of like techniques of showing, you know, the flow of vector fields are called line integral convolutions and the way you implement this is If you can if you want to use the GPU that also doesn't have any lag if you if you implement it that way And so these very same technique can be used to do fluid simulations and And things like that in which you interact with the screen and then you simulate the the actual flow based on something called texture advection Which is moving a little bit the texture that that's behind it This is an image behind the cursor and we're just like moving a little bit the points of that texture And that's why it's called texture advection So you could use it for things that are actually more useful For example have this like magnetic deep hole and you can kind of like explore how you know Different charges interact with each other and how how an actual electric field looks like because you cannot see an electric field Unless you throw sand and then you'll see the actual lines and this is what this is showing and Finally, there is data art. So we built this for the screens in the Twitter office the main idea was to have some sort of a Nice, you know visual that would show us different vines based on the number of Revines or views each vine had or loops actually each vine had and then try to explore what were the best vines Along there and so we did this for hack week. This is a WebGL implementation I Implemented this sort of algorithm that it's a space-filling algorithm that shows It's a fractal space-filling algorithm So it covers a space with this sort of like shapes and you could use different primitives like, you know rectangles and circles and Hexagon and so on and you can be sure that it will always fill in the space without any overlap and then what we do we just like explore each one of these videos and And we keep them looping and then we This solved the actual kind of like structure and load a new structure afterwards And actually we did this for hack week And we didn't win, but we got an email from dick saying like hey, this is a great project. You should totally You know implement this in the screen. So we already have well, I don't work at Twitter anymore, but We had the space for that And some of the things so this is actual like interactive thing So you could you know zoom zoom out and you can see all these all these different objects playing the video This is something that definitely cannot be done with any other web standard other than what you know cool So hopefully you've seen and in any of all of these examples something that you might be interested in or you might be you know Knowing to know how it works. So we'll we'll just dive into like WebGL This is kind of like the more dense part, but if you get this you probably get like 80% of how WebGL works So WebGL JavaScript API right so main thing Let's say you have an array of floats an array of points XYZ coordinates You want to send that over to the GPU to be rendered in a kind of like a 3d scene? So this JavaScript API does this it allows you to store an array of floats Into a buffer, which is a space in memory that then can be accessed by the GPU and be rendered So that's kind of like the simplified version a bit more complicated is that the fact that we have a programmable pipeline Let's us be able to write some GLSL code to write a vertex shader and a fragment shader So what are a vector shader and fragment shader a vertex shader is? Imagine this as a callback in In a callback in which you receive this array of floats one by one and then you need to return One float one by one. It's a one-to-one mapping the vertex shader is usually usually used to transform points to rotate points Scale an object and so on Whereas the fragment shader you can imagine this as another callback that it gets called for each pixel in the in the screen And it decides which color the pixel in the screen is going to be Of course, this would be super costly if it wasn't done in parallel and this is why you can you can use a GPU to do this So besides writing your JavaScript API to send over the array of floats You need to write a file that does the one-to-one mapping between those points to some other points That are transformed scaled or rotated and then you need to write another callback that will get for each pixel We'll decide which color the pixel will be right Cool. So how does the pipeline work? And so this is behind the scenes some things are programmable some are not so the vertex shader We already covered that up you write this this is programmable. You decide where these four points you just Eight points you just send over are going to be Triangle assembly phase most of the time you don't want just to render points You want to render faces and so this part is not not programmable But the computer will figure out how to build faces out of the points And so you have a few options on how to pick up the vertices to make faces for example imagine that you have Four vertices that make up a square Now this square is made up of three triangles right because the minimal phase in a in a 3d thing is a triangle So how would you pick the points to build the triangles each triangle will have three points? So you will need six points, but you're sending over four so you need to send over like indices on like how these triangles are going to get drawn The other step is the restoration phase you have all these like 3d concept right, but this gets projected into the screen So this is what the restoration phase does it converts whatever 3d notion you were handling into pixels and Finally you have the fragment shader phase that callback that gets called for each one of those pixels And you get to decide which color the faces are going to be of your of your cube for example Hopefully you're still with me So let's talk a little bit more about the GLSL right so you're gonna have to write this fragment shader code and this vertex shader code What does GLSL looks like so it's called GLSL because it's GL shading language It's a DSL for graphics a domain specific language for graphics It actually looks pretty familiar it looks like see so if you've ever looked at see It kind of looks like GLSL The other good features it has is the fact that it has built-in types for graphics Right, so he has built-in types for vectors and matrices and so on it has a lot of built-in functions Not only math functions, but functions for graphics you need like reflect and refract And finally the feature I love the most And I love JavaScript to have this but it doesn't is operator overloading you can do operations between different types and You know it will figure it out and it will work as you would expect so you can add two vectors and use a plus sign Or add two matrices and use the same plus sign and it will figure that out This is how a bit of code in GLSL looks like this code actually doesn't do anything. It's just like a demonstration So in the first line we're building a vector out of this Constructor so you can use VEC 4 as a constructor you pass in two arguments and you can compose constructors So you can pass in the other two values as a VEC 2 for example Then what you can do is you can refer to vector dot xyz to get a copy of a three component vector And you assign that over to a point Now here you could use any sort of like sub indexing so you could do vector dot yy yy And that will return you a VEC 3 with the three components being the y component vector 4 or you could you know write like vector Z yx or W Z yx and so on and that will return a new vector Now you can refer to vectors Indices like xyz or you can refer them to RGB xyz W or GBA Why is this because the vertex shader changed the position of the points But the fragment shader changed the color of the pixels in theory so It's nice to be able to refer to the same components in two ways Finally we're creating a matrix and then we're multiplying a vector by a matrix and we'll get the desired value now the second block of code It's interesting as well because it shows the operator of loading thing, right? So we define two VEC 4s one vector that is An empty vector and the other one is a vector 1 and we want to interpolate between between these values Let's say we're kind of like animating something and so the delta value will go from zero to one And we want to kind of like find the the mid-range value in between those two And so we can use plus signs and minus signs and multiplication signs and and all this will work just fine because of Operator overloading the plus sign is defined for VEC 4 is defined for mat 4 for VEC 3 and so on So we can always use the same signs for everything Otherwise if we were in Java or in JavaScript, we would have to do vector dot add dot sub dot molt Mat 4 and so on Which is horrible As far as the built-in functions go there's a lot of like math functions So that is all covered and then there's a bunch of like graphic free Specific functions like refract and reflect that you can use Okay Let's go over through an example This is going to be even more complicated But it will be fun. So This example, it's called the hot vibration. It's this sort of like mathematical thing that I Ran out into like a few weeks ago, and I really liked it and I wanted to implement this So how vibration is a way to kind of explore four-dimensional shapes So how do you explore four-dimensional shapes? Well, you try to project them into 3d and try to see the different shapes Just like you do in a in a globe versus a map, right? You have all these different map projections because you want to explore all the different properties of this globe So this guy hop, let's let's try to kind of understand what he did So first step will be try to understand what he did second step will be trying to understand how to implement this in an example So hopf map is a function One too many function it maps one point in the sphere in a three-dimensional sphere Over to a circle in a four-dimensional sphere What this means is? What does it this mean? It's a nice way to explore a four-dimensional shape because you can you can imagine a sphere You can imagine a point in a sphere and this will generate something different in a four-dimensional shape now The problem is I'm showing you a circle in a three-dimensional space Not in the four-dimensional space, right? So that is kind of like a trick So I need a second step a step to project the circle from four-dimensional space to three-dimensional space And we do a projection a map projection. So now question for you What map projection will preserve circles from a higher-order sphere over to a plane for example? The stereographic projection of course you all knew this, right? So if you have a circle in the sphere and you project that over to a 2d plane Using the stereographic projection You'll get a circle in the 2d plane Why is this important because if we have this four-dimensional circle and we use a Stereographic projection on the four-dimensional circle will get a three-dimensional circle, right? And so it's nice because it means that kind of like keeps the shape It doesn't kind of like stretch or do anything weird with the shape Whereas like if you use this on on planet Earth You see this kind of like really distorted map, which is not nice to see though It preserves the circles in the equator and all the other Kind of circles you you might have So let's see a demo now that we all understand what this does So now we have a pointed a sphere. Do you see it on bottom left that maps over to a circle, right? I'm gonna replay this So point is fear orbiting around the sphere, right? Maps to a different circle Let's try to explore this shape If we add a bunch of points in the in the equator We'll see this new shape on fault, which is some sort of torus this interesting torus can be Flipped actually upside down. So if we rotate these points will kind of like rotate upside down the torus interesting shape, right Now let's try to explore it more and we'll add a bunch more points to the sphere going up and down the sphere and see how this Kind of like four-dimensional shape behaves So it's kind of like, you know some sort of like nested Toruses that can be upside down Easy finally we open this up a little bit and We'll see that we can actually interact with the shape and kind of like explore this a little bit more on our side I'll go through this in a sec. Cool Very useful thing to do, right? Very practical Let's cover how the hop map work like forget about the example. Let's think about the code So I need to write these two functions in GLSL one function will map one point in a sphere Over to a circle that has four coordinates. So that's the first function the hop function The bottom one is a serographic projection. This one will take one point in 4d space and projected into 3d space So if we kind of combine these two functions together, we'll get one point in the sphere and a three-dimensional circle Right, we have One point in a sphere and a four-dimensional circle and then we have another function that maps a four-dimensional circle into a three-dimensional circle So this is what we're seeing right now one point in the sphere and a three-dimensional circle So let's think about the top function So let's think about the signature of the function. So the function would return a four-dimensional point But we'll get a latitude and longitude in the sphere right those two last arguments there on the top right So given a latitude and longitude we'll get a four-dimensional point But we should get a set of four-dimensional points the points that make up the circle the four-dimensional circle So for these we kind of I have this extra parameter called eta which will go from 0 to 2 pi That will give us come kind of like sample points of these points in the circle, right? So if we get let's say it that will go 0.5 and 2 pi then we'll get three points that are part of this four-dimensional circle and Then the second function you don't really need to think much about it It's just like it takes a four-dimensional vector and returns a three-dimensional vector So we will take a four-dimensional point and show you that point in the in the screen in three-dimensional space Cool, let's think about the data model. We have this function in the vertex shader, right? It's a one-to-one mapping, right? So on the data model ideally in JavaScript We would have this array that has a bunch of lat-longs, right? And these lat-longs will get converted into a set of points in a set of circles Now the problem is as I said before The vertex shader will only have a one-to-one mapping So if you send if you send one lat-long to the vertex shader you will get one point But we want a set of points. We want the points that make up the circle So in order to do this we need to send over extra data We need to send a lat along in a third argument, which is kind of like the parametrized circle shape Now this is a lot of data like an a lot of like repeating values and so on So hopefully we have this thing called instanced arrays, which is an extension to WebGL That let us load the different data separately and in this case it will make the kind of the cross of The 0 to 2 pi column and the column of lat-long in kind of the GPU So the first line What it does is it takes the hop Program the one that refers to vertex shader that I mentioned in the previous slide And it sets a buffer which is an array of floats from 0 to 2 pi Which is the gradient rarity or the sample points we want to have for this circle the second line Sets just the lat-long points But it's calling this property called instanced which means okay We're gonna have to do the cross product between the 0 to 2 pi over to these lat-long points But ideally kind of like if you need to to remember something out of this is that in JavaScript You're setting the buffers with the proper points You want to visualize and in the vertex shader you do the conversion over to the actual screen points you want to see So good this is kind of like how it looks like for the first time It looks like a bunch of points right because this is what we got back We send over a bunch of points. We got over a bunch of points transformed into a circle If we use gl line loop or gl lines, we'll see a bunch of lines But if you think about the example again Well, it looks a bit better, right? We have a bunch of tubes and we have about just lighting and so on So we're not actually rendering lines. We're rendering tubes So we need we need to find a way of rendering tubes instead of rendering points or lines So let's think about this We're sending an array from 0 to 2 pi That array gets converted into a circle We would just sent a canonical Tube from 0 to 2 pi We could topologically Make it a tube right that that kind of makes up a circle Right, so instead of sending an array that has from 0 to 2 pi and Having back a circle we'll just send over points that make up 3d tube And then we'll we'll deform or stretch the tube to be the sort of circle in 3d I won't go over the code of this, but it's doable The third thing we have the interactions and this is really interesting because it's it's kind of like the main Pain point on the webgl applications you might have is like, okay We have all these things rendering really well But I can't really interact with it or I can just like move the camera, but I cannot actually click on anything and so Here we provide a bunch of interactions Actually if you hover the sphere here you can kind of explore the the shape and so you can you know Move around and see how that actual point in the sphere would map with with the actual circle You can click here and kind of like store those if we clear the points like you can see how it wow this place here Makes a weird shape, right? So, you know, we discovered this new spot So in order to interact with it What we need to do is We are hovering a sphere Right, but we don't know anything about the 3d shape of the sphere canvas is Webgl is just like 2d canvas in the fact that it's rendering over to a bunch of pixels in the screen It's like hovering an image like you might get the pixel and the color of the pixel But you don't get the actual shape or the object So how this works is we render Into an offline image an image you won't see imagine you render something over on a canvas But you don't append the canvas to the DOM. So this is how it works We render over a sphere that has this really special texture This texture is One different color per pixel So when you hover the sphere and we get the pixel from that sphere that you're hovering That color will be unique and we'll map map uniquely to a position in the sphere It's easy to map from a color to to a position right because rgb goes from zero to one It has three coordinates in this case We have a sphere that only has two coordinates latitude and longitude So we could only use the red and green channels To encode this in this case. We're using the red and blue channels And so when you hover the sphere You get the pixel out You get the color of that pixel and then you can map that color over back to a lat long range And then you send over that point and this is what happening Uh in the example And this is called color picking. So usually What you're doing webGL is that if you have let's say three Three spheres and you want to be able to hover these spheres and know which sphere you're hovering You render offline In a frame buffer Each one of the spheres in a different color and then when you hover the sphere you get back the pixel And you know, which which pixel maps to each sphere, right? Cool key takeaways You've seen a bunch of code. You've seen a bunch of use cases Uh good things about webGL and bad things about webGL Good things is that it uses a GPU This means that you can render hundreds of thousands or millions of objects like and not have any any bottleneck issues You can render 3d stuff if you want to you can render 2d stuff as well As we've seen with the data art examples You can do a lot of things with webGL not only 3d games You can use it for for visualization. It can be scientific visualization. It can be exploratory database You can do data art as well but Related to this and the example as you can see the APA is pretty much low level So I would definitely recommend you if you're starting To use 3js or there are other libraries like stack GL for example They use npm modules to kind of like build your your own WebGL framework. I build a library called phyloGL Which is uh, it's a pretty simple library It's it's nice to start with if you want to kind of like grasp the concepts of webGL But yeah, definitely use a library and with this Thank you