 Hello, OpenViscount. Do you enjoy the conference so far? Hands up. Great. Okay, so let's not change that then. So I'm Dominicus, and my topic today, as Irene said, was what to do about the various graphics technologies that we have on the web, and the title is waiting performance against pain. And the problem, as you can see in this very scientific graph here, is that pain and performance are unfortunately correlated. So the more performance you want to squeeze out of the browser, the heavier it gets on you as a developer, hopefully not on your users. And as you can see, we have three main technologies, so SVG, Canvas, and WebGL, to work on that. So WebGL is the latest one, and I'm really excited by that. And as you can see, the closer you get to the graphics card, this GPU on the very end there, the harder it gets to actually get anything running. And we've already heard some talks about WebGL, so you'll notice that sometimes it gets really complicated. Anyway, I will also briefly talk about the various little things that can go wrong even in your JavaScript independent of the actual graphics API that you're using. Because when it comes to performance, it's kind of like driving a car at very high speeds. So even if you make just a little mistake, once that mistake is repeated a thousand times a second, things can go really wrong pretty quickly, and stuff can blow out of proportion. OK, so you might be wondering why this in-depth look at browser performance. I have to show the slide, because it's a visualization talk. So, Charles Mina, who did that like 300 years ago, he didn't have a computer, right? And he was still able to produce beautiful graphics. So that's cool. And I myself am also a big fan of paper. So paper is really an amazing medium. And if you want to, you can go all tufty and just skip computers all together, and then you might end up building stone sculptures at some point. That's one of the side effects. Anyway, also, you don't really need to know too much about performance when you're only building simple things like bar charts or so. Because usually you have so little elements there that you can't really mess up in a really bad way. But what if you want to push the boundaries? What if you want to do really interesting stuff with web-based visualizations and especially interactive ones? Then you actually need to know what's going on. And this is kind of the topic of the talk. So let me tell you how I slid down that slippery slope to performance optimization. So my first chance to learn about this topic came when working with Moritz Stefano on the Better Life Index. You might know the Better Life Index. It's been around for a couple of years now, and Moritz's first version was in Flash. And Flash is really amazing because you could, or was really amazing, unfortunately, because you could do really, really fast and responsive graphics without thinking too much about them. It handled it for you. But, of course, Flash was proprietary, and it aids through batteries like a Tesla. So, yeah, that was the thing. That's why Steve Jobs killed it, I guess. That was probably the only reason. Anyway, so here's the Better Life Index as it's currently online. So you can find it on ocdbetterlifeindex.org. And the idea is you have one flower per country, and you have leaves per flower. And these leaves depict certain attributes of this country, like the housing situation, the income situation, jobs, and so on. So it's about the quality of life. And the cool thing about this visualization is that you can adjust it to your own like. So if you're really into education, for example, you can drag this little slider here, and then the flowers adjust. And the visualization tells you where you're supposed to move. So Finland looks really good. When you're interested in the overall quality of life, I think Australia has won three or four years in a row now. So it might be a good idea to move there if you're interested in that. Anyway, so this visualization here, this version is based on canvas. And what you can see is that we have 40 canvas elements floating around the screen, and they're repainted, so every flower is one canvas element. And as you can imagine, they've made the code really, really complicated in parts. And I also tried to kind of squeeze out all of the performance that I could. So I've only recently been working through this old code, and there are all these little conditionals that happen in the very weirdest cases. So the code was really kind of weird, to be honest. So I'm very happy to show you a sneak peek of the latest version that we're currently working on, the Better Life Index 2015. And no, can't reach it. Okay, that's better. All right, so this is the new version. And again, it looks pretty similar. That's kind of the requirement every time that it still looks the same. But what happens when you drag one of the sliders? Then you get this much, much smoother animation. It's really beautiful. It's as smooth as Ryan Gosling. It's wonderful. Anyway, okay, so how did we do that? So that's no longer canvas. And it also made the code, the overall thing, much easier to work with because we only have a single WebGL element now. I could get rid of all the various minor performance optimizations that I did because they no longer actually do anything in WebGL. And it makes it much more straightforward, which was really cool. And as you can see, it's still pretty fast. So here we have, what, 60 frames per second on this MacBook. So that's cool. So what is it about WebGL? I was here two years ago, as Irene mentioned, and I was talking about kittens also. But back then, I set this here. So we have things like WebGL, for example, and hardware acceleration in the browser. And once that moves to tablets, then I guess we're all set. And our beautiful D3 visualization should work again. Okay, so that was the dream. Actually, Apple caved last year and moved WebGL to iOS and also Safari on the desktop. And that means that we can now actually use it and work with this technology. So, and that's what we're doing in the Better Life Index. I will tell you more about WebGL and how to best use it as a developer without getting too much into the pain territory later. One other, so yeah, that's coming up in May 2015. One other project where performance optimization was really important was on Broadway. We did that, so Moritz and myself did that together with Lev Manovich and Daniel Gaudemeyer. And the idea behind on Broadway was to create a picture of a city. And people have done that before. So one of our inspirations was at Russia's Sunset Strip from 1966. And to pull that off, Russia actually put his camera on a pickup truck, I think, and then drove along this whole street and took a picture of every building and then built this beautiful, more or less endless photo. Which was very nice and a heavy inspiration for this project. But of course nowadays we have much more data sources to represent a city. So we picked the Broadway, the most iconic street in New York. And we tried to find as much data as we could to put in there. So we have things like Google Street View. We have taxi statistics. We have also the sky view of Google Street View. And then of course we have all the social media stuff. So Twitter, Foursquare, and Instagram. And then we also have Instagram photos and we have color palettes for these photos. So a lot of stuff there. And as you can imagine, it's going to take a while to actually load. Let's see. Nope, works. Okay, as you can imagine, that's a lot of data there in the browser. So here we have a couple of canvas elements again. So basically each of these layers is one canvas element. And if you click anywhere from this broad overview perspective, the browser zooms in and you can explore the street level more or less and look at the various places in your browser. And you can also hit the reset button. And it also works seamlessly. So you can zoom in a bit or a bit out. And there's a lot of data as you can see. So again, I had to try to pull as many tricks as I could. So one thing is that if you zoom in, the browser has to re-render these 700 or so images so each of these slices. And to make that faster, initially only the canvas elements are scaled. So usually you can see that during the animation. You can try to look for it. So initially it's not redrawn, but it's just scaled. And then only once it gets to a saner part of the numbers, it actually starts redrawing them. Okay, so that was on Broadway. You can also find it online and try it out for yourself. Maybe not now because it downloads 30 megabytes of images before actually doing anything. So please be, go easy on that. Okay, all right, let's talk a little bit about the performance basics. What are the things that you kind of have to know when you start out with this whole performance thingy? And actually my first recommendation would be to know when to stop because performance optimization is, yeah, it's kind of a fickle mistress. So sometimes you, it feels to me like a puzzle sometimes. So you can, you have some problem that you want to solve and you attack it with your dev tools and you have the feeling that you can improve something but sometimes there just is no improvement. There's just too much data and then you maybe have to look at a different way to get it faster. So yeah, nowhere to stop but if you don't want to stop yet, what's really important is to know your tools. So if you haven't looked at the various things that are possible with Chrome's or Safari's dev tools or Firebug in Firefox, learn that by heart because there's so much great stuff in there that actually lets you zoom in on the single line of JavaScript code that messes up your whole performance. So that's really powerful and there's, there are lots and lots of resources on the web to learn about this stuff. Okay, but what's the simplest way to improve your performance, right? Removing things. So leave out all the unnecessary stuff, all the console locks and so on. But also think about which elements you actually need, which CSS patterns, for example, you want to use because like some CSS things like gradients, for example, actually cause a lot of performance even though it's CSS, right? You can use it as much as you want, but unfortunately that's not always true. One more concrete example about this leaving out things. One thing that I often end up writing is something like this here. So I tie a couple of functions to the resize event handler of the browser. So once the window gets resized, this calculates how much space it has available and then re-renders, which kind of makes sense. But the problem is that the browser is really, really good at throwing these resize events. So you might end up having these calls 10 times a second and usually that rendering a visualization takes a little bit longer than that. So one way to solve that is relying on the concept of a throttle, for example. So this here is the underscore version and throttle lets you throttle a function call to a certain time frame. So with this command here, for example, this calculate dimensions and render is only called every half second at most, which is usually what you want and it improves your performance noticeably in case that anyone actually re-scales their browser window someone just told me that developers are the only people who are ever doing that anyway. But yeah, just in case. You never know. Anyway, the other thing, if you can't get rid of stuff, then try to shift it somewhere else. So if you have really heavy calculations, for example, it's probably a good idea to move it to something like a web worker. We've heard about this a couple of times now. So JavaScript is single threaded and every calculation that you do actually blocks your GUI and your app becomes unresponsive. But you can use web workers to do these heavy calculations in the background. We're doing that, for example, in the Better Life Index, where we pre-calculate all the animations in a web worker before actually playing them back. And that, I think the pre-calculations run for a quarter or a half second and nobody really noticed it, but it makes for a much, much higher performance. Okay, but apart from these things, once you reach a certain point in performance, the question really becomes, are you smarter than Chrome's Firefox's or Safari's dev team? I will assume that you might be smarter than the dev team of some other major browser. So the thing is that, once you start to go deep into performance, you actually have to do the stuff that the browser does. That's really the key to optimizing performance, taking over the browser's job. And if you ever had a look at any HTML or CSS specification, there's so much stuff in there, and usually you can throw any of the billion websites at a browser and it kind of displays what it's supposed to do. So that is extremely impressive. So that's impressive tech. And I guess browser engineers are some of the smartest people on the planet, I assume. So you really want to think twice if you actually want to do all of that stuff. So, yeah, you kind of take over responsibilities, and you can see that, for example, SVG is pretty close to the browser. It handles more or less everything for you. And the further you move towards the GPU, the more stuff you actually have to do, and the browser no longer does for you. And that makes it difficult. That's also the reason why, for example, CSS animations are usually much faster than JavaScript animations, because the way a CSS animation works is you tell the browser, I want to move element X from A to B, and then the browser integrates that into its rendering processes, and everything's fine. And if you do that with JavaScript, it's like this black box that the browser doesn't really know what's happening. It's very dynamic, so everything could happen. And basically what it looks to for the browser, if you have a JS animation, is the browser move element X a little bit towards B. And then in the next frame, move element X a little bit towards B, and so on. And the browser really knows when this, never really knows when this animation is going to stop, and so on. So that's why CSS animations are usually faster. And of course, then the question becomes why don't we move everything to CSS while having some dynamic parts that still makes for more interesting websites, I would say. So that's not really a solution. Anyway, so again, the core to performance optimization is taking over the browser's job, and of course that also means that you actually have to know what you're doing, think of power and responsibility and stuff. So Chrome source code is actually 60 million lines of code, and you probably don't want to rewrite all of that. So, yeah. You have to know where to start. So what's the problem with the browser? Why is stuff that's close to the browser a little bit slower than the other ones? And the reason for that is the DOM. So you've probably heard about the DOM when you did anything with the web. So the DOM is the internal structure in which a website is organized in the browser, and it also gives you the API to change pretty much anything. And the problem with letting you change anything is that you can easily mess up the browser's plans. And what usually happens then, if you look at this hierarchical tree is that the browser has to recalculate things. And it does that in two ways. There's, for one, the so-called reflows, which are basically the layout of the whole page, so where elements are and their sizes and so on. And then once this re-layout has been done, there's the repaint, which means that the whole website is re-rendered and redrawn. So usually when you load up a website, you have both these things, so first a reflow then a repaint, but you can easily trigger these things when using certain JavaScript commands that maybe aren't the best idea. Okay, so it's probably very obvious that if you change, for example, an element's width or height that the browser has to do a reflow or a repaint, but it can, you can trigger reflows in a much, much easier way. So here's an example. So if you're only asking for the width and the height of an object, that already triggers two reflows. So you didn't really change anything about the website, but the reason for that is that the browser wants to give you the latest values, so it does a reflow immediately before that, and it does that for every single call. So that can lead to a lot of problems, as you can imagine. So it's really important to know which parts or which calls would actually trigger something like that. Yeah, so one way to make it easier for the browser is to detach certain parts of the DOM. So you can remove nodes, whole node structures from the DOM, then do your things to them, and then reattach them, and that basically causes two reflows and two repaints, one for the detach and one for the append, but it's in general much, much faster than doing that on the fly. Okay, so the DOM, really interesting topic, but let's talk about dynamic graphics. Okay, so initially I wanted to say that there were three main ways to create graphics, that's what I said, but there's actually a fourth way as well, super hidden way, so you use diffs, and you fill them red, and you place them on the screen, bam, bar chart, so that kind of restricts you in your expressiveness of your visualizations. You can only draw rectangles, but yeah, so let's skip that here. Let's go to more interesting technologies. First one is SVG, which stands for Scalable Vector Graphics, and SVG really is amazing, because it lets you draw pretty much anything, and you have all the benefits of the web, so you can use CSS, for example, to style your geometric elements. It also displays text really nicely, at least single line text, and if you check out Mike Bostock's D3 examples on the web, for example, they're all written in SVG, and for good reason, because they run smoothly and beautifully. So, yeah, SVG, right? But yeah, I have to say that SVG actually is my favorite of the three technologies, but it's also the slowest, unfortunately. So as you can see here, we're pretty much in browser land when it comes to SVG, and you can't really do much wrong with that. So what are the benefits? Of course, as the name implies, SVG is vector-based, in contrast to the pixel-based canvas, and WebGL APIs, which means that it looks the same on pretty much every device in resolution, and it even supports things like ready-nudder displays, for example, out of the box, which is really cool. You don't have to do much about that, and it just works. The next big benefit is that, as I said, it supports CSS, so you can cleanly separate your code and your presentation and your content, which always makes it much easier to change anything. And then we also have the DOM integration, which is really useful because in a way, this DOM integration works like a scene graph. Scene graph is a concept from computer graphics. That means that we have this hierarchical tree of elements, and once we manipulate something further up in the tree structure, the other elements get dragged along. So usually you hear the example of the car, which has tires and so on, but cars are boring, so let's talk about dinosaurs. So dinosaurs have teeth and arms and legs and the tail and so on. And if you want to move them around, it would be a real pain to take every little part of that dinosaur and move it. So instead, you just take the overarching node, maybe a group or so, and move that. So that's one line of code. It makes it much easier. And you also get that for free in SVG thanks to the DOM integration. And the other thing that you get for free with the DOM integration is support for interaction. So you can actually tie event handlers to your elements. You can tell the browser to call a certain callback once you click on an element, which is really helpful, and which is something that you don't get with the other APIs, unfortunately. So there you have to do everything yourself. Okay, and to give you an idea of how well SVG and the other technologies work, I wrote a little benchmark thingy here which loads really big data because we need that node, it's not that big, but still it's 120,000 stars that are somewhere around Earth and it renders these stars to the screen. So let's try SVG here. So it statically renders these stars to SVG. And as you can see, cool stars. But you might have noticed that it actually takes a long time to render 120,000 elements in SVG. So that's one and a half seconds, which isn't really the right thing for interactive web applications, I would say. So you can, of course, reduce the number of elements. So let's throw it down to 10,000 maybe. And then it takes 100 milliseconds, which is acceptable, I would say. But it gets really bad once you go into animations. So I don't want to crash my browser. So that's why I'm reducing that. Okay, let's use 1,000 stars and animate them. And as you can see in the upper right, we have like 55 frames. That's totally fine. But once you ramp that up to 4,000 maybe, okay, we go down to 10-ish frames, which is no longer pretty and even more than that will probably crash the whole thing, so I won't do it. Anyway, okay. So as you can see, SVG really nice to work with, but it has clear limitations. So static SVG, I would say, roughly 10,000 elements at most and animated SVG maybe less than 1,000 even. What's cool is SVG is also really neatly supported on mobile, so you can maybe divide this number by half and still get an acceptable performance on mobile devices, which is very powerful. Okay, but what if you want to go beyond that? That's what these other two technologies are for, Canvas and WebGL. And as you can see, we're slowly moving towards this pain space, which is called the GPU. So with Canvas, we actually have a clearly defined API and we can work with that and it's not too bad. But the thing about Canvas and WebGL is that they're both in so-called immediate mode. So we've heard about declarative languages yesterday and what SVG basically is a declarative language because it lets you declare what kind of objects you want and the browser handles all the rest for you, just the rendering and so on. But with the immediate mode in Canvas and WebGL, you no longer have that. So you actually have to keep track of your own objects. You have to know what colors they have, where they are on screen and what to do with them. And that also means that you have to do the rendering. So you have to know what to draw and when to draw it. And you basically have to do everything yourself, which is kind of a pain. And JavaScript gives you a couple of options to do that. So one great idea from game development is the so-called game loop or render loop, which is basically just a couple of functions that are called infinitely often, so as often as possible. And what a game loop usually does is it processes input, it updates the game state and then it renders the thing. And we can easily adopt that for visualizations. We're processing input. We're updating the visualization and then we're rendering it. And to do that in a browser, you probably want to use request animation frame, which is a function that calls the callback function in this example once the browser has the time and the capability to do so. So we can write our own little this loop that does exactly these things. So checking input, adjusting the visualization accordingly and rendering it, and then it's asking the browser to call this function again as soon as possible. Yeah, so that's an idea. But again, you have to do everything yourself. And especially the input part is really hard when it comes to Canvas because this is what Canvas looks to the browser. So as you can see, we have this neat 3D view here of the whole website. And there are elements that are stuck out a bit. And those are the things that you can actually tie event handlers to. But this SVG, or this WebGL element in the middle with the flowers, it's totally flat. So to the browser, Canvas looks like an image. And all the interaction, you have to manage yourself. So usually what I end up doing is I'm tying an event handler to the full Canvas element. And then I project the mouse coordinates into the space and try to find out where the user clicked on. Another way is color picking, which we've seen yesterday. So you can encode your elements in colors, invisible colors, magic. And then find out where the user clicked on. But of course, Canvas must be good for something, right? So we have a performance boost. And we can see that in our little star example here. So same idea. Let's start again with 1,000 stars. And that renders in 34 milliseconds. So it's a bit faster than SVG, but not much. But the cool thing is with Canvas, you can even render the full data set and it only takes 300 milliseconds. So roughly a fifth of the SVG time. And it looks pretty much the same. And what's also cool about Canvas is that you can do animations much more efficiently. So let's try this here. So as you can see, we have almost 10,000 stars and it's still 50 frames. And I don't know how far I can go. Let's try 20,000, 17,000. That's good. And we still have 23 frames per second, which is nice. And maybe you can read that. So it says Canvas sprite here. The sprite concept means that we have little graphical objects that the browser takes from memory and then just draws at a certain screen position. So I have little black dots here as images. But these sprites also let me change it to any other type of image. So for example, here's the super secret avocado mode at the bottom that changes the sprites to little avocados. You can tell by the greenish color. So useful. Anyway. So Canvas is a little bit faster than SVG. Not too much though, unfortunately. Which leaves us with WebGL. So we've heard a lot about WebGL in the last days. And we're actually using WebGL in the latest version of the Better Life Index, as I mentioned. And the idea of WebGL is basically going to the very, very basics of computer graphics. So basically what a graphics card does is it's this heavily parallelized array of triangles. And that's basically all you have at your disposal. So all the objects that you want to create are triangles at its core. And you can texture them, which is really nice. So a texture is a little image that you can put on one of these triangles or a square is probably the better idea. And then the browser displays that for you. And you can really, with WebGL, work at this level of detail. So you can adjust the actual triangles that the graphics card is working on. And you can see it here, of course, but in the latest version of the Better Life Index, these leaves for the flowers are each little squares. So two triangles each that are textured. And you can quickly change the texture to get these animation effects. And you can also easily rotate and scale them, of course. So that's really nice when it comes to performance. Unfortunately, WebGL is probably the furthest you can get away from all the non-pane parts. So here we have 175 lines of JavaScript code. Don't worry if you can't read them. I just want to make a point here. The point is, what do you think this amount of code could give us? That's a good one. Yeah, no, it's a bit more boring than that. So it actually, it doesn't only give us a triangle. It also gives us a square. But still, you can see that there's, well, yeah, it's a real pain to work with raw WebGL. Another thing we've heard about yesterday was where GL, the GL shading language, and that lets you work on the actual vertices and the actual fragments, the actual pixels that the graphics card works with. So here, for example, we're changing, I think we're putting some vertex at the right position given the current camera. So this is basically what happens on your graphics card. And you can use GLSL to write your own little shaders and do interesting things with that. And as you can see, they're not JavaScript, but they're based on C. So you have things like data types, which is neat. And you can really do the craziest things with them. But of course, it's quite low level and quite complicated. But I would say that the complexity of WebGL is also one of its benefits because everything complex is quickly reduced to something simpler through libraries in JavaScript. So we have things like 3JS or PixiJS that make it much easier to actually work with that. So we move a little bit back on the paint scale here. So 3 is really cool and totally recommended. So 3 is like the D3 of WebGL. It's heavily supported and it has a great documentation. And as you can see, lots and lots of examples for that. And it still gives you the full flexibility of WebGL without all the nonsense that you still have to do. So I don't know how many lines of JavaScript you actually need to display a blank screen in WebGL but I think it's 20-ish or so. So 3 takes that all over and does it for you. And you have still full access to the whole 3D stuff. You can also use shaders if you like. And they also have really impressive demos on the website. So I really love this one here, which is just beautiful. I don't know what it visualizes, probably not much, but just look at the water and the reflection. It's so great. Okay, I could play around with these examples all day. Anyway, so this is basically just a shader. It's two shaders that create this effect. And you can also have that in your browser if you're brave. Okay, so performance, getting back to performance. Okay, so what does that give us? So let's go back to our 10,000-dish stars maybe. Still much more than with SVG. And maybe use 3.js with sprites and look at the animation, which really isn't that great. But once we're actually going in deep and are using shaders, it gets much smoother. So now we have 60 frames per second with shaders and 10,000 stars. And the great thing about WebGL is that we can go even higher. So now it's the full 120,000 stars, and it still runs at close to 60 frames per second. So that's really amazing. But of course, that took me a while to actually get the shaders running to do that. It wasn't that easy. So, again, you have to know what you're doing if it's actually worth it. But then you can squeeze out a lot of performance for that. Okay, but I hear you. I mean, all this stuff was very 3D. And as visualization people were always wondering, really, 3D? Isn't that a terrible idea? And I have good news, because there aren't only 3D WebGL libraries. There's also 2D ones. And my favorite one is Pixi.js. And Pixi.js is interesting because it's actually a game engine. So it's meant for rendering games. But, I mean, if we're honest, visualizations really are kind of like games, extremely boring games, but still games. You can only hover somewhere in the box up here, so that sucks. Anyway, so you can easily use Pixi to do visualizations. And these examples here on their website are, of course, geared towards games. But there's really a lot of stuff that's totally useful, like sprite tinting, for example. And what I like best about them is that they have this benchmark, which is called the bunny mark. Yeah, where you can click anywhere to create bunnies. More bunnies. But it's actually a 15,000 bunnies now, and it's still 60 frames per second, so it's a really cool engine. And if we look at our boring stars, which aren't bunnies. Sorry. I also have a Pixi version of this whole thing. So this is Pixi in its graphics mode, which means that you can draw your regular SVG type of geometries, for example, so little circles and so on, which is kind of slow, but you can also use this sprite idea of having little images that are placed anywhere on the screen. Let's switch to avocados for that. And that runs up to 60 frames a second, and if we're going for the full dataset, we still get, what, 25-ish frames per second. And Pixi is much, much easier to work with than 3JS, for example. It even replicates some of the functionality of SVG, for example, so you can have event handlers that trigger in case you click on anything, which is totally useful. So yeah, highly recommended. Anyway, one last thing that I want to mention when it comes to graphics on the web is hybrid approaches. The cool thing is that you can combine pretty much any of these technologies with the other one, as long as stuff is transparent and you are not drawing over anything. So you've probably seen this one here, I think twice at this conference already. Anyway, it's a great graphic. And this yield curve here is, of course, drawn in WebGL, but what you might not have noticed is that the labels down here are divs that are manipulated via JavaScript, so it looks really, really seamless. But of course, the text display is much better in divs. That's why I guess they used divs for that. And it works just beautifully. So you can combine any of these things. There are globes based on canvas with SVG overlays on the web and stuff like that. So that is really powerful. Okay, so that was pretty much my overview of these graphics technologies. As I said, the more performance you want to have, the closer to the right side of this graph, you actually have to go. And I think most problems actually appear on the very left here with these minor things that sometimes just break and aren't optimal. And yeah, I think all of these three technologies totally have the right to exist. They're good for various cases. Mostly depends on the amount of graphics that you actually want to display. And yeah, you kind of have to know what you're doing. Thank you.