 Hello. Thanks, Irene. My name's Noah Veltman. Do data visualization at Netflix when I'm not drawing title cards from 1990s sitcoms on New York Times graphics. Thanks, Amanda. And today I want to talk about shapes. And specifically a sort of humble little shape animation puzzle that I ran into and the kind of journey of discovery that it led me on. And that journey starts here. So Amelia mentioned North Carolina when she was talking about gerrymandering in districts. North Carolina is in the midst of a lot of litigation around their court districts. And last year there was a new proposed plan. You can see the proposed new map on the right versus the current districts on the left. This is from something by a great journalist in North Carolina named Tyler Dukes to let residents look up their address and see whether their district had changed. And I saw this as a non-North Carolinian, very much not the target audience. And I was interested in the sort of overall effects. I didn't have a specific location I was interested in. I was finding it very hard to keep all this information in my head at once. Looking at both these maps, 30 or so shapes toggling back and forth, I couldn't really get it. And I thought this might be a fun time to use time as a third dimension and animate the transition between these two states. I am a big believer in a little bit of motion as a great narrative device. So let's say you paid close attention to Rob's talk. You're now a master of geospatial data and map projections. You log on to D3 Express. You just heard Kai's talk, so you're now ready to render your stuff in Canvas or SVG. And you get your data, and you plot the old coordinates, and you get the new coordinates, and you give it to someone like D3 or some other animation library, and you say, OK, animate this for me. What you're going to get is probably something that looks like that, which is like I see like knowing nods from people who've really experienced this problem. And this is not great. It's not really the effect that I had in mind originally. It's got this kind of weird Picasso effect, and it's very hard to keep track of what's actually happening. So this is like the beginning of the infomercial where the world is in black and white, and everything is terrible, and we have to find the solution that gets us out of it. I should have made this slide black and white, actually. So to start towards the solution, we need to talk about the concept of interpolation. And interpolation is basically a fancy pants way of saying coming up with a value between point A and point B. And there's a related concept called tweening that many of you have probably heard about, short for in-betweening, which refers to generating in-between frames of an animation to create the illusion of continuous movement. So if you were an old-timey Disney animator, you might be drawing like 60 pictures of a fox to make some sort of animation. But if we're doing, say, a D3 canvas animation, or D3 SVG animation, it might be more likely that we're saying, I have a dot that's over here. I want it to go over there. Please do all the math for me. I don't want to do it. And interpolate the x value from 0 to 100. And it will take care using request animation frame that Kaia mentioned to get all the in-between states of that dot as it moves. But something that we have to get on the table early that I'm going to kind of come back to is that there is very rarely one correct way to get from point A to point B. And as an example of why that's true, imagine you had a red square that you wanted to gradually animate into that blue square. What's halfway between red and blue? The answer is that it depends. So depending on what color space you're using, Kaia mentioned lab. There are many others. You could get very different results. These squares all start at that same red. They all end at that same blue. They all change gradually. But they all take very different routes to get there. Take another very ordinary example. The old saying we're all familiar with, there's more than one way to interpolate between a horse and a chair. This is from an amazing paper called Zoolomorphic Design by this guy, Noah Duncan. It is full of incredible diagrams about how to hybridize animals and household objects. He's actually written two papers on that subject. But getting back to shapes, so to start with a simple example, imagine you've got the square on the left. You want it to turn into the square on the right. You plug that into an animation library. You're probably going to be happy with the result. You'll get something simple because it's a very straightforward way to interpolate from the beginning to the end. You've got four corners. They're moving to four other locations. It stays a square the whole time. But we start to run into trouble if you want to turn your square into a hexagon. Because you've got these four corners that correspond to four points on the hexagon, then you have these two very lonely points that have no partner on the other side. And this is what is known in the academic literature as the correspondence problem. But you can think of it as the apples to oranges problem. Essentially, in order to interpolate between two values for the purposes of an animation, first they need to be the same kind of thing. And a square and a hexagon are not the same kind of thing in the way that matters. So if you were to plug those in, you'd get something like this, which is what we saw in North Carolina. Those two points suddenly blink into existence, and they suck the whole thing over, which is, and it inverts. And it's not great. It's not the worst thing, but we can probably do better. And in order to do better, we have to introduce the undercover hexagon. So the undercover hexagon is a dapper, well-traveled hexagon with many disguises. What I mean by that is by adding two points to the square that do not change the shape, you have turned it into a hexagon. So it still looks like a square for the purposes of your animation, assuming you're not actually drawing those dots. But from your computer's perspective, this is a totally different shape. Now you can get this smooth animation, which is probably much more along the lines of what you were going for. This technique of adding these invisible points is also useful if you're dealing with curves and straight lines, which is another sort of correspondence problem. A trick that you can do is basically decompose the thing that you have that's curved with a whole bunch of points to turn it into like a 100 gone that still looks curved, as long as you don't look really closely, but you can now animate it smoothly. There's a second problem that we also have to solve, which is that the order of the points matters. It doesn't matter to you, the viewer, the shapes will look the same, but it matters a lot to your computer. So right now, the shapes will start in the upper left and go around clockwise. But imagine that the first point in your destination data starts in the upper right. Now the way they match up, your computer is going to do something like this. It's not really what you had in mind, I would imagine. If they were up by 2 instead of 1, the whole thing would invert and flip upside down like that. So the solution to this is pretty simple. We test every possible starting point until we find one we like. We change the order of our data in a way that doesn't affect the way it looks. And in terms of what starting point you like, there could be a number of criteria for this. The rule of thumb that I think works pretty well is to say, pick the starting point where the points will have to travel the least. That all is being equal. The most natural animation will be the one where the points don't have to travel unnecessarily far to get to where they're going. So to recap with a more arbitrary shape, to show that this doesn't just work with squares, you first add these dummy points. And there are different strategies you could use, but I just evenly space it along for now. Then you pick your best starting point by testing every option. You see it's evaluating the way the points might have to travel in each version. And then once you have that, you can get a pretty smooth back and forth animation. Not perfect, but considering how weird those shapes are, I would say good enough. So if you can apply that to North Carolina, you know, end up with this. Much smoother, probably what you had in mind originally. You can now see what's happening with different districts, see which ones are going from very complex to much simpler outlines, which ones are moving a lot or a little or changing size a lot. There's one district, the gray one, you'll see actually flies across the state completely as no overlap with where it currently is, which is interesting. I took the same technique, and I used it to compare electoral cartogram layouts. I thought it was very interesting that different news organizations all took very different approaches to the same design problem. We've got hexagon, the daily coast, like turning the mountain west into a giant gulf is a very interesting strategy. Hexagons, these sort of slanted diamond shapes squares, smaller squares. Adam Pearce is here somewhere. After I had published that in North Carolina, I think he mentioned that he had used a similar technique for a project for Bloomberg about Florida's Fifth District, although I guess he sort of lovingly handcrafted some very artisanal polygons. As a maybe cautionary tale or counter example of what happens if you don't do this, this is a 2013 Guardian Australian election results map. And when you toggle back and forth between the kind of cartogram that Emilia mentioned yesterday and the geographic map, because the shapes don't correspond at all, you get those jumps and everything kind of explodes into these weird fragments. I went too far. We're not on the quest yet. These are all map-y examples, but this general idea of animations and correspondence comes up in a lot of visualization context. One very ordinary way that it pops up that I think is pretty common is if you have two line charts with different numbers of data points, the way that you would animate between those can go wrong in a lot of different ways. And Peter Beshai, who was also here somewhere from Boku, has written a nice library to address that problem called D3 interpolate path that I would recommend. So now we're on this quest where after I did this North Carolina project, I started wondering, like, this worked pretty well. Seems like sort of a crude hack though. Like, I just kind of was just sort of guesswork and I wanted to see what else was out there if there was some more formal like full solution in like the academic literature or something like that. With the criteria that I wanted to be able to throw any two shapes at it and have it just work and look good and like have to make no decisions. And so these are sort of my three criteria. So if we go looking for prior art on the subject, if I say morphing, a lot of you might think of something like this. Probably not something exactly like this, but like this kind of animation where you've got this image morph that where the two images both deform and then also crossfade to get this illusion of a transition. So this is an interesting kind of subfamily of morphing techniques. There's a foundational algorithm called the Bayer-Neely algorithm that's very cool. The paper is like surprisingly readable for an academic paper. Also like makes like a bunch of gratuitous references to the movie Ghostbusters for no reason. But it's kind of disqualified for our purposes. A, because it's really concerned with pixels and raster data rather than vector shapes. But more importantly, it's not automatic. So you see here, someone has had to go in and outline the matching features of these two faces in order to make it work well. So it fails that first test. So it's out. But before I move on from image morphing, I have to show my favorite thing that I found in this whole process, which is one of the pieces of software you can use is called Phantomorph. And if you go to their website, they have like a morphing hall of fame of things contributed by their users. And there's one which is a YouTube video. This is the only thing I need sound for, by the way. I told them I needed sound and it's only for this. It's like three and a half minutes of orangutans turning into other orangutans. All set to this song. Kiss me by six pence and I'm the richer. How am I in time? Can we watch? We can't watch all four minutes, but. So I'm just gonna stop that. But I recommend you watch it all in your free time. And animals are a recurring theme as you get into more like Google scholar deep cuts on shape morphing. You find like everyone has their like favorite animal to use as their demonstration. There's turtles turning into wolves. There's a penguin that turns into a dolphin. Cow turning into a deer. There's like a really weird 3D demo that's a woman turning into a triceratops. There's an elephant turning into a giraffe. There's a paper that's like very chicken heavy and also has this dancer. It's actually by the I think the same author as cow to deer. My favorite is this matrix that's like all possible in between states of a horse and a brontosaurus. That's like very formal. I mean, it's got scales and everything. So but as I was doing this research, I came across a paper that was very intriguing to me called 2D polygon morphing using the extended Gaussian image by this computational biologist at MIT named Manolis Cambicillus. And without going into a lot of detail, the basic idea is that rather than just thinking about points moving to other points, you deconstruct a shape into something called the extended circular image, which is about its angles and its weights. And you use those to decide what the in between shapes are gonna look like. And so you get a different effect that to my as kind of looks more natural in some cases. Unfortunately, it turns out it will not work if your polygons are concave. It will only work if your polygons are convex like the one on the happy polygon on the left. So it fails that second test. So I kind of gave up on it. But it did get me thinking whether thinking about a shape as a set of points was missing something, whether I was losing something that mattered and how to approach this problem. And as an example of why this might be true, imagine you've got this T on the left and you wanna turn it into the T on the right. If I were to give you like a piece of clay in real life and ask you to do this, you probably would not stretch out the clay. You would probably just turn it upside down, which is not something that the sort of naive point-to-point method was gonna do. This is the second half in the academic literature. You've got the correspondence problem and the second thing you have to solve is called the trajectory problem, which is what path are your different parts gonna take on the way to their destination. I mentioned that the Gaussian image thing was written by a computational biologist. As you research shape transitions and shape analysis, you run into a whole bunch of biologists, which makes sense because they have to do a lot of measuring, quantifying, comparing forms of physical things. And there's the whole discipline of morphometry. And in the old days, that was like measuring a bird's beak or something like that. This is from a book about 100 years old called Ungrowth and Form. That was like a structuralist biology text that showed how you should think about different species in terms of their geometric transformations between them. I think it turned out to be totally wrong, but the pictures are very cool. In the modern sense, though, this is very algorithmic. There's these whole disciplines of computational biology, computational anatomy. I think there was just a conference. I saw someone tweeting about a conference on Sunday at Harvard that covered some of this stuff because you have to deal with problems like we took a bunch of 2D scans of a bone and we need to try to infer the shape of the whole bone from only those scans, which is the same kind of problem as an animation interpolation problem. And I ran into a family of techniques that was interesting called Percrustes analysis. So Percrustes analysis is basically to address the problem that, if you want to compare the shapes of things, first you have to control for all the things that don't matter. So these four hands have pretty similar shapes, even though the actual coordinates of the hands are very different. They're in different places. They're turned at different angles. Some are small, some are big. Different Euclidean transformations in math speak. So you want to adjust for those first and then compare the shapes. And you can do this by a process called Percrustian superimposition, which is a three-step process. First you make your things the same size, then you move them so they're in the same place, and then you rotate them so they're approximately lined up. And then whatever differences you have left over are actual differences in the shape of the two bugs, rather than just the angle that you happened to photograph them at or something like that. Side note, does anyone here know the story of Percrustes? I had assumed that this was just named after a statistician like Lisa Percrustes. And then it turned out it is actually the Greek myth of Percrustes. He was a roadside bandit who would tell travelers that he had a magic bed that could fit anybody. And then he would lure them in, and it turned out it was just a normal bed, and he would stretch them out to make them fit in the bed. And some statistician in the 70s thought this was the perfect, not morbid at all, name for his statistical technique. But I thought that this technique, this sort of three-step process, might be interesting in the context of animation of taking two shapes and saying what if we control, what if we separate out those three parts and animate them differently? So that's where you would get this sort of effect on the top, where you animate the rotation separately, the scale separately, the translate separately, and then whatever's left over, as opposed to the bottom, which is what we were doing with North Carolina that just thinks of it as a list of points and doesn't care about anything else. And in this case, I would say it probably looks more natural, but it turns out it works for almost nothing else. And the reason being that these techniques, it took me far too long to realize this, but it turns out that these techniques are really designed for when you already know your shapes are very similar and they have common landmarks. It's like you have two butterflies and you say these are the wingtips, and then you compare them. It's not meant to compare like North Carolina's old District 5 to North Carolina's new District 5. So it will formally work for any two shapes, but it will look really weird unless your shapes are already almost identical. So after all this, I ended up pretty close to back where I started, and I was working up a demo to kind of clean up the code a little bit. I like using US States as demo shapes because they kind of run the shape gamut and you've got like tall, skinny, short, fat, square, not square. And here you can see the yellow points are the original vertices and the pink points are the invisible ones that are getting added just for animation purposes. And I thought, the more shapes I throw at this technique, the more it really does seem to work in all but the most extreme case, and I was pretty happy with it. But there was a problem which is you also have Michigan and Hawaii and Alaska, not pictured, which I had no idea how to handle those because they're like our beloved multi-polygon states that don't fit into this idea of turning one polygon into one other polygon. And I had been very satisfied and I was suddenly not satisfied at all and I could only use 47 states in that demo, I just had to leave them out. And so now like the quest was revived where what I really wanted was a technique that I could also just throw multi-polygon combinations of shapes into and it would work. So the puzzle kind of became if you had to animate California turning into Hawaii or some equivalent or if you imagine like a bubble chart, you have a big bubble that's gonna break apart into a bunch of smaller bubbles and make it up. What would that look like? And as I thought about it, I thought, well, what I would probably do in the physical world and presented this problem is break California up into puzzle pieces that fit together in some way. So when they're all together, it looks like California but you can also break them apart into as many pieces as you need. So I took a crack at this and there's a four step process, this is step one. You draw some sort of grid over your shape, your one shape first. I use something called a Voronoi diagram that Kyle also mentioned just cause I wanted it to look random and blobby but you could use squares or hexagons or whatever you want. Once you draw that grid, you assign one cell for each of your destination shapes. So we've got eight Hawaiian islands, there's now eight different colors, each assigned to one cell. Step three, they take turns claiming a neighboring cell. This is sort of like the islands are playing a game of risk and California is the game board. And it's nice because you can distort the rules to preserve relative areas. So the reason there's so much purple up here is because that's gonna become the big island of Hawaii which is a big island. It is most of the land area of Hawaii. Step four, you merge them back together into single shapes and clip them down to the coastline instead of that weird jagged edge. Then you can animate this. So now you have California one piece turning into eight pieces. Thank you. So I think this is cool, I was certainly very happy to get it working. But there's a catch. It will not always work, we're not done. For an example of why, look at this close up of San Francisco Bay. So you've got these purple cells and these orange cells, this is all fine, nothing's breaking the rules. But when you do that last step and you clip them, you now see the golden gate, that little gap, has separated the purple on the top from the purple on the bottom. They used to be touching, now they're not one shape anymore. So now you're right back where you started and you have shape problems. You have two shapes and you wanted one shape. So this is not guaranteed to work, it depends on what your input shapes look like and what kind of grid you draw and all these things that you can't necessarily control in the moment. So I thought about this some more and I realized I had a sort of Eureka moment that I shouldn't be trying to carve up California from the outside in, I should be trying to carve it up from the inside out to guarantee I stop at the border. And then I remember there's an awesome way to do this already and we've heard some references to this yesterday. People talking about WebGL. A lot of triangle love. I love triangles. Your GPU loves triangles. There's a method called triangulation that's used in a lot of computer graphics work that says we need to break up a 2D polygon into a bunch of triangles that fit together to make that shape in order to do things like cool 3D meshes. A triangle is like the atomic unit of a shape. And the fact that this is a common problem is awesome because it means I don't have to do any work. People have already figured out how to implement this in JavaScript and make it very fast. There's a library called EarCut written by someone in Mapbox that I love that I used for another example. I moved out, I stopped picking on California. We're gonna mess with Texas for a second. So you take EarCut and you say, here's my one big shape. I wanna cut it into a bunch of triangles. And you have way too many triangles, but that's okay. Once you've done that, step two, and the only other step is one at a time, merge one into one of its neighbors, keep doing that until you're down to the number of shapes you want. It's like 99 triangles of beer. And you get down to eight, and then once you're down to your eight shapes, you have what you wanted. You can now animate back and forth. Texas, turning into Hawaii or California, if that's your thing, it's my thing, clearly. And unlike the first method, this should be guaranteed to work for any simple polygon, even if it's got weird bays and other things that would get cut off. Here's another just totally gratuitous demo. Toph Tucker, who's here, tweeted something about this that I did not understand at all, referencing some math thing that sounded cool. But it's more just random states to get an idea of how this works in practice. So that's kind of as far as I got. The puzzle I'm now wrestling with is, what would you do if your shape was a donut? You have a hole in your shape. And what would that look like for that sort of animation? And I can sort of think of two possible approaches that might work for this, depending on the situation. I'll call them the cut and the wink. So the cut is where you would make a little incision at the beginning of your animation from the outside of your donut into the middle. So now you no longer actually have a shape with a hole, you just have one that looks like it. And now you can kind of push the hole out to the edge, like an air bubble, you're like squeegeeing out of a screen protector or something. So that's one approach, you get one sort of look. There's also the wink. And the wink would say that you move the hole with your shape, and then you kind of like blink it out of existence as it moves. And this is not that hard to figure out if you have two very simple shapes, like a circle and a triangle, but do this automatically for any weird shape where you don't know where the hole's gonna be or what sort of path it needs to take. Seems very complicated. I haven't gotten very far on it. I'm working on it. I'll let you know what I find out. I wanted to close with a quote. A lot of people are closing with quotes. I haven't quoted anyone. So I was reading all these academic papers where I was clearly not the target audience of these authors. I didn't understand most of the words, lots of Greek symbols, and also lots of things that were very impressive, but also really complex, working towards some definition of success that didn't really make sense to me. That was some formal criterion of correctness that only a mathematician could love that didn't apply to my real world animation problems where I just had a blob. I wanted it to be a different blob. And then I found this very refreshing sentence. This is actually the paper with all the chickens in it and that dancing guy. And near the end of the introduction, there's just the sentence that's just dropped in with no comment that just says, the solution presented here is a heuristic whose justification lies in the fact that it generally seems to work rather well. So they don't say anything about, here is the mathematically approvable reason why we did this right. It's just like, this seems fine. That chicken looks like a good chicken and you should stop there, which is kind of now how I feel about this after going through all these steps. I feel like you should not follow in my footsteps. You should instead keep in mind that all that really matters is getting the right effect in your visualization that you want and it's okay to do weird tricks behind the scenes and like hitchcock cuts and shapes kind of hiding in different ways to get the effect that you're looking for. And that's pretty much all I have. This talk has been very light on code as in there was none. But at this link, I've got links to like a bunch of live demos of a lot of this stuff to a lot of academic papers that I either mentioned or ones I didn't mention. I don't think there's a direct link to the orangutan video but there is a link to the morphine gallery that you can find it from which is probably the most important. And a couple of other things on there. If you have questions, we have like maybe two minutes but also just come find me or if you have questions, comments, ideas, complaints, corrections, I would love to hear them. Thank you.