 So, my name's Cameron Bacario, and I'm here to talk to you about my project Earth. So what is Earth, aside from the planet we live on? Very simply it's an animation of global weather conditions that runs in the browser. So today just a quick rundown of the major things I'm going to talk about, motivations, how it built it. What happened when I sent it out into the world, and lessons learned. But really quick I want to introduce myself, I'm from Cedar Rapids, Iowa originally, it's this small little spot right here near Chicago, Chicago's like pretty close, five hour drive away. And I graduated from Iowa State University with my computer engineering degree, and after school I went to Microsoft in Seattle, and I worked on the VisualBasic.net compiler team where we helped release the first version of .NET. After that I got a little burned out, decided to completely change things up, moved to Tokyo as one does, of course, and bounced around a few places and ended up at a company called TradingScreen, which is a financial services company, we built stock trading platforms. And after doing that for a little while, moved to another company I'm now working at Indeed, which is a job search engine, if anyone's looking for a job, go to Indeed.com, it works anywhere in the world, mostly. Okay so, no it does, it does. So motivations, in summer of 2013 I had no job, I collected stamps, for real, and my girlfriend said you better do something else, or I'm going to put a torch to your collection, or like mail a lot of random people. And she was right, I did need a new project, I'm not going to sit the whole time being unemployed, collecting stamps, which are cool by the way, they are, but she was right, I needed a new project, I needed new skills, I spent most of my career doing server side or developer tools, and I felt that my resume had some pretty big gaps in it, right? I wanted to help fill those gaps, I wanted to do something, so I wanted to focus on HTML5, some JavaScript, didn't know any JavaScript, I figured Node seems popular, give that a try, always interested in data visualizations, so I figured I should do something around there, but what to build. So I looked at a few things for inspiration, one of the things I looked at is this site, which I can imagine how many people have seen this site, yeah it's like most people, right? This was released in 2012 by Fernanda Viegas and Martin Wattenberg, and I remember the very moment I saw this for the first time, it just shocked me that, I mean 2012 is not that long ago, but in browser time it's pretty long ago, and this worked really well, right? I mean to have this happen in a browser, it just blew my mind. So it really stuck with me. And I'd also seen this kind of effect used before when I was in London on a business trip. The BBC had a very similar kind of wind particle animation that they had in their weather graphics. So I thought that was actually really cool. Also weather is really awesome, right? Weather is a universal human condition, right? I mean we all live on this planet probably, and the weather is the greatest icebreaker in the world. You can talk to anybody about the weather, right? I've heard many conversations this conference about how cold it is, and the amount of snow Boston got, et cetera, et cetera. I mean it's terrifying and beautiful at the same time, right? This is Typhoon Mesak, which was rampaged through Micronesia in the Philippines last week I believe. This was just shot from the International Space Station. So I thought, hmm, weather, wind, yeah. Sounds like a good project. Let's give that a try, right? Let's do that, where to start. So I did some investigations and did some Googling, and I found this kind of obscure library called D3. You might have heard of it, I don't know, and a great tutorial written by Mike Blostock, like let's make a map, and it goes step by step. For someone who doesn't know anything about coding in JavaScript or coding with geographic data or D3, just go step by step, like what you need to do. So I thought, wow, this is really, really cool. This was a huge, huge inspiration, or a huge kick forward for me to have found this kind of library and this kind of tutorial. But I needed weather data as well. So did another search on Google, found that the US National Weather Service provides a product called the Global Forecast System. This is a numerical weather model that runs on supercomputers every six hours and produces I think 15 days worth of forecast. And it's the basis of weather forecasts used by commercial entities worldwide, especially in the US. And the nice thing about it is all the data is public domain. So in fact, I'll just show you real quick. This is the directory you can just download the data like April 2nd, let's take a look at that. Here's all the raw data files pushed out every six hours. And they're in this format called, if I can find it, GRIB, stands for Gritted Binary. It's a common format used in the meteorological world. It's just a lat long grid, regular grid, and then the weather value data points at each point. So I had D3, I had that tutorial, I had the weather data. Sounds easy. I can do this, right? So how to build it. All right, this was the first moment I was able to get up like the whole coastlines of the world, kind of a weird mercator, you know, whatever. But at least it's up and running in the browser. I was so proud I could get this far following that tutorial. But it's flat, it's mercator, I don't like that projection too much. I wanted like a nice sphere, so the next step was to get it into a sphere type structure. And you'll notice white dots around the globe. This is the location of those data points I was talking about inside the GRIB file. I didn't know how to read the GRIB file, but at least I could try plotting the points along the sphere and then go from there. And that was the next step. The next step was to extract the data and then color the points with the wind speed magnitude. And I thought at this point, I'm like, okay, it's totally possible, this is doable. But this is where it got hard. So you notice that the grid points aren't dense enough to paint every pixel on the screen. There's a lot of gaps. So I had to put in some bilinear interpolation to fill in the gaps. Once I did that, got like a nice smooth sphere. The next step though was to add the animated particles, which ended up being really hard. I tried, and it's really hard because a flat map makes it a little easier. There's some distortion. In a sphere, like there's parts, you know, side of the sphere you can't see. It highly distorts, you know, where north is, et cetera. So I tried a number of different ways to visualize, wrap around my head, like what, wrap my head around like what's going on. I don't even remember what this was. It didn't really help. This happens, I highly recommend taking screenshots while you develop your stuff, especially when you see weird things like this. Now what I don't know what that was, ew, this is kind of disgusting to me. I don't know. But getting closer, right? And after a while, after a lot of blood, sweat, and tears, you know, got to the point where, yeah, the particles are moving along the globe exactly the way I wanted. So I want to show a quick demo. Who here has seen the visualization before? About half. Who hasn't seen? Oh, maybe like 60%. Okay, great. So let's just jump over to the live site. This is what it looks like. So we've got a, you know, a sphere with the wind moving. These are current conditions. You can interact with the sphere. Quick shows you down here the location and the speed. You can click through the units if you like, imperial units, metric units. You can interact with the sphere by dragging it. Any orientation you'd like, like let's check out Antarctica. That sounds fun. Try doing this on Web Mercator, right? Pretty cool. Let's zoom in. Let's take a look at the US. It's got some cool stuff going on, like whatever that is. In the middle, it's probably some kind of front moving through. This might have been a Nor'easter type thing. And there's a lot actually you can do here. If you hover over the word Earth and click, a menu appears. And the menu shows you the current date of the data you're looking at, what kind of data you're looking at, a color scale. You can hover over the color scale, and it shows you the value for that color. Lots of controls, so going forward and backward in time. There's five days worth of forecast data available, so you can step forward if you like. I can turn on that grid of points that I was talking about. Everything in between those points is interpolated. Let's turn that off. And we can look at different layers of the wind. So we can go up in the atmosphere, which is decreasing pressure levels. These are, this is air pressure and hectopascals. It's the same as millibars. You can go to like 1,500 or 850 or keep going up. We're getting closer to the jet stream. So this is like 500. I think around 250 is where the jet stream really kicks in. Let's take a look. Something like this. We can go higher than the jet stream. Let's go into the stratosphere, 70 hectopascals. Totally different kind of field there. And then 10 hectopascals, I think that's about 30 kilometers or so. And we can see the polar vortex that sets up over each pole during the winter. Right now it's in a transition period, we're in spring. So it's setting up over Antarctica now. This will start getting faster and faster and faster over the coming months. Let's see, we don't have to look at just wind. We can look at lots of stuff is available in the GFS data. So temperature, relative humidity. We can look at total precipitable water, which is the amount of water in the atmosphere. Right now I'm realizing my menu needs to be smaller. Not so good at the CSS stuff. So cloud water, so how much of that water is available as clouds and other things. I added one of these, I call it the misery index. Which is a combination of heat index and wind shield. So it just shows you what the temperature feels like as a human on the planet, Earth. Let's see, we can also, through the magic of D3, just switch through various projections. It just works, you know, it's ecorectangular, stereographic's a little funky, something like this. And you'll notice it's pausing every time I change the orientation or change the layer. It's because there's a lot of interpolation that's going on to save on performance, right? So that the animation can be smooth. So there's a little delay. I like this one, waterman butterfly. This one's pretty cool, it's pretty funky. You can do this too, for no reason. We also have ocean data available. So, oh, we don't want this, let's go to orthographic. So this is wave data, significant wave height and also peak wave period. Big storms tend to kick up a lot of big waves, like down here. It's not much going on in terms of extreme storms right now on Earth. Ice line gets beat up pretty bad. We can also look at ocean currents. So here's the Gulf Stream, let's take a look at that. This is the Gulf Stream, this is data that's updated every five days. We can also look at, drag that over there. We can also look at sea surface temperatures. Oh, one moment, there we go. So sea surface temperature, how warm the ocean is. And also, sea surface temperature anomaly, which is how far from normal the temperature currently is. Normal being the 30-year average. Yeah, I tried to keep, let's switch back to the default. So I tried to keep the imagery really simple, right? No social buttons, no ads, no share of this with your cat stuff. You know, very, very simple, right? And also, if you do change any of the orientation or you set it up some way that you like, the URL encodes the state of all of that. So you can just copy that URL and send it to any of your friends and they'll see exactly what you're seeing. I think that's very, very important, especially for interactive visualizations. For ones that are rich, right? Because if people are interacting with it, and they see something that they like, they wanna be able to send that to their friend, right? So I highly recommend you take that state and encode it in some way that allows that to be shareable. In my opinion, the URL is the best. Okay, so that's the site. So I wanna go into in depth a little bit about how it's done, particularly the particle animation. Okay, so we start with a vector field. This might be something that is like a typhoon, right? Or a hurricane, something that's got a circular motion to it. And we just start drawing particles in one segment at a time, right? And each time we draw a new segment, we look at what the direction of the vector is at that location and we draw a line in that direction. Now, if we fade particles that we've drawn previously, then we get kind of like a nice little tail that starts showing up. And if we increase the speed, and spawn it at a random place when it's too old, add more particles, increase their speed, add a lot more particles, and we get what is undeniably circular motion that we see, right? And that's basically how it's done. That's it. Now, the hard part was the distortion bit, right? How does the projection distort particles? So let's take a look at that. Let's start with the globe. It's interesting, right? Because north, which is represented here as red lines, is not always a line straight up on the screen, drawn on the screen. It depends on where you are. So north is kind of straight up over here, but over here it's like a line that goes down, and east has the same kind of problem, right? So if we go along every pixel, like kind of brute force every pixel and calculate what the direction of north and east is, then we get effectively the shape of the distortion that we can apply to the wind data that we have. To have it like match this curvature of the earth. So let's just look at the east, eastwardness, as its own vector field. And we can do the same kind of thing. We just draw a particle one segment at a time, increase its speed, add a bunch more, and we have effectively particles that are now compensating for the distortion caused by the projection that we're using. Okay, so the animation itself is composed out of four layers. At the very bottom is an SVG layer using D3 and Topo, JSON to draw the coastline. And then sitting on top of that is an HTML5 canvas, which is where all the particles are being drawn, right? That technique we just looked at. On top of that is another canvas that encodes all the color information. Either the temperature or the wind velocity. And then finally on top of that is another SVG layer just to clean up the edges, right? It's just a black circle, gives it a nice anti-aliased look around the edge. Kind of a hack, but it works. The data, the wind data and the weather data ends up being like kind of large and takes a lot of time to download. So I originally downloaded the browser, and I originally used JSON plain text to do this. And it quickly became annoying because the size of the files would get so large, especially when the GFS upgrades its resolution like it did in January. So I added a compression technique to the data, which we can do with four steps, scale the values, filter them, binary encode them, and then GZIP it. GZIP we get for free, the browser does that for us, it's built right in. We just have to put GZIP content encoding on the header. So let me look, let's look at how the filtering is done, how we use filtering to help this whole process along. So if we look at a really small grid, maybe of just temperature. We can see that's a bunch of kind of big numbers, right? What we wanna do is massage these numbers so that the GZIP process can be even more efficient, right? Now what we can do is we can say, well, let's, for every grid cell, like every four points, let's look at three points, A, B, and C, and they form a plane, right? So whatever the temperature is at these three points, it forms kind of a plane, and nature is kind of predictable. It's not entirely chaotic and random. So this plane ends up being a good sort of predictor for points around, like the fourth point. We could extend that plane out and predict what the value for that fourth point might be. And if we just do a standard, like a normal linear thing, it's just A plus B minus C. Now the actual value might be x, right? Probably not exactly P all the time, of course. So instead of encoding x into the file, we just encode the difference between our prediction and the actual value. So for example, if the temperature we predicted is to be 72, but the actual temperature was 68, we don't write 68 to the file, we write 4. And we just keep doing this for every successive cell in the grid. So predict, find the delta, predict, find the delta. And when we do that across the whole grid, we get something that might look like this. The numbers on the right hand side require fewer bits to encode in the data. And furthermore, they're normalized, right? So the GZIP process can actually find greater runs of values or more common patterns and end up doing a lot more compression. Quick discussion of the architecture of the site. So all of the weather data is available on a server run by the National Weather Service, it's called Nomads. And that's where all those grid files were. That's what I was showing you, where all those grid files are located. And I have a rental server using DigitalOcean. I call the server Gaia, which stands for Global Atmospheric Information Assimilator, which I made up for this talk. I did call it Gaia, it worked out really well, the background. So this runs Node and a Java utility that I wrote that uses a open source library called NetCDF Java. That library knows how to read grid files and the Java utility writes this binary encoding, which we just discussed. I call those E-packs, like Earthpack, I don't know, it's stupid. And then Node.js, which is kind of like the overall cron style, like orchestrator of when to download, what to download, how much of it, etc. And uploads all of these files to Amazon S3, right? For permanent storage. Now Amazon S3, static storage of all these files, can be a bit expensive for bandwidth. So in front of S3, I put a free service called Cloudflare, which is free, right? Bandwidth is free, amazingly. I don't know how they make money. I mean, they make money with extra features, etc., which I don't need. Which works great. So the Cloudflare plays a really good role, it's not only free bandwidth, but it's also replicating the data across the world in multiple data centers. So when the browser running D3 wants a particular file, it goes to Cloudflare and Cloudflare says, I have this file, here it is, immediately. If it doesn't have the file, it goes to S3, comes back, saves it, and then serves it out. And by doing this, monthly costs are with digital ocean, S3 costs, it's like $15 a month or something like this to have the whole thing running. Okay, so built to scale, got it done. Is anyone going to bother visiting this thing? I have no idea. So let's take a look at reactions. So came time to launch the site. Launched it on Twitter, sent out a tweet, created a Facebook page, told some friends, sat back, waited for the thundering horde to arrive. And nothing happened for two days. I think that was my mom, this is me, is the site still running? And then whoa, hey, wait, 64 people visited, right? In one hour, well, 400 users in a four day period, that was awesome, right? Oh, wait, claps is back down to zero. Is that it? It's like, that's kind of disappointing, but, well, it was fun. So, seven days later, half a million users. That was kind of crazy. 30,000 users in one hour, that was pretty crazy. And this happened while I was on a plane to the states. And I couldn't really see or control what was going on. Okay, so fast forward to today. This is the total traffic to the site since launch on December 10th to March, total 29 million sessions, 12 million users, 40 million page views. I don't even know how that's possible, because there's only one page. Like what is this, like what? Whatever. But the impact that the engagement that the community had was really great, right? So we've got organizations like Oxfam International tweeting out using imagery from the site to inform people about CyclonePam, which went through like three, four weeks ago, decimated Banuatu World Meteorological Organization, which is the weather arm of the UN, was tweeting out about the site. NASA Earth Observatory, Al Gore, like what? He was actually one of the first to tweet out about this. And Forbes Magazine, when a big storm hit the Bay Area over the winter, tweeted that out. I mean, this is a huge broad swath of organizations and people around the world. It's been on TV, so Japanese television, the weather channel down here in the lower left. This is when Hurricane Arthur came through last summer. Upper right is BBC World News, again with CyclonePam hitting Banuatu. It's been featured in print as well. Irish Independent had a big front page feature on this. Not about the site, but as an accompanying graphic to the storm that was being hit, or that was hitting Ireland. On the right is an article that was published in Science Magazine, and the researchers asked if they could use this accompanying figure for the discussion on polar vortex that caused some really cold weather to hit the US back in January 2014. People have also done some awesome stuff, made their own stuff, right? Like I love this, some little artsy thing. The upper right is a, that's a hand embroidered, like one meter wide piece done by an artist in Montreal. Pretty crazy. And she used the water bin butterfly to do it. Funny story, like shortly after launching the site, I get an email from a guy named Steve Waterman, and he's like, thank you so much for using my projection. I really like it. And I'm like, what? I thought like all map projections were invented 200 years ago. You know, and like apparently it's still a thing, right? So the things you learn, the more you know. So, okay, lessons, what we learn. All right, usability. No one knows where the menu is. Really, I thought I was being clever and like, oh, let's keep it simple. You know, let people find it so they can engage with it and no one finds it. So like I did some measurement and 80% of visitors never click on the menu, which is terrible. It's really terrible. I kind of don't care. I don't know. It's difficult because anything I try to think of that would highlight the menu as being a thing kind of destroys the aesthetic of the site for me. And I haven't been able to find something that would make it look good. So I don't know. I'm just kind of tolerating it for the moment. The other thing is people really don't know what they're looking at. I didn't even know what I was looking at. When I started this and let's take a look at what this means. It comes down to the difference between streamlines and path lines. So let's take a look. So this was the example we looked at earlier and you'll notice that the clock is not moving, right? So this vector field is a snapshot in time and yet we're animating it. So the animation is occurring over space, but not time, which is very different than the way humans normally interpret animation. This kind of technique is the particles are following what are called streamlines, which is the line of the vector field as it exists as a snapshot. So if we highlight a particular particle and just follow it, we can see that it's a circular path, right? A circular streamline. But this is not how the weather actually works. Things move, right? So here's our cyclone moving off to the east. The clocks moving up ahead or up above top. And if we do that with the particles, we can see the particles are following them. Let's highlight that same particle and follow its path through time and space. And it's completely different, right? This is what people think they're looking at when they visit the site. It's what I thought I was looking at, but it's not. And that's kind of concerning, right? Because how it's a very subtle difference and I don't know if it matters. I think it matters. I mean, it's not, when your users have one perception and the visualization you know is showing a different thing, like that matters, right? But I don't really know how to communicate this difference very well. And I'm not a news organization. This is not supposed to be used for actual stuff. So, yeah. And keep that in mind when I show you this next slide, which is how various media outlets are now incorporating this technique into their own graphics packages. Like CNN, NBC News, might be a little hard to see NBC News. NHK, Weather Channel. I know the Weather Channel one was directly influenced by my site. I don't know about the others. I think they occurred after the site went live. But it doesn't matter, right? I mean, this is an animation technique which is engaging to people. But again, it's like, is this, these are now news organizations showing this. And is it reality, right? So I have concerns about that. I mean, on top of that, people are using the site for actual stuff, like blogs and tweets saying that the reason why Boston had such terrible snow, this past winter was because the sea surface temperature anomaly was so, so very high, right? Off the coast of Cape Cod. And Michael Mann's like a major, person in the climate change area. And he's using my site and other people are using this imagery. And I'm like, how do they know I wrote it correctly? I, how do I know I wrote it correctly? I mean, I'm not a scientist. I don't know if the techniques that I used to construct the SSTA in a layer was correct, are correct, right? They looked okay to me. That's why I put it up. But, I mean, when something, I guess this speaks to the power of visualizations, right? Especially if they look somewhat legit. Like they confer an authority that isn't warranted, right? Even if you put disclaimers, people still look at it and they go, oh, this looks legit, but I'm not gonna use it. So that's kind of an open-ended concern I have. Final lesson is free, like free stuff, right? None of this would have been possible without free stuff. Free geographic data, free frameworks, like D3, Backbone, WENJAS, free bandwidth, right? Freeze and beer bandwidth. From Cloudflare, had I not used Cloudflare, I would be bankrupt, probably due to Amazon costs, and free help, right? From Stack Overflow, from Twitter, just from anybody, right? And most importantly, weather data, right? All free, all public domain, right? Think about this, right? Our governments are building satellites, and they're building rockets, and they're launching those rockets, and then they're building sensor networks to detect all the, to aggregate all this data, and they're building supercomputers to run all this stuff, and they're hiring scientists to run these things, and then they give it out for free. And this is like literally billions of dollars worth of data available at your fingertips. And I find it just amazing that an unemployed developer sitting in his pajamas all day can have access to this kind of data and able to build stuff like this, right? I really think that's a great thing, and I really think that open data in some sense is almost more important than open source, right? It's the data that really matters. So anyway, thank you.