 All right, so hi everyone. I think some of you may recognize this picture It's one of the many famous images of the sky over Chetobinsk, Russia back in February 2013 Now that morning a huge fireball hit the sky and a rock about 20 meters in length weighing over 10,000 tons Hit the Earth's atmosphere and exploded with the power of several atomic bombs and for a lot of people When they woke up and in the morning and read that news It was the first time in a while that public discourse and the media really focused on on asteroids or what was beyond the Earth's atmosphere and so on But for people like me and other scientists and astronomers we think about this stuff every day and I want to talk to you all today About the work that I've done to make it easier to browse organize and analyze the information that we have on these objects and also even to discover and contribute to the science and Detect objects similar to the one that exploded over Russia in 2013 and I'll tell you a little bit about the story about how all this came together how the data came together and ultimately the visualizations which were really key to getting this message across and Attracted the interest of a number of different organizations in the government and so on so Some quick background on myself. I'm a software engineer at Google prior to that I worked mostly on front-end, but also some infrastructure at a startup that was eventually acquired by Google I worked on this space software stuff separately wound up selling it to a spacecraft company and worked there for a little while as well So in order to get into the details We're going to have to bring in some context around asteroids and space So asteroids are rocks in space They range from anywhere between a couple meters in length to hundreds or even up to a thousand kilometers in size and Most asteroids reside between the orbits of Mars and Jupiter in what's known as the asteroid belt or the main belt But a subset of them are considered near earth objects or any os and of these and even smaller set are considered potentially hazardous objects And they're about two thousand of these objects where there's some probability That they may eventually collide with earth or cause some problems So it's the neos and the potentially hazardous objects that we are particularly interested in and scientists and astronomers are most interested in With the notion that it's only a matter of time before the earth gets hit by one of these things again and maybe it'll explode over Boston instead of Chelyabinsk, Russia and That would be a bad day for a lot of people Asteroids are not just big existential threats They're also the remnants of a primordial solar system one that formed over four billion years ago and The interesting thing here is that we see in asteroids the same basic building blocks In nature that we see in earth so volatiles like hydrogen nitrogen oxygen Substances like water various silicates platinum group metals nickel iron cobalt and magnesium are all present in asteroids And they tend to be present in higher quantities Then we normally find on earth because you don't go through the same weathering processes on earth and just to give you a Sense of scale here. We're looking at a false color image of Vesta, which is about 500 kilometers in diameter So we're talking a lot of stuff in this picture and there are companies that are actually Going out and want to build spacecraft and go to these asteroids and extract these materials. So Planetary resource is an example. It's funded by Google CEO's Larry Page and Eric Schmidt there are other companies like this and all this I tell you to Sort of set the the background for me, right? So where do I come into this? Well, I'm sitting here reading the news about how these billionaires want to go out of mine asteroids I'm wondering if it's really going to happen if it makes sense And just in general, I think space is pretty cool And I looked into this a little more I realize that there are about 750,000 asteroids out there and the data is out there We know a lot about them scientists write their dissertations on them But it's not really something that I personally can wrap my head around and I want to answer questions like which asteroids are going to swing by the earth next how long would it take to go out and Reach an asteroid, which ones are the best candidates for manned missions for automated missions that sort of thing So I went to the internet and I found that the best The best place for this sort of information is NASA JPL So NASA JPL has this really great form where if you know what you're doing If you've been like working on a PhD or in the field for 10 years You can click all the right buttons and get the information that you need about any asteroid in the solar system It's a very powerful tool But probably not one that's well suited to someone like me or to probably most of you people as well There's also the Minor Planet Center So the Minor Planet Center is even a step down But the MPC is actually the canonical source for in front for this sort of information So they are the ones that decide what is an asteroid was not an asteroid Where is this object? What is its orbit and so on and they provide it in this very convenient text file that is space delimited and And barely suitable for human consumption let alone You know computer Consumption so I think a lot of people have Implemented the same script that sort of massages all of this and puts it In into a format that is actually useful So I wound up creating this tool called aster ink and the reason why it's called aster ink is because I wanted to give ranked query results For arbitrary questions that you have on asteroid data or small-body data So answering questions like which asteroids have return on investment or just which or just simple questions like which ones are easy to get to or which ones You know might land on a city one day So asteroid is just a way aster ink is just a way to organize browse and visualize asteroid data It's been used in the scientific community. It's been used by these commercial aerospace companies And other organizations as well But above all and really what I'm here to show everyone today Is that aster ink is a way to let the public engage with space and the goal here is to make it so that We can take all this data that these spot scientists have collected on asteroids and put it into the hands of normal people So that they can understand it so that that sort of work is meaningful to them and so that they can contribute back And this is where being clever about how you present the data You know allows you to make huge improvements on the status quo So let's talk a little bit about how this will work the first step is to gather data So I wrote scrapers for NASA JPL the Minor Planet Center about half a dozen other scientific sources as well as economic sources because I was interested in these guys that got funding to go out and mine asteroids and that lets me pull in sort of objective truth things like the orbits of asteroids and so on and Put this all together in into one database I finally have like a unified data model that contains everything that humanity knows about all these asteroids Then I wrote a bunch of calculations. So things that estimated asteroid composition Estimated their their spectra. So the light that they gave off Things like mission cost shortest routes, etc And I found that really the the main thing that we're interested in here is this quantity called Delta V And Delta V is the amount of energy that you need in order to get from point A to point B in space And that's like the single biggest thing that you care about Because the more energy you need the more fuel you need and the costlier everything is so lots of calculations go on to compute that and also compute mission windows and that sort of thing and finally you put it all together and You wind up with a spreadsheet that looks a lot like or you wind up with this which looks a lot like a spreadsheet and Sort of ignoring the numbers here because this was more just playing around and trying to figure out What I could what sort of questions I could answer using this I think that the reason why this is special is because all this data is in one place instead of scattered around the internet in these little scientific repositories I'm a more visual sort of person So I wound up using a technology called WebGL, which we just heard a lot of To build something that runs in the browser runs on your laptop, but is completely scientifically accurate or you know with some Artistic license, but all the positioning that you see here all the individual objects here actually represent something in space and All the positions are accurate and so on so the idea here is to let people actually go in and View all this information in context and this is how science becomes alive, right? You go from a spreadsheet or you go from that space delimited notepad file to something that you can actually reach out and touch and That is the goal of this project Asterank so my favorite view which we're actually about to switch to is called the accessibility view and This shows objects that are the cheapest to get to from earth and you can see here how they sort of cluster around the earth's orbit and it's really beautiful and I think one of the most interesting results from this from all these calculations that you can see here now is that there are over 2,000 asteroids that have lower delta V costs than the moon which means that they're easier and more within reach than the moon was Back in 1969 so that's a pretty cool result, but it's not just a toy when you pair it with the data view You get this really rich experience that allows you to view this information that NASA and other Research organizations have collected and you put it all into one place, right? And this makes it easy to ask questions like what are the upcoming passes? So you can see they those all just clustered around where the earth is and You could also do things like which ones are which are like the smallest asteroids and NASA's particularly interested in this because they're Investing in asteroid redirection missions So this is the data view I think the overall The overall takeaway that I was trying to communicate when I was building these visualizations is that space is not empty, right? Space is full of opportunity. It's full of interesting science and this is really what I think the power of WebGL was in this case is allowing people to communicate the research and the data that they've collected and Give it to the world, right? The data doesn't mean things to people if it's not something that you can reach out and touch that you can play with and that I think is What made this really special to me and to a bunch of other people? so I made I made these visualizations that was great, but I also was trying to think of a way to Make it easier for people to get involved in the actual science Understand What went into discovering all these little objects? Circling in the solar system, so I was doing some research and talking to some people I discovered that we are finding about one to two one or two new asteroids per day We discovered, you know, a new rock in space And about that rate and the way it's currently done now is you have these big telescopes these big sky surveys that are typically publicly funded and It's usually take place at universities or government Locations and so on and what they do is they take a bunch of pictures of the sky and they run a computer algorithm over it That was written back in the 1990s if the algorithm finds something it flags it and a human takes a look Otherwise that picture is stored away forever in some hard drive or on like magnetic tape So that's how sky surveys work and taking a specific example. We have the Sloan Digital Sky Survey or the SDSS and it's been running for, you know, a very long time and This this image may not make that much sense to someone who Is not used to looking at this stuff, but the basic point is that it's covered a lot of the sky And it's taken a lot of pictures of Those portions of the sky so there are terabytes and terabytes of Imagery and millions of images and I took a subset of these and ran them through a very useful tool that Does what's called geometric hashing in order to try to locate based on the positions of stars and other objects in the sky To figure out what part of the sky That we're actually looking at and when I put it together I created something called asterisk discover and this is crowdsource asteroid discovery with millions of sky survey images Over these gigabytes and terabytes of sky survey imagery and the reason why I'm talking about this at a visualization conference is because the process of Discovering an asteroid is inherently visual what you do is you there's this process called stacking and blinking So you take a couple pictures of the same spot in the sky take in over time over the period of say several minutes And then you just play them in in a loop in an animated gif and You can see asteroids in these images Scientists estimate there are up to like 10 million Asteroids that we haven't found in our solar system So it's not so much a question of whether these objects are visible in the imagery But it's a question of whether we can find them whether we have the right algorithms the right people and so on so This was my this is sort of like an eye test actually But you can see if you look very closely an asteroid that's moving on the otherwise Static backdrop of stars which don't move as fast as asteroids in the night sky So this is a strength discover We've reviewed about a quarter million images this way so and and of those about 10,000 images have been marked as potentially interesting and maybe that will translate to like 10 new Asteroids that are found right? I don't know yet because it's a long verification process but to me anything is a win right because I'm letting I'm bringing in all these people they're about 10,000 people that have contributed to this project and I'm letting them use their own eyes and their own skills and their own excitement about space to Get involved in a really fun and visual way and if you discover an asteroid And you're the first person who spotted it you get to name it so you can think about You can think about what you'd name your asteroid If you name it after your mom or something like that But but that's that's what gets me excited about this So the the overall result of all these visualizations and that sort of thing is Millions of people have seen it quarter million sky survey images reviewed This is used as a teaching tool in schools and universities. It's used by Commercial aerospace companies to try to raise funding I've like helped people through that and There are hundreds of developers who follow it on on GitHub for example and and a subset of those have made Contributions back to the code that has gone live either in the discovery aspect or the data aspect or or the visualization So that's pretty great I wanted to dive more into the technical detail So I've sort of let you guys I've given you guys a taste of like what I'm talking about But now we can get into the nitty-gritty The bottom line here basically is that this app doesn't look very different from probably most of the apps that you're used to so the only Different part maybe is that I have a pretty extensive like data processing pipeline image processing pipeline, but the rest looks pretty normal and The the way this looks is in on the data pipeline side You have a bunch of scrapers and these are mostly written in Python But have a bunch of bash to glue them together and then you have a data pipeline, which is really just a cron To go out and run these scrapers every night and bring in all the data And that just gets written to a database and I chose Mongo for no reason, but other than it was easy at the time Right The image pipeline is a little more complicated So these raw images come in from the Sloan Digital sky server or any of these sky surveys as like these huge fields basically big chunks of the night sky and sometimes they're not even Sometimes they're they're stored in proprietary binary formats so you actually have to go in and Twiddle things bit by bit in order to get an image that you can actually show to a user so the image pipeline takes care of all that it goes through each of these fields and crops these these images into smaller bite-sized chunks where you can actually do the visual diff as a human being and then it adjusts some things like it decreases the noise it jacks up like Certain parts of the image like contrast and that sort of thing to make it easier to spot these asteroids in The images and then it writes it to disk with a pointer in memory and everything there is served by The the server which is just a simple flask app Which sits behind engine X and then cloud life through and cloud flare for good measure but I actually the reason why I do such heavy caching is because Ben with did become a problem and especially when something like this gets written up by like You know, you know in the media the site will crash and burn starting with Mongo and So so the cash layer became necessary and everything is served to the user So the the evolution of this project brought me through a bunch of these different steps Where I was learning as much about visualization as probably everyone here today and I want to share with you what I learned at each step along the way so But just a quick overview the start as a D3 visualization just looking at everything top down And it looks like a mess, but it was at this point where I realized that it would actually be really cool to Look at this in 3d. I think there's something here So this is my first stab at the 3d version, which I think was Canvas animation and I could render about 20 or 30 asteroids Before my computer, you know got too hot and things started getting a little jumpy So that this is version one version two I Learned a lot about the way that the stuff works and I tried to make things more efficient I could render maybe a thousand and then eventually we got to the point more or less where I showed you earlier Which is showing all these objects of thousands and thousands of objects Moving in real time and in a way that is scientifically accurate. So I think people who are Familiar with WebGL already know that WebGL is very powerful and you can render hundreds of thousands of particles and that sort of thing I think that this was sort of uniquely challenging because each of these Little dots is positioned according to the laws of physics according to orbital dynamics, and it's not too complicated There's a lot of trigonometry and some basic calculus But the bottom line is that each of these points is unique and it's moving in its own Unique way that's that's defined So getting more into the lessons learned I think the main takeaway as I was putting all this together is that the web is basically single threaded So this means a couple of things it means you should be smart about how you handle your assets And I think this sort of goes without saying but you want to parallelize You know your assets you want to only use them and show them when they're necessary And then the other things that I want to talk about are web workers and then time processing of data So getting more into web workers Those of you who have done desktop applications may be familiar with threading maybe at an operating systems level you can think of Web workers as sort of threads for the browser. There's they're they don't map It's not the same as an OS thread, but you can Really use them in the same way and the point here is that whenever your JavaScript is executing you're blocking the UI Right, so this means in practice you will you will see jumpy visualizations And things won't work that right and the way web workers work is that you can Basically create this worker which won't block the UI and send messages back and forth Between them. So what this looks like on the UI side is that I create a web worker. I want it to Be this JavaScript file and I send it a message, right? So in this case, I'm just sending foo and bar That goes over the wire from the UI to the worker And on the worker side, I have an event listener where I receive the data and this is where all the magic happens So you take the sort that you take the initial data you do all the heavy processing on it that you don't want to block the UI So I'm just checking for equality But you can do all your calculations and that sort of thing if you want and then just send back the result So I send back the result to the UI and the UI just has a listener That prints out the result some takeaways from working pretty heavily with web workers is that Serializing messages is very costly. So this means sending data from the UI to the worker thread We'll take a good amount of time. So the way this works in real life is really you want to send all your data up front and And and and have the worker and then just send back the result instead of doing things little by little and Sort of for this reason. I've also found that one worker is usually enough You don't get linear returns when you keep on adding web workers, even though it's very tempting Some limitations of web workers is that they can't touch the DOM this includes the window object which means that you don't get things like console log and You don't really know anything about the web page So the use cases are somewhat constrained and you need to be thoughtful about What actually makes sense to put in your worker and of course? It's not supported on all browsers So you need to use some fallbacks and one such example is the next topic Which is time to ray processing or chunking and the basic idea here is that you just take a lot of data maybe your your Three-quarters of a million asteroids or whatever you want and break it up into smaller pieces The way that this looks naively is that you loop through each particle and you sort of do something to it So in this case I'm updating something to do with the particle. This will block the UI, right? So you'll see the particles will like jump forward The visualization will lag and then they'll keep jumping forward again, which is undesirable so an alternative to this is Basically yielding execution flow back to the UI let the browser do things like reflow let it catch up with whatever user-generated events is Going on anything else that's going on And I know this is sort of like a mouthful of code But the basic gist is that you break it up you break up your array into smaller chunks And you work on that either until you're done with your chunk or you're out of time Once that's true. You just set a very very short time out and what that does is that it gives execution flow back to the browser so you're no longer blocking on your single complicated or CPU intensive loop So that's time to ray processing Eventually it became clear that a lot of this really just had to happen on the GPU, right? So WebGL is extremely powerful 3js is a really useful WebGL library That is really the most popular JavaScript WebGL Framework out there So quick takeaways here is that it's useful to use a particle system So it's very tempting especially if you know nothing about visualization like me I would I would go in and I would want to represent each little particle as its own object Right because that's how I model it in my head but in real life you want to use a particle system which just uses a single geometry and models these These particles or asteroids or objects as vertices that are like constantly shifting and that's a lot more efficient under the hood You want to put everything you can on the GPU? So even though it's tempting to write a bunch of JavaScript to do everything you want You really do have to invest in those shaders that we learned about earlier and write that GLSL and move as much logic as you can Onto the graphics card There are a lot of gotchas when developing WebGL you need to really test on every browser and operating system And this is more so than just like the normal internet Explorer headaches that you guys get There are there are all sorts of differences. So on Windows, for example I think if you use directX you're really working in HLSL and it gets there's a layer called angle that translates GLSL to HLSL and that will drive you crazy. So you need to You need to test per browser But also with all the operating system permutations and you'll notice that there are things that work differently or that don't work as efficiently on even the good browsers like Firefox versus Chrome for example and then My other tip here would be to monitor your frames per second and then gracefully degrade your simulation So a lot of people put an FPS counter in the bottom left, but instead of monitoring it passively My advice would be to actually use that number and turn on or off certain parts of your simulation That you think will keep the user above 40 and ideally at 60 frames per second So you put together all this you keep all this in mind and you can improve performance about 10,000 times which is what I did in this simulation and Just to prove that it's not Not just a crazy one-off, you know asteroid specific set of tips. I What I was looking for I was looking for a different data set. So I'm still interested in space So you're gonna hear more about space, but I found this this computation Computational undertaking called the Millennium project or the Millennium run and at the time it was the largest and body physics simulation that was running the world and the idea was to model the entire life of a universe and all the galaxies in it and See how gravity and dark matter Interconnected you know did stuff with one another So what we're looking at now is a cube of two billion light years two billion cubic light years in size So that's like mind-boggling every point of light that you see here is a single galaxy right Which in itself could be hundreds of thousands of light years across and I think the really cool thing here, and I don't know how well the visualization shows up on your screen But you can see a lot of the really intricate beautiful Gravitational structures that occur on a very large scale So it's really cool when you when you increase the number of particles you can see these crazy designs and patterns that occur in nature at sort of the largest level that You know that that we've ever studied And they look surprisingly similar to other things that you find in nature So I think that's really cool, and that's an example of something that you don't really get when you're just looking at these numbers I mean these scientists were querying data out of I mean they were working basically with like my sequel and That's that's great Like if you have all the context and stuff, but if you want to show people what you're working on you really need these sorts of visualizations So I applied everything that I knew from before But I also learned a couple other tricks to make it easier to display literally Millions and millions of these data points that represent individual galaxies so Really here at this point the the gains that you get all happen offline With with pre-processing so what I did was I grouped all these particles. I had my entire universe I grouped all these particles Using an archery, which is basically just like a spatial way to to organize data and once I got it into a little quadrant that I Determined roughly matched the resolution of a screen or of the human eye I Did everything that I could to Basically cut corners So the name of the game here is to try to remove as much as you can from the visualization and from the data But preserve what actually makes it to the user and an example of that would be to take two galaxies that are close to one another And then merge them into one galaxy and then increase the luminosity to make it visually the same thing that you would expect to see but Under the hood, it's you know, it's it's a lot easier For your machine and for your graphics card to do that other things I did are to Adjust particle size and sort of apply and remove shaders on the go So there's a lot of stuff that goes on in this simulation a lot of it depends on your current FPS But a lot of it also just depends where you are in the universe How far zoomed in you are and how far out with the goal of preserving that same user experience? And conveying the same information, but perhaps not doing the same amount of computational work So that was the idea behind this universe visualization so I Thought that you know, I put to I put a lot of work into these visualizations but one of the biggest things in Taking data and bringing it to the general public is being able to provide it to developers so I Wrote a lot of API's I wrote I provide an asterisk API and API for Kepler from the Minor Planet Center and for Sky Survey imagery and All this is accessible on the website and you have people that actually use it to make their own sorts of interesting frameworks and or visualizations and that sort of thing And one such example is this guy who created this crazy like Empire building game around the basic asterisk simulation So you can go out and colonize different asteroids. I don't even I don't know what's going on In You know in in every corner here But it's it's really interesting and also honestly pretty rewarding to see people use your work in that way And I've seen other things like people Sort of like showing a night sky and having you mouse over certain portions and pulling up imagery Sky Survey imagery of that night sky as well So this is pretty cool I started working with some of the Kepler data and basically just dumped it into the engine that I built so all these little dots are Planets that have been discovered outside of the solar system So like the future potential earths or whatever And but they're all they're all like overlaid on top of one another so you can really see here It gives you a better sense of like what we've actually discovered What's habitable? What's not how it compares to our own solar system? If you're interested in that sort of stuff Another example where I just sort of reappropriated the visualization engine is I'm doing work with one of the top scientists at the SETI Institute and his work is on meteor showers and What we see here is there's a Comet that went through our solar system left a bunch of debris and every time that that Orbiting debris intersects with the orbit of earth you get a meteor shower And this happens about every August and you can actually see this in the visualization If you go in and play with it Which I think is really cool because I had no idea that this is what meteor showers actually looked like in our solar system I do consulting work for a lot of Like space and aerospace companies this company is interested in the moon So these these are all the places that we've landed on the moon and you can see all the little orbiting particles, but shows you the the lunar orbits of the Apollo astronauts and Of a bunch of other missions there have been we've sent a lot of stuff to the moon And it uses the same orbital physics and the same shaders essentially as What runs the main asteroid visualization? You could also do more normal stuff with this is just like an object That is in WebGL and you can play with it turns out asteroids look a lot like potatoes but and you can you can just like go in and play with it and see it and It turns out there are just rocks in space I send up I send up a lot of I just threw this in here because I send up high altitude Weather balloons and they take pictures, but they also produce tons of data this is the data the flight path of a previous flight and I haven't done that much with it, but you know There are huge opportunities even in like these like weird esoteric things to really take visualization and turn it into something that The normal people understand so With all of this I think that this is just part of a global trend, which is that space is becoming a software problem I eventually Sold a lot of this to to this company planetary resources and went with them to build spacecraft and I learned that a lot of the basic lessons that I took from this and a lot of the tech same techniques that I used for data Apply when you're building actual spacecraft to go out and visit asteroids So things like writing onboard telemetry to a database. I learned that I would not use Mongo on a spacecraft for that but but But I mean, but in all seriousness, I learned ways to Really narrow down the data to store it efficiently to have efficient recall and that sort of thing When you talk back to the earth, you have very very narrow pipe very low bandwidth Right. So you need to be smart about what you transmit and how you compress it And I ran into a similar problems trying to get data over to the client in such a way that wouldn't cause a loading screen for like five minutes and then there's just like spacecraft autonomy, which is that I Came up with a lot of heuristics and a lot of calculations that solve problems that turned out to be Neat problems for a spacecraft to solve and In general, I want to Like I think that space is becoming a software problem But I think everything is becoming a software problem. Everything is becoming a visualization problem So in space the hardware is becoming commoditized, right? And we're gonna we're soon gonna be able to write software without having to think about What platform we're putting it on in space, but just in general in every field, you know, like healthcare Law Oil energy that sort of thing. There are so many opportunities here for software and My experience has been that visualization is at the forefront of it because like I said It's very hard for people to understand the software and understand the data Without being able to actually reach out and touch it, right? And that is the the power of visualization and I think it's really the The visual stuff that will open up a lot of these industries and pave the way for better software solutions so I hope that gives all of you ideas because I think you're already ahead of the curve and I'm excited to see how visualization is really gonna shape the future and also just like Inspire the next generation of people You know who you know for me, I would be excited to see us go to space for you might have another thing that you're really into But but that's why this stuff is exciting. So it's more than just a tool It's a way to make change. Thanks