 Thank you, Irene. It's definitely good to be back here. So this is the Canyonlands in southeastern Utah, and it is the last place in the United States that was mapped in detail. Actually, a few separate attempts were made to do that mapping. In 1869 and 1871, John Wesley Powell led an expedition down the Green and Colorado rivers. A scientific expedition to map and explore this area, mostly geologists and surveyors. But they were limited to traveling along the rivers because the area is so rugged and inhospitable and remote. And even when they could clamber off the river itself and up onto the canyon edges, you know, this was their view. So hoodoos and mesas, but they really didn't get a good perspective, which you need to build a map. So in 1885, the USGS produced this. So it was the first of the official US government maps of the desert southwest. And this went through five different editions over 65 years, with only the only difference being the typography, so the labeling of features. And it wasn't until the mid-1950s that we got these very detailed topographic maps that we're familiar with today. And that really required the development of some new technologies, and it also was forced on by a uranium boom. So prospectors actually crawled over this entire area looking for material to fuel the US's nuclear program. And the technologies required were aircraft, big heavy aircraft that could carry these huge mapping cameras, the cameras themselves that could take wide area, high contrast, high resolution photographs, and then the science of photogrammetry. And what that is, is it allows, by looking at overlapping photographs, it allows you to precisely plot where each feature is in relation to the other photographs and eventually onto the surface of the earth. And from those, we have the wonderful high res USGS 1 to 24,000 maps with these great, highly detailed topographic contours. So that took 65 years. In the intervening 65 years, we have gone from not being able to map a place at all to being able to see it almost every day. So this is a planet image from March 12th and another one from April 10th. And in addition to this color imagery, there's also an amazing amount of other types of data. So the entire wealth of US government satellite data, plus the data exhaust from our phones, geologic maps, everything. So there's dozens of different data sets of every place on earth, and they're all useful separately, but they become incredibly powerful when you start combining these different types of data sets together. But then that becomes a challenge because they're all in different formats. They all might be in different map projections. So how do you really stitch everything together to make a greater whole? And one answer to that is GDAL. So GDAL is the Geospatial Data Abstraction Library. And it is, in fact, pronounced GDAL, not Goudal. Maybe back in the old days it was called Goudal, but then Frank went to work for Google and there was Google Goudal, maybe a little confusion. So it's GDAL. And it's a library that basically allows you to manipulate geospatial data, so data that is tied to a specific place on earth. And it's great because it's fast, it's free, and it's flexible. So it supports a vast number of data formats. It runs on multiple platforms. It's automatable, both through bash, any command line scripting, and then it has powerful Python bindings, so you can bring things in and then do analysis and numpy and things like that. Unfortunately, it's really kind of deep and obscure. So it's inconsistently documented. A lot of times, if you read the GDAL page on it on a certain function, it'll be like, well, the widget makes widgets. The widget function makes widgets, and that's not super useful. It also requires a thorough knowledge of map projections and data formats, and a certain amount of being able to manage, like, installing things in UNIX and managing the dependencies and things like that. So hopefully I can show why GDAL is cool and then give you some of the tools so that you can then start to learn it on your own afterwards and, like, even know the right terms to Google. So I'll just start out with some first steps showing some very basic commands that are essentially the GDAL's hello world, and then also introduce a few of the more obscure strings that you might find in options, then talk about map projections and how they're necessary because the earth is round, not flat, despite what you might think if you Google my name on YouTube. Then cover some functions, GDAL warp, some of the more esoteric options for how to control that, and then the concept of a virtual file, and then go into local map projections. So it turns out earth not only isn't flat, it's not perfectly round either, so we have all of these mathematical constructs to sort of model that non-sphericalness, and so that matters a little bit for local projections, and then some techniques for making a large scale, which means highly detailed maps from either a big data set or many smaller data sets put together. And then because the earth isn't static, we can also manipulate satellite data with GDAL, and so I'll talk about reordering and combining bands, scaling data, and then getting data from different formats to play well together. So for installing GDAL, Kanda is a good route if you're already using it. Be aware though that it installs a slightly older version that won't read JPEG 2000, so you might need to use GDAL Forge to get GDAL 2.1.3 going. If you're on OS 10, you can use King Chaos. It sounds like a mid-80s Hector Collective passing out floppy disks with Commodore 64 games on it, but it's these really nicely packaged binaries of GDAL and some of the supporting libraries and some of the things like QJS that are built on GDAL. Windows, you can go through the UCLA, has a really good demo and tutorial of how to install it, and then if you're running Linux, you know way more about this than I do, and you're on your own. So once you have GDAL installed, and I'll give the URL in a second, but I've written a blog that goes in detail in every single step and also has demo data sets that you can download, but if you download a GeoTiff, go ahead and run a command called GDAL Info, and the basic structure is you specify a command, you point it at a data set, and then you give it options, and so you'll run GDAL Info and you get back this big long string of text, but the important things for us is that a GeoTiff is just a normal Tiff, but it's got another header on it which has the information so that we can precisely place every single pixel on an image to a precise coordinate on the Earth's surface, and so part of that is the coordinate system. You'll also see corner coordinates, so you can sort of know where the corners are, the edges, and place the entire image in context, and then it'll give you information about each band individually, and that becomes really important when we start working with the satellite data. So one of the easiest functions you can do with GDAL is GDAL Translate, and that's just from going from hundreds of different data formats into whatever data format you want, so for making an image of a map for this slideshow, you can say GDAL Translate is the command, and then specify an output format, it's simply dash OF. There's a dash CO, a CO is a creation option, and this is one of those kind of obscure, really difficult parts of GDAL in that they're specific to each individual file format, so they're not really documented all in one place, they're spreaded all over the place, but quality is the JPEG quality from 0 to 100, just set that to 90. Outsize in this case is in pixels, so 1920, great for HD display, and then that's for X width, but I didn't specify height at all, so GDAL will actually keep the aspect ratio if you set one or the other variable to be 0, and so that's sort of an important concept about GDAL, is that it tries to do a lot of the work for you, which can be super convenient when it works right, but occasionally will break spectacularly, and trying to figure out exactly what went wrong can be quite tricky. Then I specify a resampling method, bilinear is a nice way to do smooth, it will default to nearest neighbor, which gives stair steps and looks pretty terrible, so it's a good thing to keep in mind, and then just an input and output file, and you end up with this map perfectly sized for the screen. So far, I haven't really shown anything that's particularly unique to GDAL, you could do this in pillow, you could do it in image magic, but as I said before, the earth isn't flat, and so the power of GDAL is really being able to work with things that are directly tied to real positions on a 3D surface. But the earth looks flat from ground level, it looks flat from an airliner or from the summit of Mount Everest, and it's not until you get to about 70, 75,000 feet that you can begin to see that curve, and so this is from a balloon expedition launched from South Dakota in 1935 from just above 72,000 feet, and it documents the first time humans ever saw the curvature of the earth, and it's not until you get to about a million miles away that you see the earth as this perfect disc against a black sky, and from this far you're actually getting very close to that orthographic projection that Mike was showing when he's spinning his globe, where it's basically seeing everything from 90 north to 90 south projected onto a perfect sphere. So each of these views is in effect a different map projection because you're looking at the earth from a certain perspective in space and then have a 3D object that then you're representing in 2D, so the 2D of a screen. Something we've known about as a species and a culture for a long time, this is the first globe. It was published in 1492 before Columbus had returned from his expedition, and so you can see that each of these sort of laws and shape things is what's called the gore, and you would cut them out and then paste them onto a spherical substrate to make the globe, and so cartographers really have a challenge because there is no perfect representation of the 3D earth in 2D, and so a well-designed map is going to be solving a particular problem, and for each different problem there's a specific map projection, and GDAL is this great tool for actually moving between the different projections. So if we start with the natural earth, which is this incredible free data set, they have different scales of raster data in a bunch of different styles. This one's really good for making base maps. They also have some vector data, but it's in what's known as a cylindrical equirectangular, so it's just an even grid of latitude and longitude, so it comes out as this 2 to 1 ratio, and then in GDAL we would invoke a function called GDAL warp to do the map projection, and so some of the options are similar to what you would see in GDAL translate, and there is some consistency between commands. Unfortunately, the syntax actually is quite different from command to command to command, so you really have to pay attention and sometimes look it up, and then at the bottom is a link to the blog series, where again you can go through and I have examples and step through in more detail what each of these things is doing. The next is specifying a target spatial reference system, and so that's sort of the mathematical magic that allows each pixel to be precisely placed, and rather than going into detail about every single component of this, I will just point you to spatialreference.org where you would basically look up a map projection, and it gives you an option of formats, several options for formats, and if you just click on the Progsh4 link, you'll get something that you then paste in single quotes into that target spatial reference system and do the map projection from there. There are some map projections that get super complex mathematically, where you can go in one direction but not the other, and GDAL really wants to be able to go in both ways, so there are some map projections that it just doesn't support, and somewhat conveniently, most of them are not supported by Progsh4, so if you click on that link and get a blank, then you probably can't do it in GDAL, or at least not without a lot of extra work. One of the other interesting flags that I've set in this command is DST alpha, and what that does is all of the areas that are off the map, because we're going from a rectangular map to a elliptical map, get given an alpha channel, so it'll default to just filling it with black, but it's really nice to have an alpha so you can overlay it on anything. And this next one is what's called a warp option, and these are starting to really get into the nitty gritty of GDAL, so it's manipulating how that very convoluted trigonometry is from going one map projection to the other is working. In this case, it's actually giving an extra border around the edges of the map, so we can calculate the edges of the destination map, which again isn't a perfect rectangle anymore. And this is where, again, I was saying GDAL does a lot of guesswork for you, but it can go spectacularly wrong, and so what the source extra is doing is just, it's giving you a little extra boundary, so if you run something and it doesn't look right or there's some missing information on the output, try adding this line or looking up some of the other options that you have. And then finally, I just give in an out file, and I get this as the output. And this is mole-wide, it's what's called an equal area projection, and so it's great for thematic maps, so things like temperature, rainfall, and population density where you really want them like per square kilometer, and if you were saying, you know, like showing change in temperature in Greenland, it would give you a disproportionate view of how the world was changing. But like most global projections, the poles are highly distorted and shoved off to the edges of the map, so what if you wanted a projection that focused right on one of the poles? So there's another one called polar stereographic that's optimized for that, but it blows up at the opposite pole, so you need to pre-prepare your data set beforehand. And so what I've done here is do a two-step process instead of a one-step process where I'm chopping out the bottom 30 degrees of the globe, so basically from the tip of the Antarctic Peninsula south, and then doing the next step of a GDAL warp, and I'm doing that with what's called a VRT, which is a virtual file. Instead of being a full-blown TIF with all of your information in it, it's a little tiny text file that is telling GDAL how to do that transformation. And so if you're doing a huge bulk job, it's really useful to do things in terms of these VRTs instead of TIFs to save time and space and processing power. And the next important bit was setting the projection window, so negative 180, and you know, that's just defining the edges, and so I get a little strip of the south pole and then do the projection. And the one interesting thing about how I did the projection versus the mole-wide is I referred to an EPSG code, and so there's an Institute of Petroleum Engineers who have predefined a ton of different projections for like sort of well-known areas, and so instead of that long string you can just enter in a numerical code, and then you can look those up at EPSG.io. Run the command and I get this, which is lacking something, and that's because I didn't include the source extra command, and if you do that it makes sure to calculate the boundaries correctly. As I mentioned earlier, not only is the earth not flat, it's not precisely round either. There's a bulge in the middle. And in the 19th century, well before John Wesley Powell set out to map the canyons of the Colorado, the East India Company did this incredibly detailed 70 or 80 year long expedition to survey India. And they weren't setting out necessarily to make a highly detailed map of India, and they weren't planning on determining the height of the highest mountains in the world to within 30 feet from 100 miles away, although they did both of those things. They were really trying to determine the shape of the earth. The earth has an equatorial bulge and in fact it's pronounced enough that if we measured elevation in terms of distance from the center of the earth, Chimborazo and Ecuador, which is I think a quite modest 18,000 feet, would be the tallest mountain in the world instead of Mount Everest. So these effects don't really matter much at a global scale or even a continental scale, but they matter quite a bit if you're trying to connect two railroads in the center of a continent or drill a well precisely where you have a permit. And this kind of shows up in some really complicated concepts in these projection strings that you'll see. You don't really necessarily need to internalize all this, but just be aware that they exist and be aware that if you are in a certain datum, it's usually best to stay in that datum instead of trying to move it to another one because that may be an undefined problem or might require a ton of coding to get it to work right. So basically there's an ellipsoid or a spheroid and this is an idealized mathematical shape. There's a geoid which is an approximate shape which is equivalent to the mean sea level and then there's a datum which sort of ties these two things together. And so some common datums you might see NAD 83 or 27 and then there's individual ones for Great Britain and Europe. But for a lot of satellite data, it's WGS 84 and if you can find data that's in this map projection or in that uses this datum which is confusing the ultimat projection, use it because it'll solve a lot of it will just prevent a lot of nasty problems. So how would we go about making a global map? You can either a local map, you can either start big and go small or start small and go big. We'll start with the former. This is using a continental scale map of the U.S., a variant on the natural earth data set. And then we're going to crop that down and re-project it and move from a conus alberus projection to what's called the Lambert conformal conic. As I mentioned, there's all these different map projections optimized for different things. You really have four options, area, angle, distance, and direction. And no map can do all four of them perfectly. And so there's various trade-offs. And so to clip a larger map, you can run this. Important part from what I've shown you before is again using a Proj4 string to specify the spatial reference system. But instead of pulling it from a library, I used a really handy tool called projection wizard.org. You go to that site, you drag a box around the area you want, you pick the parameter you want to optimize, and then it'll spit out this string for you. And then you can just paste it into GDAL warp. And then also you need to define the extents. So these are the edges of the map. I've done that in lat long. Generally speaking, GDAL will default to the coordinate system of the destination. That can be really tricky because it's often defined in meters versus some possibly arbitrary centroid. So if you specify your own spatial reference system for those target extents, in this case, EPSG 4326, which is that same lat long, you can just type in the latitude and longitude corners. And then from that continental scale map, you get this reasonably high-resolution map of the Canyonlands again. But say we wanted a super high-res map. We can turn to another dataset. This example is from Nape data. It's aerial photography. It's done like every one or two or three years over the entire U.S. one meter per pixel or higher, but it comes in these little tiny tiles. So we're going to use a new program called GDAL merge. This is actually, was originally a demo program, but it became widely used and is super, it's just commonplace, so it got kept in the code base, but unfortunately it's like all this syntax is inconsistent with all the other GDAL syntax. So again, it's just something that you should really keep, keep aware of. So you can just run this and you specify an output file, and then it's really nice because you can just give it a wild card. So I'm just going start.tif, and it takes every tip from my directory and stitches it into a nice local map. So it's great just because of how simple it is relative to the rest of GDAL. And then what I've shown you so far is all pretty much static maps. A map is an idealized representation, but the Earth is always changing, so we can turn to satellite data as this extremely powerful way of looking at Earth as it evolves. And you might have heard California had this extremely wet winter. So this was what the Carrizo plane, which is where the San Andreas runs north of Los Angeles, you can see that horizontal, that straight line running right through the middle of the image. This is what the area looked like in the winter. And then this is what it looked like at the end of March in the spring. So all this rain led to this incredible profusion of wildflowers. You've probably seen some pictures, but satellite data is another way of looking at how widespread this phenomenon really was. And so GDAL is once again a really good tool for working with SATDAT, specifically because satellite data is strictly RGB, so it needs to be massaged to be useful. And plugging my blog endlessly, I have links to a sample data set on the blog, but you can also sign up for my company's Open California data set, where you can get all the data of California that's more than two weeks old under a Creative Commons license. So if you download some of that Rapid Eye imagery and you open it up in a TIF viewer, it would look like this. So not only is it super dark, the colors are also kind of wrong. And that's because the satellite data is packaged in sort of a scientifically-sensical way and not a otherwise-sensical way. So it's blue, green, red, red edge, near-infrared instead of red, green, blue. So it's got not only visible channels, but also these extra bands that are useful for doing specific measurements. So in GDAL, we need to reorder, or to view it at all, we need to reorder it, and GDAL provides a handy way of doing that. We go back to GDAL Translate. And most of the command is the same, but we specify each band individually with a flag. It's just dash B. And then it goes in order. So band three is red, and I'm putting that in the first channel. Band two is green, and it stays where it is in the second channel. And then band one is blue, and it's getting moved to the third channel. And then one other thing I need to specify, another of these really obscure creation options, photometric. And this sets the image to be interpreted as a full-color image rather than separate grayscale bands. So if you were just to load this in a photoshop without setting this flag, it would say gray and then alpha one and then alpha two instead of giving you an RGB image. We do that. It's still dark. Scientific data is linear. Our sensors are nonlinear. And so we want to do a little bit of scaling on the data to make it approximate how we see. We can do that, again, with GDAL Translate, but I've run GDAL info first to get the image statistics that we need to then know the endpoints. And then after that, I can add this dash scale flag. And what that's doing is it's taking the minimum and maximum extents and then stretching them into the full range of a 16-bit file, sort of from zero to 65,000. And then I also do an additional step of scaling exponentially, again, because we don't see linearly, we have this curve that sort of compresses black and expands white lighter images, lighter information. And if we do that, we get a much more viewable picture. Satellite data color correction is like a whole another thing. I could go on for days and days about that. But this is a really good first approximation. So what would happen if I wanted to compare this year to last year to get a sense of, you know, how extraordinary of an event this is? And I went back, checked for rapid eye data, and there was none that wasn't cloudy. But there was some good Landsat data. Another data set, this one's free, provided by the U.S. government. You can get it off of Amazon. And I downloaded the data for a day last spring. And unlike rapid eye, which is multiple bands in one file, in Landsat, every single band is its own file. And so we need to use a slightly different technique to do the combination. Again, using GDAL merge and using the separate flag to specify the red, the green, and the blue bands. And you would probably have to look this up independently for every satellite because every satellite was made by a different team of engineers that wanted to precisely solve their own problem. So they're all different. It's totally nightmarish. But for Landsat, four, three, and two are red, green, and blue. And you just combine them into a single file. And then this is the color corrected version of this. And you'll notice that it's a much wider area. And you can't tell, but it's also a lower resolution. So how would we match this data set with the rapid eye, which comes in a little tiny tile? We use a couple other commands. So GDAL tindex is actually walking around the corner of the file and then creating a second vector file called the shape file, which is a really common spatial data set. But instead of storing things as pixels, it's storing things as points with a latitude and longitude. And you do that with the tindex and create the shape. And then you point GDAL at the shape file to create a cut line. So it's basically making a box and superimposing it on that larger image. And then just tell it to do the cropping. So the crop function and then the file you're cropping and then the output. And the final little bit of GDAL magic is I'm setting this target resolution with TR. And this is not in pixels. This is actually in real world units. So rapid eye data is five meters, each pixel is five by five meters. And so I just set that flag. And then it resizes the Landsat data to match. So this is March 25th of 2016, March 31st of 2017. And you can see 2016 was a fairly wet winter but not extraordinary. 2017 was like biblical. And so we got much greener and these like incredible colors over the place. So hopefully I've given you a fairly good introduction and maybe comfortable enough to start looking into this on your own. Some functions I'd encourage looking at are GDAL, DEM and OGR to OGR. There's some ways to rapid rasterize vector data. You can pan sharpen low resolution data with high resolution data. And then you can do band math between like the red and near infrared bands like addition, multiplication, subtraction to do some fairly sophisticated analysis purely in GDAL. References I recommend including Derek Watkins GDAL cheat sheet which is a good thing to just keep open because it has specific commands and then syntax. And then I'd like to thank all these people for proofreading and sitting down and showing me source code and all sorts of wonderful stuff. And as a teaser, this is the Stamen watercolor map which was pulled out of web tiles and into a geotip with a single command. Thanks.