 Audio's good. It's on. I'm going to talk about... I'll first give a bit of an introduction, talk about some tips when installing GDAL, how to use it with Python, some random tips, and then a bit about how to use these cloud optimized and how to make cloud optimized geotips, and also where to go for some help. So, there's a quick introduction about myself. I'm the team lead for the pipeline team in Berlin for Plant Labs Germany. I've been there since 2014, working for Blackbridge, and we were acquired by Planet. I am Belgian, but I was raised in Florida, so that explains the accent a bit. This is my fourth year attending FOSSTEM in the first time speaking. So a bit about GDAL. Who here has used GDAL or is using GDAL already? All right, good. So I'll just quickly go over this. GDAL is the Geospatial Data Abstraction Library. It was initially created by Frank Warmerdam. He's actually my colleague back in San Francisco. The project is now managed by OSGO. It's a CEC++ library. It can read a whole bunch of vector, so shapes and raster or image formats. There's a whole bunch, over 150 rasters and 90 vector formats. It's, as you heard, it's kind of like the base of everything that basically every geospatial project uses this internally. So something like most common GDAL utilities are like GDAL Warp for, if you're doing mosaic and reprojection and warping, GDAL Translate to change from one raster format to another, and then the vector equivalent of OGR to OGR, and then GDAL Info where you can not just get the information, but you can also generate a whole bunch of statistics out of that and get it in a whole bunch of different formats. So on the installation side, on Mac it's pretty easy. If you use Homebrew, which you should, there's the OSGO for Mac tap that you can use. It's also managed by OSGO, and they're always looking for some help there. And then you just install the GDAL 2 and GDAL 2 Python bindings. On Ubuntu, there's a Ubuntu GAS PPA that has GDAL there, but there's also a cross-platform way of doing this. So if you're already using Anaconda, you basically just kind of install with GDAL, and that kind of sets up a lot of the dependencies that you need there. But I don't really recommend doing either of those because you get a lot of dependency problems. So on the planet, we use a lot of VMs and containers to isolate all the system dependencies to basically make it so you can just rebuild the whole system and get back to a working configuration. You don't have to worry about upgrading something in the package that you might be using and breaking your whole GDAL installation. And it also kind of lets you avoid having to use the Python virtual lens where you also get some problems if you're trying to do multiple versions of GDAL in different virtual environments that just gets very messy. And basically you can just put the setup instructions in Dockerfile or vagraphile or use an existing setup to kind of get started and then just build up there the dependencies you might need. So working in Python, GDAL 2 gave us a lot of powerful Python bindings. So you don't need to use the command line call anymore. You can just basically put the command line arguments in the Python call to GDAL warp, for instance. And there's also... I was reading a bit about this at the OGR to OGR, for instance. It's not called OGR to OGR in the Python bindings, but it's GDAL vector translate. There's a whole RFC about how this was set up, but yeah, it makes it a lot easier to have that and you don't have to worry about the whole sub-process call. Also, there's a very good NumPy integration. So if you are doing things like windowing, you can actually just read the array and specify the offset that you want and then you get a whole NumPy array. You can do all the raster computations that way as well. Also, if you are just kind of trying to do some small things, you don't actually have to use the GDAL Python bindings directly. There's a few good libraries that we use as well. Rasterio or Rasterio from Mapbox. It's basically a Python version of GDAL. So it's all GDAL under the hood, but it's basically just a Python API that's Pythonic. And it's a lot easier to use when you're wanting to open. You can do a lot with that for vectors and shapes. Shapely and Fiona from the same person. It's also Pythonic. Shapely only has Geo's dependency, if I remember correctly. And Fiona has the full GDAL OGR dependency. And of course, they're also a lot easier to use. You can load GeoJSONs and WKTs and all this stuff and do your validation and intersection, error calculation and all these other transformations. So some other tips. What I really recommend is that if you are wanting to learn a lot more how to use GDAL, especially the C++ API, there's a lot of the utilities that are some of them that I mentioned in the previous slides, that you can just read their source code, try to understand how they work. And then that gives us basically a bit better than the documentation on how to use things, especially if you're looking at GDAL Warp that shows you how to actually do all the things that it does under the hood. As well as some of them that are Python scripts. So the GDAL calc, for instance, is GDAL calc.py. That does all the loading into the NumPy array and does the, for instance, if you want to do like a difference, then you can just look at how it extracts the operations and executes them. So that's basically my first tip on that. Also, if you're working with geometries that you get from random places or that you're generating, sometimes, especially around the dateline, you get some weird issues where the geometry doesn't validate anymore. And basically you can just apply a zero buffer on it and then that OGR kind of magically makes it a valid geometry. So really helpful whenever you're having weird problems with geometries, just apply a buffer to it with actually no buffer, but that makes it work. There's also VRTs. So VRTs are virtual data sets and you can basically treat a whole bunch of imagery tiles as like one big image. You can also basically take a big image and chunk it up into smaller files and then join it all back together in this VRT which just references all the images and where they should be if they were one raster. Oh, sorry, screen saver. So VRTs are just basically XML files that describe all the operations and they do need everything to be in one common projection. So just keep that in mind if you are trying to do that. The other really cool thing is that a lot of imagery providers are now making cloud-optimized geotifs. So what's cloud-optimized geotifs? So these are geotifs that work a lot better for cloud processing. So essentially what they do is they have like the header information in the very beginning and it's a tiled image. So usually 256 by 256 or 512 by 512 tiles. And they also have all the overviews built in. Sorry. By doing this you can essentially just do a range request so you can get the header or GDOL can basically get the header and then figure out what byte range the imagery that you need if you're doing a window or where the overviews are if you're just getting some if you just need the overview of the image and then it downloads only the data that it actually needs. So if you're using the VSI Curl Driver in GDOL you can basically just put the URL in there and of the image. If you're getting it from OpenAreaMap, DigitalGlo, Planet, the Landsat 8 bucket on AWS is also in this format and there's a whole bunch more. COGO.org has like a lot more information about this. But it can really save on the download times and not having to download the whole raster just to get the small piece that you need. If you are generating image and you want this to be in the Cloud Optimize GeoTip format there's basically two things that you need to do. If you don't already have overviews built then you would just use GDOL at O, add overviews to build the overviews. So basically just put your image where it says in.tip and run this. It should be pretty quick and that actually modifies the image and puts it in directly. If you are doing an external overview it's not going to work. The second thing is to then do GDOL Translate where you basically tell it that you want it to be tiled and you want to copy the overviews that were in there and then this recommends to use LZW compression. You can also use Deflate but there are some issues. Some things aren't fully compatible with Deflate. And the other important thing is that when you are hosting it the server needs to support the range request. So HTTP 1.1 if you are just putting it on S3 or Google Cloud or Azure they all support this byte serving as it's also called. To use the Cloud Optimize UTIF basically if you are using the VSI Curl driver that's all under the hood handle magically. Some examples are on the GDOL Wiki. There's also a good basically Jupyter Notebook that describes that and it's linked here. I also have the presentation uploaded on the Boston website so if you look up the page for this presentation you'll see the presentation linked there as a PDF. So here's an example of using the VSI Curl driver. As you can see it's basically just doing the slash VSI Curl slash the URL. So this one is a Landsat 8 image on the Landsat 8.0 S bucket which is a great open data source if you want to start playing around with raster and need some imagery. So where to get help? This is also a very important topic but the GIS stack exchange which I think I saw a shirt there that's a really good place. You'll find a lot of people from the development community Frank and I both answer questions on there. There's a GDOL IRC channel as well and the GDOL Dev mailing list is a great way to get answers. GDOL.org the main website has the wiki and everything else so you can really get all the documentation on there. But another great resource is there's a series of Medium posts by a colleague Rob Simon where he goes over really introduction to GDOL a bit about the history of projections and what not. So it's a really good read. It's like three part series. There's also the Python GDOL OGR cookbook. I've used it quite a lot when starting out with GDOL to figure out how to do things in Python doing that and reading the GDOL utility code as well. Those two things were important. I also made one of these awesome geospatial lists on GitHub so a bit of shameless self-promotion of that there. So basically these people helped out with the presentation mostly some of my colleagues and Evan who's basically one of the main developers of GDOL now and all the imagery here is from my company so all the beautiful pictures you saw on the slides. We also have an open data set over California so it's Creative Commons license imagery from like two weeks ago or something like that or you can basically just go on the plan website and make an account and get some data if you want to have some more free data sets that you want to use for your stuff. And that's it. This picture of my cat. Thank you. And you can reach me on Twitter, GitHub. You can send me an email and that's it. Thank you. Any questions? Yes. No, this has been supported in GDOL for a while actually. So the cloud-automized geotips are nothing really new. It's more like leveraging existing features of GDOL in kind of like a bit more standardized way. So the question is when you have all these different libraries and they all recommend to use Docker but they have their own Docker containers and you want to basically put them together. We kind of have had the similar problem of doing that. We rely heavily on GDOL but of course there's other libraries that you need to use. And basically one of the things you can do is to kind of try to glue these different Docker files together so to like look into it and build it that way. We also rely heavily on VMs at Planet and so basically we have like a big script that as you start trying to integrate these different projects you kind of build up a VM and if you start going down a really weird path then you can kind of still recreate the whole VM back out. So it is a lot of work unfortunately but the VM makes it really like sure that you can always back out and kind of start over and that's kind of like your source of truth. If it works on a fresh VM then you just use that but if you're like doing something weird then you can... So I don't think QGIS fully supports it yet. I think in QGIS 3 which should be released sometime soon they will have some of that support but yeah it's not fully supported in QGIS yet. So the question was are Cloud Optimized GeoTips supported in QGIS? Yeah, once it's supported then basically it's streaming the data. Right. And especially if you're viewing it at a high zoom level for instance it will just use the overviews instead of downloading the full raster. Any other questions? Alright, thank you.