 Cool. Hi guys. So I'm gonna be giving a spatial talk. It's sort of about GIS and open source GIS sort of as an intro to how you can start playing with some things. So who am I? I'm Pete from Land Information New Zealand. So we do a lot of sort of spatial data and whatnot. We look after properties and titles. We look after crown land. We create hydrographic charts and my team creates topographic maps, which you can see there. So maps of land features, contours, trees, forests, roads, railway tracks, whatnot. In particular, my job is to enhance topographic data and tools. So enhance the tools to make them better at capturing and processing the data, maintaining the data. Cool. And currently I'm doing a lot of development, plugin development for QGIS, the open source GIS. So are we going through just a quick summary of what GIS is for those who don't know? I'll go into QGIS. I'll talk about Python and QGIS. Give you a link to some useful resources to get you started. Some real quick sort of fly-over examples and a little demo of what we've played around with. So have many people just show of hands played with GIS before? Awesome. That's real cool. Because it's obviously the latest thing, because BBC did it, I'll be forcing you to look at some maps of that New Zealand as well. So here we go. Get, decide New Zealand doesn't exist and even in a giant credit card, they can't seem to fit us in. But in my trawling through the internet, I found out sometimes we get placed on twice. So next to Japan there. This one's actually legitimate. The world scrolls around back to us. So we get to be on there twice. And then the next one, I think we've been stalked by our ghost twin. I'm not too sure. Ghost of Future Past, I don't know. Anyway, so it's GIS. So I'll try and breeze through this. So it's just manipulating spatial data, creating, editing, visualising, analysing, publishing. So this is just the QGIS interface, just to give people an idea. That's grabbing how you can grab different types of data. This is displaying the data, the layers. That window in the middle is the map canvas where you can visualise them off the side, some styling tools. This is a cluttered version of it. You just reduce it down to whatever you wanted. That's the processing algorithm. So these do a lot of your heavy lifting. So if you want to combine some layers all the way to maybe you want to grab some digital elevation and run some crazy algorithms to predict flow and flooding and whatnot. Up the top's just a bunch of tools. So selecting, modifying data attributes and whatnot. And these are the plugins. So these are the extensions that you use to add more power into it, to extend it. I'll just go back. So that one down the bottom right, the processing inside there is the processing scripts that I'll talk about later. Cool. And this is just the composer, how you make a map. So you can throw in your title and your scale and your whatnot. Cool. I just was going to explain layers and features to people who hadn't seen them before. So layers is just like a type of data if it's building outlines or it's trees or it's rivers or it's contours. It's one layer, one set of data. And the features are the individual components of that. So for a tree's layer it might be an individual tree and it might have a type of tree in a height or something like that. So why extend to add functionality or modify enhance or to automate? So there's a whole bunch of tools but they might not do exactly what you want or it may be just a real hassle to get it to do it. You only need the first 10% of it or you want exactly those variables every time you just want to be able to hit them. So why open source? I've got a huge list but I'll sort of track it down. So no license over here. It's not locked into proprietary for the transparency, ease of collaboration, access to expertise. So really I think for the programming side that transparency is awesome because you can see all the code that other people have got in their plugins and whatnot. The collaboration is all these sort of set ups already so you can chat to all the core developers and whatnot so you can get some answers straight away. So QGIS itself at the moment the long-term release is 218. It's got 30 core developers, 100 contributors. It's definitely extensible and powerful. It's moving to QGIS 3 shortly which will have Python 3 and QT5. It was two the other day I think and it's not but it'll happen. But if you're going to move into 3, wait for 3.2 because that's going to be the stable build. This is going to have bugs and it's more so they can get out there and people can test it for them. And that'll be next year I believe 3.2. Cool. Here's some more. The first one had the word programs and web pythons so there we go. And the second one obviously New Zealand government's in on the joke because that's their page not found image. So QGIS itself was written in C++ but in 2007 they added Python as a scripting language and so a whole lot of people started adding in plugins and it became a whole lot more popular and it's one of the reasons it's had its rise. So Python is definitely a big part of that. And also 75% of the C++ classes are now available via the PyQGIS API so that's the QGIS API. And because QGIS is built on top of QT you can then use the really nice QT API to grab some more functionality. So sort of yeah just keep moving. So some of the ways you can add it in there. The bottom two I haven't done so I can't tell you much about those but the simple way there's a little console so you can just type in some code or you can copy and paste. The next one was those processing scripts that I sort of pointed to to the side but that's got a real simple way of grabbing a quick UI for you and I'll show you a little bit of that later. And the last one is the plugins themselves. So they take a bit more to set up but there's a plugin to create plugins so it's kind of robots creating robots but I'm sure it's safe. I was going to jump into resources first before I give you the examples because I'm just going to keep saying this is real basic but if you go to the cookbook you can learn a whole lot more of how to do this and some more bits and bobs. So the cookbook is definitely a nice place to stop as an introduction to all the things because I'm not going to cover layer styling but that's in the cookbook and some other bits and pieces. So that's the PyQGIS API. It's written in C++ making it a bit hard but they're supposedly working on producing better documentation for it. QT API, it's a lot nicer. The plugin repels itself just to go and make some code, see what other people have done, how they went about doing things if you want to copy it. A bunch of blogs from the developers themselves. Two of these are Australian which is quite cool because a lot of the stuff happens in Europe so it's nice that we have that sort of connection and we can chat to these guys and if you ask a question they're in our time zones really good too. The first one is actually a link straight to a blog post on speeding up scripts which is really good. So it's more about the layers and the features and how you can iterate through them really cleverly. And down the bottoms just a nice quick tutorial on building a plugin to get you started. And then just some places to ask for help. So the Slack community, there's a QGIS channel and a location area and we're the second largest on that community which is pretty awesome. Maybe it's just us trying to get New Zealand added to Maps, I don't know. Stack exchange obviously with QGIS tag, there's a Gitter and there's some developer mailing lists which is really nice if you want to communicate directly to the devs. So a lot of them look in that so if you had a question someone probably has actually done that directly influence that can answer you. Cool, so once again this was just going to be like a real quick run through just to sort of give you a broad overview of some of what's happening when you interact with it. So first up loading layers, it's based off that GDAL OGR translator library. So there's 17 maybe 80-odd data types that you can grab and take in. So it's kind of nicer than the proprietary in some ways when they're not going to force you to use what they want you to use. These guys have tried to just expose a whole lot of stuff. I was sort of in line with Adamus earlier, with the Python, Python is the sort of the go-to language for spatial development. So if it's not inside of the PyQGIS API or whatever, you could probably find other libraries. I know someone's going to do a talk on GeoPandas. So that's a nice segue, I hope they appreciate that. And I've used it myself, I had a weird thing where I needed to find the closest point on another feature and all I could get through QGIS was closest feature. So I was able to mix some interaction from another library, which was brilliant. So the next part down was adding it to the registry. So the registry is where all the layers are stored from the QGIS session. So that's sort of rendering it. So if you don't want to render it, you don't have to do that if you just want to do some stuff in the background. An active layer, I just wanted to show that because it means you don't have to keep adding in file pass if you're just playing around and testing some stuff. You can go just grabbing the active layer, I want to test this layer, pop in a different layer, does it work with points as well? Oh no, it stuffs up. Okay, that's awesome. So creating a layer, this is just a simple memory layer I've created. So it's set up a vector layer. So obviously there's two types of data, vector versus raster, and this is a particular class QGIS vector layer. It's making a point, it's got a coordinate reference system, so that's just how it takes a glow ellipsoid and puts it on a flat plane. It's got a field, so that's your attributes chord, it's only got a single one called inter IDs, and it's a string, and it's got a name, and it's going to be a memory layer as opposed to save somewhere. So in the cookbook, there's more descriptions of all sorts of things, including using a writer to write directly onto your hard drive. So I thought it'd be good to go through efficient querying. So Nile's blog post takes us a little step further, and you can pull back only certain information when you ask for it, so you can say, oh, if I grab in the geometry, it takes a bit of work, so if I just don't need the geometries, don't do it. But what I'm going to talk about is bounding boxes and whatnot. So a bounding box is the smallest area, smallest rectangle that will enclose your feature, and what that is, it's a really simple geometry, so it's a simple way of doing some quick analysis, and then asking further stuff later on. So it's a way of doing spatial indexes and whatnot, and you can do proper spatial indexes as well. Expressions, Qtas expressions are used throughout Qtas in different places now, especially coming up, and this is just a way of using it. So that one there is just asking for, I think, a field, which is RodeID, which equals a RodeID I pulled from somewhere else, RodeID, anyway. Next one is setting some filter IDs. That'll be a list of IDs I've processed and done some stuff. And down the bottom is how you then grab those features, so using get features from a layer will grab you the features. Checking that request in there will grab you just those ones that I've asked for, so speeding it up a whole lot. Updating layers. So it's a bit of a chunk of one, but it's kind of nice to see interaction between layers and creating something out of it. So once again, the first bit is grabbing every feature, and then I, so for each feature you've got an ID, attributes, and geometry. Here I am grabbing the geometry. Next I'm throwing it in, I'm getting the bounding box of it, throwing that into one of those feature requests we saw before, then asking of another layer, give me all the features that fall within that bounding box, and then because it's a bounding box, so if it was, say, a diagonal line, it would, and they close. My feature might not actually intersect, so I do need to still ask, did it actually intersect? Is it, is it okay? And then I cruise in, and I make up a curious feature, and I throw in some geometry. This time I'm throwing in the point where they intersect, I'm throwing in some attributes, and the zero is the first attribute, and then I'm using the provider to update that memory layer we created earlier. And down the bottom you want to update the extension so it knows now that the, what, where does the geometry encompass? Do I have something else to say? Probably not. It's slightly dodgy in there because it says it's just pure intersection, where if it was multiple intersections you would get back something that it couldn't be used, so you do a little bit of other stuff beforehand. So that's all playing with the features and the layers, and now you can start playing with actual parts of QGIS itself, so using the iFace, the QGIS interface. So I can grab the extent of the layer and move my whole canvas where we were looking at the map previously to that extent. So we were using this, we had a roads maintenance plugin, which people play with some roads done in the background, but at the end of whatever they've done we wanted to zoom them into where they'd done everything, so they could see exactly what they had changed and weren't unaware that they'd accidentally done something off to the side and deleted something and forgotten about it. So just playing with that, so you've got access to the map canvas, you can set the active layer, I forget why we're using that, oh I think just in case they wanted off and looked at some other layers and came back to editing, they didn't start editing the wrong layer, we're pinging them back onto the right one. And here we're starting to use some QT stuff, so I'm using a QT toolbar, so I've made one, I'm adding that into the iFace itself and then I look through all that, there's a digitizing toolbar in QT, I look through the tools that are already there and I'm nicking some of them based on their names and throwing them into my toolbar and then the next slide I'll show you, throwing our own tools in, so that was the idea for the roads maintenance, we kind of wanted to constrain them to a certain set of tools so they didn't go using things they shouldn't have that we hadn't sort of predicted. And yeah, it was nice. So this, we made a snapping tool and they've since added it into QT, so we didn't need to, but anyway that's a lesson learnt. So snapping is where you, if you're going to draw a feature, if you don't have snapping on, if you just rely on human eye, there's pretty much no way you're going to get exactly on and snapping lets you define what it, where you want it to sit exactly on another feature and this was really good for us for our rose database because we wanted to specify exactly what layers they should be snapping to and not snapping to when they turn it on and off. So that's simply, I set up an icon, I set up a new action for my snapping, I get a signal ready which connects to a snap toggle function which would just look as snapping on, if it is turned it off, if it's on, go find those layers to snap to and set the settings up. And then I throw that action into the toolbar, nice and easy. So next one's processing tools. So this is another way of adding that sort of processing power in there. So you can use model builder and QGIS to string some processes up together, this allows me to throw some code in between and be clever about it and add some other stuff in there. So the first bit's just grabbing two layers from my path and then I throw them into a tool, a saga tool called merge layers. So QGIS has leveraged a bunch of algorithms, saga algorithms and grass algorithms to add a whole lot of power to it. So you can go find those and use those. And so it's just what's the tool, what are the layers I'm playing with and adding some variables, I've got no idea what 00 is. And this time I'm saving it as a memory layer and then I've gone back into the registry and I've asked for, grabbed that layer by its name and give it to me as a variable so I can play with it some more. Algolis just lists all the algorithms for you and the help is really good too because you're not going to find a whole lot on them elsewhere. You can also go to the actual individual tools and see what options they give you in their UI to kind of piece together what they might be asking for. So yeah, so earlier on I was talking about the ways of getting it in there is there was that console, there was processing scripts and there was plugins. So this is just a quick look at the processing script. So that double hash notation where it's double hash the variable and then it's going to be a particular type of element, UI element. So this time it's saying it's going to be something that loads in vector layers. So that's based off whatever's in your QGIS session. And I'm going to throw in something that lets them save it somewhere and it's really nice and easy and you can throw in integers and whatever else you want to do. So that can just get you started. You don't have to do any of the other complex stuff of setting it all up. And the bit down below is all in the cookbook and just tells you how to use that. That file write-up obviously just takes in where it's off to. Fields would be a list somewhere else saving the different fields. I've got a lot of type of variables there. It's a point and the CRS I've nicked from the original layer. So that's that coordinate. It's a coordinate reference system again. So plug-in builder. I talked about it. It's that plug-in that makes building a plug-in easy. It just sets it all up for you so you're ready to go. And then the plug-in reload is just nice. So you can make some changes and then quickly reload your plug-in. Otherwise, you have to go in a bunch of menus and figure that all out. I've got mine. I've got a sim link between my GitHub repo wherever the plug-in that I'm playing with directly into the folder for plug-in. So that's nice. I can play there, hit reload and then carry on doing whatever I want to do. QT designer, I think most QGIS packages come with it bundled up already and that's just drag and drop to make a UI real nice and easy so you don't have to do anything. And you can load up other plug-ins ones to see what elements they use to do it. So you can kind of trace it back to see those QT classes that they're playing with, which is really cool. And I love motivational posters obviously. So enjoy your world as long as it's not New Zealand. Go find yourself, just not in New Zealand, you won't be there. She's going places, not New Zealand, just all the other places. Cool. So I just quickly show you two plug-ins. I guess these are simple plug-ins just showing you the idea that you can sort of build on your knowledge, like we did if you want. This one grabs a layer of first off all the map sheets as rectangles. So they look like a grid, but it's actually individual rectangle polygons. And then we just use some cursor tools to you select which particular one you want or you select a bunch of them or deselect a bunch of them. And there's three curses. There's imagery which us and the councils collect. There's satellite imagery and there's a topo map sheet. So this is kind of a little bit redundant now because the web feature service, the way of grabbing layers off the web, has been improved in QGIS, but originally for us it was too slow. But we've carried on using it because now we've got like one source of imagery when people do stuff. So there's someone selected a whole bunch of little images there. Obviously that's way too zoomed out to do anything with and you cruise in. Satellite images has more cloud, but if you've got the two, you can sort of maybe get a spot where there's not cloud, a map, and I can zoom in. And it zooms in even further. So LIQA, a quality assurance plugin we made to compare some new versus old data. So that started off as a bunch of individual processing scripts and then we found a way of throwing them all together as a configuration. So all those map sheets for each of those, there'll be a different set of data. So we can have one configuration run at 80 times or whatever. So that's nice. And then the second part, that's just a menu, don't worry about it, loads it up with this particular symbology. It's pretty utilitarian symbology and often they'd have a map underneath at the exact scale of what the topo would be so they can see what it's going to look like. But the second part is a nice little interface. So instead of going into each layer and modifying those attributes, you've got one little interface to do that. And I can also sort by ID. So I can see that particular row had like 10 problems rather than dealing with one problem at a time and then coming back and going, oh, what, feel like this or something happened here? Did it happen here? And then there's some other little tags, quick ways of checking in tags. And we've got a way of pulling out the problems that they were asking, addressing people they might know something about. And so we can give them a shapefile directly and say, here's the problem and here's the comments and you guys have a look at it for us. And I think we've also got a way of exporting a bunch of stuff out as a CSV. So we've got a record. So if we don't want to keep those layers, we don't have to. Learnings, I guess we're going to make a map about New Zealand, something involving New Zealand, probably throw it on there. Especially if you're from New Zealand, that'd be really nice. Yeah, that's pretty good. So what am I hoping you guys take out of this is that it's easily extended. So that transparency lets you see the code and play with some other people's code and whatnot. There's ways to collaborate and talk. It's easy because it's open source. You can go in at different levels. You can start at just a little console, tiny thing if that's going to help you out. Just copy and paste because that way you don't have to enter the variables every time. Or you could have a couple of little scripts or you can move into more and more elaborate plugins like we've done. There's a huge variety of possibilities. So that was like, was it 75% of the C++ classes available? So there's all sorts of things you can interact with and start using as you get involved in it. So we're using a lot of like signals from if a feature gets added to a layer, we were remembering that and then saving that sort of information somewhere so that we're updating a relational database with that. So we couldn't save the layer directly because we'd pulled this sort of merged layer in but we could know in a clever way what had changed and then split up some really nice postgres functions that would update the correct things based on the knowledge of what had changed and then just clear the layer and reload it from the database. So and definitely the documentation in C++, which is not really great, due to improve we'll see and there's bugs because it's open source and I guess people don't have the same sort of leverage to complain, I've paid $3,000 for this and you've got a whole lot of bugs, can you fix them? But just has got a nice thing recently where the management of it, the governance of it changed so they've got a little kitty that they can put aside to things. So a lot of that's going into like the background maintenance. A lot of the stuff in 3 is actually fixing a whole lot of stuff up in the background that you wouldn't know is dodgy like I think the print composer's got all sorts of things which makes it hard to deal with and so they've completely reworked that and done some clever nice things. And I've obviously talked way faster than I expected so that's awesome but don't tell work I delete in New Zealand because I'm pretty sure that's a sacking offence but there we go. That's my email, that's long-winded link to where I save the slides but you can just ask me for them. Is there any questions? Is anyone using QGIS? Awesome. Did you know there might be Australia, New Zealand, FossilG next year? Let's talk about it and if not we're hoping to get the QGIS conference running again but yeah. I play around in Sublime but we've got Eclipse running as well so that we can use a debugger. So if you want some notes on how to get started in Eclipse I can flick you something because it's it's kind of a mess to get it all set up. For annotation? Ah yeah I just inside of it. It's pretty simple. So you're talking about how I add comments into my code so people know what I've been up to do you mean? Topo itself we have individual points of the labels and whatnot. I think it came out USGS tried to go fully automated and they had a huge issue with that and people a lot of complaints because the labeling was just automated and just decided to do whatever it felt like but there's a nice and the QGIS coming up maybe in 218 where you can look at visualise what the algorithms had to choose from where the label should go to help you choose what way of labeling you want to use so it'll say oh you're based on this and you can see there's a whole cluster of them over here why was it over there and use that to help decision which algorithm to use for your labeling? Yeah yeah yeah yeah yeah yeah I'm not sure because the back end still sort of lamps and something else puts it all together in ways that I'm not allowed to touch so I'm not sure exactly how it does it but yeah it'd be nice if that was sort of in there a bit more yeah I can't say any more on that unfortunately yeah you guys have all found the LDS and know where you can grab our data for free so I'm not just spying on you and nicking with your houses I'm giving it back to you sort of oh guys I'm gonna let you go early here we go