 Okay, thanks for coming guys, I've been given the thumbs up so I can get started now. So today's talk that I'm giving you is just about mapping and geolocation in Drupal. I'm actually just mainly going through the Budget Victoria website that Solicitor Digital built back in April. So a little bit about me, I actually got our designer stupidly to take a look at this presentation and all of a sudden this very impressive slide turned up. So yeah, I've been working with Drupal for about eight years now and you can see that it looks like I've done lots of really impressive things. So a little bit about Solicitor Digital, an enterprise digital agency in Victoria. I've been around for about 12 years, mainly Acquia certified developers. One of the GovCMS partners, a lot of you have probably heard about GovCMS while you've been here and I'm the GovCMS trainer as well, so if anyone's in Canberra and wants some GovCMS training, I can come and see my face again. And here are a few of the Drupal and government projects that we've worked on in the past. Okay, so let's go on to some of the content. So in general, why do we need maps on websites? I don't think many people are building websites without a map these days. Basically everyone wants to know where are you? Where is the stuff that you're actually selling or doing? Also what things are actually around the location that whatever it is you are doing, what things are around there? Obviously responsive is one of the things you really need to think about when you're building a map. There's a lot of difficulties with mapping once you go down to a mobile size in particular. And so essentially the main thing you have a map for is to show people where they can actually get what they want. So geolocation I think is probably the other half of that story. Obviously a lot of traffic is actually moving to mobile, so people actually want to know what's relevant to them as they're actually walking around and doing things. So I mean one of the things I do because I travel a bit for work, as soon as I get somewhere I'm usually looking up the website or whoever it is I'm going to see, finding out where I can get coffee somewhere around there before I get to the meeting and how long that's going to take me from wherever I'm staying because those are kind of the really important things. One of the other things that people don't always think about with geolocation is taking that outside of mapping. We've got one of our large international corporate clients. They don't actually have any maps at all on their site, so they're using geolocation to actually serve up completely separate content. So people come in from a different country, so France or even Northern Ireland and they've actually landed on the UK site, we just redirect them back to the appropriate site for their location. So I mean that's one of the fairly basic things that you can do with geolocation, but it's still one of the uses that is quite important. One of the other things that you can do is actually use say Google Analytics events as well with your geolocation and start really pinpointing where the people are who are actually using your website and what those people are doing. So geolocation kind of goes a long way outside of just what's in a map. Actually, yeah, one other thing I think, I'm not sure if Accu has done a presentation on their Lyft product as well, but just around personalization. You can actually do a lot of personalization based on locations as well. So if someone's actually viewing your website in Melbourne, you can show sales for your Melbourne stores or something instead. So it should be quite easy and a lot of the time it kind of is depending on exactly what your requirements are. There are a number of modules and libraries out there. Lyft is probably one of the better ones that I've used. It's really great at actually putting a Lyft.js map on your site. The problem is it doesn't actually do the geolocation part of it. So in order to do that, you need to use another module called the IP geolocations views and maps that integrates in with views, integrates in with the Lyft module. All of a sudden, that's great. You've got geolocation, you've got a map, and everything's working quite well. Problem with that is the way that it uses session variables, it actually kills off your cache, and all of a sudden every single page you're serving up is authenticated and if you're trying to run varnish or other caching tools in front of your website, they're no longer in the picture. So that becomes a bit of a problem. There is a module as well for Google Maps. Once again, kind of similar story, it provides that mapping for you. You don't get all of the geolocation and things like that straight out of the box. So once again, there is actually a lot more work that you need to do outside of just clicking install the module. Instead of Drupal, there are other tools like national maps. We do quite a bit of work with government. So national maps actually integrates with data.gov.au. And it provides what is a very lightweight map, and it's a JavaScript library that you can just implement fairly easily. It actually delivers WMS tiles up, so it's actually a, it's very kind of light in terms of the download. But the problem is every time you want to click on something or do anything, it's actually sending a request back to the server again. So in terms of overall performance, it kind of is a little bit slow. So some of the requirements we had, whether we actually build an interactive map on the site, it needed to show all of the different projects. So it wasn't just a simple map, we had to implement a large number of pins. We actually had to use clustering as well on those pins and show a number of different regions. So unfortunately a lot of the modules that actually kind of come out of the box with Drupal weren't actually going to do what it was that we needed. And another issue we were facing is we were using GovCMS. GovCMS is a great tool for government. One of the problems is you can't actually use any extra module code. So you can't install new modules. You don't have access to any custom code. So you really have to use what is kind of in the box. So I'm just going to show you a couple of example sites of what we were trying to build. So regions.vic.gov.au was actually the site that we were being asked to model this new website off. And you can see that what we've ended up with is actually fairly close. The design is actually very similar. And one of the main components of this site is the explore the budget map. So you can see here what we've needed to implement was a list of all of the budget projects right across the state of Victoria. We've got some filters here that filter by local government area or local council. Those are then split up into what they've called priorities, which are just basically categories that are linked to each of these projects. And you can actually search there for our specific projects. I'm going to remove one of those filters. There we go. So we've got a number of pins that actually appear on the map. So you can see we're using clustering to actually show that there are five pins in this area rather than kind of dotting five pins right next to each other and making them difficult to click on. You can kind of click on those maps there, on those kind of areas there. And as we zoom in, we actually get to see the rest of these pins. So if we click on one of the projects and go to more info, kind of these pop-ups appear over the top of the map. Okay, how's it working? So I'm basically just showing you this is what we need to implement. And a couple of things we'll get into how it works in just a minute. I think it might be about five minutes in, so. Okay, so I'm having some mouse issues at the moment. I've just got some odd things happening with my mouse, sorry about that. Okay, beautiful. Back to where I was, so yeah, as I mentioned, we're using Cubs CMS. So one of the issues we had there was there are no mapping modules in the Cubs CMS platform. So we didn't have access to the leaflet module or anything else that could kind of help us along because there's no custom code. Once again, we're kind of a little bit hamstrung with that. It was also a project with very tight timelines. I think we started off with about 12 weeks initially. That kind of got shrunk down to 10 weeks. One of the problems we had was the federal government decided to bring the budget a week forward, which meant basically the launch date for our project got brought a week forward as well. One of the other issues we ended up facing was really large data files that we were going to have to present to people. So obviously, you can't really have people downloading very large files. So these were some of the things that we needed to kind of overcome. And mainly, that kind of affected performance. So in order to find a solution, what we did have access to was the theme layer in Drupal. And we actually added a module to Cubs CMS, which is the services module and services views. Basically, what services does is it allows you to take a view, just a standard view in Drupal, and output that as a serialized output. So we were actually outputting that as JSON. So all of a sudden, rather than having kind of a very large heavy view that we needed to consume on the front end, we actually had some JSON output where we could access that just through the theme layer, through our own custom JavaScript, and build their own leaflet.js maps. We also used SVGs in order to output another part of the map that I'll click through in just a moment. And in terms of GLOcation, we started looking at an external service called IPFI. The reason for that was, once again, we don't have any code that we can actually write. So we weren't able to do any of that IPGLOcation ourselves. So we had to kind of go outside to someone who can do it for us. It's actually a really great service. You just send a request through with the IP address. And it sends back a JSON string. And actually tells you exactly where the location of the person is. So very handy service if you overlaid it. So basically, yeah, services, as I mentioned, kind of has a serialized output. We ended up using the REST server, which comes out of that. And just accessing that via JSON. One of the lucky things we had, all of the data was actually public on this particular project. So we didn't actually worry about the authentication. But Services module does integrate with the OAuth module as well. So you can actually put some security around what you output from services. And we ended up using three separate views. So we actually had a list of all of the different regions. So the regions list that I showed you before on the map, a list of all of the priorities and things listed, linked to those. And another view that listed all of the different projects. And then we kind of combined those together in our own JavaScript. So, just going to click on this quickly. So I'll just show you a couple of things within the services module. It's actually really, really quick to set up. You basically just add in a name, a machine name for this. I've said that I want this to be a REST server and a path to my endpoint. And basically, that's most of your setup right there. In your server settings, I've gone through and said that basically I want this to be output as JSON. And under my resources, I've basically said I want access to retrieve views. So essentially, within a number of clicks, I've actually got some, don't you have a question? Okay, yeah, so very quickly we've actually got the output that we need. I'm not sure what I'm actually clicking there to keep popping out of my presentation. Yeah, so it gives me the speaker notes over here. But then, but then I can't see my notes. Yeah, okay, so here are some of the libraries that we used. Leaflet.js, it's a really popular JavaScript library for making maps. Really, really simple to use and the tutorials and stuff are fantastic if you want to actually use that on any site that you're going to build. Like I said, there is actually a Drupal module for Leaflet. It's in D7. The D8 port isn't working all that well as yet, but hopefully it'll be there sometime soon. We also used the marker cluster module. So that's how we kind of created those little clusters of pins. We used the extra markers library as well to actually come up with our own custom markers. We need to actually list all of the points in alphabetical order. Leaflet doesn't provide that functionality out of the box, but with extra markers we were actually able to add those letters in there. We ended up not using the shapefile library, but that was one of the libraries that we were looking at when we did initially receive the shapefiles, as well as the JS zip library in order to actually unzip the file that we got, and I'll go through why that was in just a second. So I'm just going to pop onto the first part of the map, which is essentially a way to explore which council you actually belong to. So this is a fairly basic SVG implementation as it is now. The initial problem that we had was we received all of the shapefile data very late in the process, and what we'd been planning for is essentially having just one single map in leaflet. So when this part of the process came in, what we were initially doing is displaying a leaflet JS map here using very flat tiles without any real design behind them. The big problem we had was the size of the shapefile meant that just to load up this page here for someone was going to be incredibly slow. So what we did, this is actually about a week after the initial go live was we took these files and used a program called QGIS to actually reduce the complexity of these files. So to simplify geometry is what it's called in QGIS, and then export all of these as SVGs, which means that all of a sudden what was initially actually a 27 meg download to get the shapefile came down to about 100k. So obviously that's a lot more reasonable if someone's trying to look at this on any device really. I managed to do it without getting out of my presentation. Okay and so here we have the actual map itself. So the main issue that we had when we first actually implemented this map was the size of the shapefile. It was 25 meg. Initially we were actually supposed to have a map that was going to render on mobile and obviously at that kind of size it's just not going to work. So we went through a process of looking at a few different options one of those was to actually zip the file up until we managed to get the file size down to about 7 meg. Still probably far too large and we ran into a number of issues then where actually unzipping that file was taking you know up to two minutes depending on the browser, a number of different browsers i.e obviously was just completely crashing whenever you tried to do that. So that was not a great option. Similar to the explore map we started talking to a Department of Premier and Cabinet about can we actually just reduce the complexity of these files. The map or the website that we were kind of looking at initially was the region's website which is this one here where they have nine separate regions and all of these shapes as you can see I mean they're quite nice squares there are only certain areas that are actually complicated so the shapefile that they were using was really quite small. As you can see in here these we actually have 79 separate regions. The shapefiles were incredibly complicated and they told us that because the legislative requirements we weren't actually allowed to simplify these down. So all of a sudden we didn't have much of an option to kind of reduce our file size. So things were looking a little bit scary because we had a very large file zipping didn't really work and basically we were trying to deliver a page that for a lot of people they weren't actually going to be able to see. So luckily we've actually got our front-end dev down the front here Alan who came up with a handy solution which is basically splitting out those 79 regions into individual regions and each one of those had their own shapefile. We also had the the largest shapefile which was all of Victoria and that was actually the largest part of the shapefile. I think that was about seven or eight megs in itself. Because that was all of Victoria we can kind of get around the legal requirements there and simplify that down so we can actually reduce that file size and by doing that we actually had a far more workable map. So the workflow for most people was to come in to the explore by map click on one of these regions and we can see that it's fairly quick now because the only shapefile we're downloading is this one shapefile here rather than all 79 areas. So yeah so that actually ended up working quite well. If someone actually does come in and they do want to see all of Victoria then once again we can still present that entire region of Victoria in that with that simplified shapefile and it actually still is quite snappy which is handy. If someone did decide to go through all 79 regions they do end up getting to that kind of 25 meg file size. It's probably going to be a real edge case that anyone does actually want to go through all of that data. I think I've actually gone through all of that a lot quicker than I expected. I'm just coming up on half an hour. Okay so that was actually pretty close. Beautiful. So I'm just trying to find my mouse again. So yeah time for questions. All right did anyone want to ask a question? We've got to start over here. So when you found you had like a really big layer like with a lot of data did you consider like having your own WMS server with tile location? Yeah so that was one of the first things that we were talking about. The problem was because we were getting all of the data from the government they actually needed to host all of that data as one of the requirements that we had. We started actually discussing it and just given the fact that the deadlines were looming we decided to kind of continue down the path we were going because essentially we didn't have a lot of confidence we were going to be able to hit that deadline and get something going but yeah that was actually a much more ideal option and what we wanted to do. Did you ever consider maybe like running a Postgres SQL database just to handle just to hold all the shape files and then now put as like Geo JSON into your leaflet map as a as another option so so then they're just getting like you know a blob of JSON and essentially out of and I know that Postgres tends to handle geometries pretty well. Actually I'm just realizing that I've said the wrong thing on when we split out all of those files we actually did convert all of that to Geo JSON but in the end we actually had those as file fields connected to the different nodes and that was once again just given the timelines and what we had coming out of data we needed to have something that was going to be a bit flexible. Postgres probably would have yeah might have been a good solution to that too. Just two questions if you could have used modules would you have used the modules instead of? It's a bit hard to say so leaflet is is really good and I've used it several times before. The issue is yeah I think there is actually a patch out there to get rid of the way that it actually manages sessions because we're hosting all of this on Acquia and we've got varnished and actually Akamai in front. As soon as leaflet starts using those sessions and authenticating the sessions we're just going straight past the cache and that's something that we just couldn't do. On the actual launch time of the budget the traffic spikes quite incredibly because you know all of the large accounting firms have basically half of their staff going in and actually downloading all of the budget documents so yeah destroying the cache just wasn't an option. And did you look at open layers or? I haven't actually used the open layers module. I know that yeah I know that it's out there but I haven't actually tried that one myself. Yeah two was good, three's crazy. Crazy good, crazy bad? Both. Oh okay. What about that thing in the 90s image maps or whatever? Maybe you could hover over the thing. Sorry. Don't know that one. Anyone remember that was cold? Any other questions? Scanning, scanning. Up here. Hey good thanks for the presentation it's really good. Just had a question. As far as I understand that page was just outputting pretty big dump of JSON. So what kind of caching strategies did you use to actually make it faster? Is it just views? It is just views and services views. So to be honest we're probably relying a lot on varnish and akamai in terms of caching but all of the views they could output they will get cached in the browser as well so once you've actually loaded it up the first time it is actually pretty quick. And there's only like a few hundred projects so the size of the JSON is actually quite small. One of the things we were looking at initially because services and services views weren't in GovCMS was something that we weren't overly happy with was basically outputting a view but using data attributes you know to actually add all of the extra data we needed and although it's possible it would have been pretty slow and we would have had to do a lot of thinking around how we were going to cache that and make it snappy and doing that on top of having the large data files and all of those things it was starting to kind of make us sweat a little bit and we were kind of as we actually got more and more of the data things just started to get a little bit scarier to be honest because for the first part of the project we were kind of flying blind a little bit in terms of knowing what we were going to have to work with. So yeah I mean it's likely that we had services there to actually use. One of the challenges of geolocation work is mobile and your experience in mobile. So could you perhaps touch on did you cater for mobile or optimise did you change layouts or that kind of thing? We ended up not being able to do mobile on this particular project which was a bit of a shame. One of the major reasons was actually just around the design and we didn't get in a design that was actually approved by the time we were ready to go live. I would say as you know mobile maps can be quite tricky if you just kind of dump one in the middle of the page as you're scrolling up all of a sudden you hit the map and then you're scrolling the map and you can't actually get back to what you were trying to do and it's it's quite painful. I think one of the good solutions to that is apps like you know Zomato and other places where you actually click on a map icon and then you go into a full screen map and you've got another little icon to flick back to a normal view. Something like that kind of gets around those issues. In terms of the size of the files like that actually ended up being one of the other main reasons that we couldn't go there but I think with the solution we delivered in the end there wasn't really a technical limitation in how we did that. Also in terms of the geolocation we haven't actually implemented the geolocation in this solution at the end either. Once again we weren't delivering it on mobile so it wasn't quite as relevant but we've actually used IPFI quite successfully on other projects. Sorry which one was that one? IPFI, IPIFY I think yeah and yeah it's actually been a fantastic solution so we do all of the geolocation essentially on the front end which means that we've still got varnish, akamai etc all running. We're serving up completely cased pages but we can then do the geolocation and react to it once the page is loaded. So in hindsight do you think there's an argument for if you had another GovCMS project doing mapping that out of the box GovCMS isn't really up for it or would you be like well we've worked out of the door let's just do it that way? No I think basically with services and services views you can actually push all of the data that you need out so that you can do most of it in the in the theme. It is a little bit tricky depending on the requirements. I think we've got possibly another client coming up soon and they almost want like a just a single page site that is a map. Been kind of causing me a little bit of stress coming up to that one and thinking okay just given no module code how are we going to tackle all of this with you know just elements kind of dynamically coming in and out of the page all the time. I think it is doable and yeah once again it's just services being able to output JSON and make those requests with JavaScript all of a sudden it all kind of opens up. Sure. Any more questions? Hi Guy, we've got Mt. Beck. Hi, yes thanks. Do you think it would make a difference if you were to serve these maps if you didn't have it in the GeoJSON format from something like a GeoServer, GeoNode instead of straight out of Drupal? Yeah look I probably would make a difference if we had access to something like that at the time. One of the things that we're looking at for the next mapping project we're doing is national maps so I don't know if you've seen that it's basically something that integrates with data.gov.au and you can immediately kind of just serve up any data from data.gov.au that has WMS or GeoJSON in it so that's actually a really good tool. What we'll be looking at isn't national maps itself but using Terrajs which is the JavaScript library that sits underneath it and doing a custom implementation of that. So in that particular project we are going to have access to a GIS server and all of a sudden our options kind of start to broaden out a little bit. Thank you. So I guess I need to thank two people. First is the designer of the slideshow that's fantastic and also okay so please give me a round of applause.