 So, yes, my name's Robin, I'm the creator of Vizzy Cities, and over the next half an hour I'm going to give you a sort of a rundown of what Vizzy Cities actually is and an overview of some of the key lessons I've learned while I've been building it. So if you have any burning questions or comments during the talk please direct them to my Twitter account, that's at Rob Hawkes. Otherwise, we'll hopefully have time for maybe one or two questions at the end, but if not, I'll be around today and tomorrow at the event and feel free to grab me either here or on the Slack channel as well, so I'll be there too. So as a little background, my history is in programming, combined with graphic design as well, so combining those two to solve interesting problems. And I've previously worked at both Mozilla and Pusher as developer relations, so talking about sort of cutting edge web technologies, sort of travelling the world doing that. And today my days are spent running Vizzy Cities as a company, trying to create an open source company, as well as doing consultancy work around geospatial data visualisation. And if I do happen to have any spare time, I like to relax with really nerdy games like World of Tanks and Hearthstone, so if anyone plays those I'd love to talk about them later. It's probably only like one person. However, you're not here to sort of hear about all of that, so taking a step back, we know that cities require things on vast scales. Food distribution systems, reliable energy, transport networks, healthcare, civil protection, water systems and all sorts of other things, and they all work independently of each other. And if one of them failed, it could potentially bring down multiple parts of the network. To ensure that cities are chaotic, they're complex, and they're awe-inspiring, they shouldn't work but they do, and you can't fully control them. What makes it even more important to be able to understand them, not only so we can make them work more efficiently, but also so we can understand their complexity as the citizens living within them. And there's a huge amount of powerful data locked away within cities, it's in places like data.gov.uk in the UK, as well as a large number of other private and public repositories. And what if you could create a tool that allowed you to take this data and allowed people to see real-world cities and overlay data upon them, a tool that helped them unlock some of the data and combine it to better understand their cities and their place within them. So it turned out a tool like this already existed, it's the game SimCity. In SimCity you create your own fictional city from scratch and you manage every aspect of it. And it gives you a data view that allows you to peel back that city to get a much better understanding about how it works and more importantly why it's not working so you can then go in and fix it. But it's just a shame SimCity is for fictional cities. So what VizzyCities aims to do is effectively do the same for real-world cities just without the game aspect. So it's keeping a SimCity-like balance of practical, beautiful data visualization and a great user experience. So the 3D platform that VizzyCities has takes data like this, which is raw open street map data, and it uses it to generate and visualize real-world cities in the browser in 3D. But you can do much more than that. So VizzyCities was actually the first in the world to publicly visualize the London Underground network in 3D too. And actually not only visualizing the London Underground network in 3D, but also the live trains and the positions that they're in right now. So this is a video, but if you looked at this live you would actually see the trains where they are right now in London. And they also looked at visualizing the London bus network in similar ways and approaching things slightly differently. So not using straight lines between bus stops, as most people do, but actually using the real road network and having live buses moving around them. And thanks to the guys at Plain Finder also looking at things like air traffic. So visualizing 3D air traffic in a city and kind of fulfilling that goal of bringing a city to life. But you can also do more basic things like using LiDAR height data to produce 3D terrain maps. So this is very useful for doing things like flood level analysis. So you can raise the water level five centimeters, half a meter, five meters, and see as a citizen what's going to happen around you. And I've also looked at heat maps and other more traditional visualization techniques to allow you to just get a better understanding in your local context so things like, I don't know, education, crime levels, pollution, and even using them together to compare and look for correlations. So it's important to be clear that Visicities is browser based, so it requires no plugins. It works out of the box, it's using WebGL, and modern computers, modern browsers, even mobile phones support it. And this is a massive difference to existing tools that allow you to do these kind of things because they're usually desktop apps that only work on single platforms like Windows. In fact, if you didn't know already, Visicities is an open source project under an MIT license. So all the code is on GitHub, and please feel free to go and poke around with it and learn how it works and look at some of the solutions that I talk about today. So it's been over two years since Visicities started, and I've learnt a bunch of important lessons during that time. There are too many lessons to fit into a single talk, so I've tried to pick and choose my favourites. But make sure to come talk to me if you do want to hear about some of the more random ones, like working out where a city actually starts and finishes, like how big are cities, that kind of stuff. And each of these lessons could actually be an open talk in their own right. So in the interest of time, I'm going to give an overview of each one rather than delving too deep into them, especially I won't be going into code, because you don't really have the time to take that in right now anyway. So you can find that all on the GitHub. So let's get cracking with lesson one. Data quality is a pain in the bum. I'm trying to find the more polite way of phrasing that. So buildings were the first challenge regarding data quality, and they're very important if you want to visualise a city because it's made of buildings. And in fact, the challenge still exists today. So to sort of put this into perspective, you could use ordinary survey data in the UK, and it's free. It's quite simplified, which does mean it's quite inaccurate, so these don't really look like buildings. And it can also slice the buildings up into multiple parts because it uses this kind of grid system. So you get this issue in the middle where you have buildings that are split into four different parts, even though they're just one building, so it's not great. It also has no heights, so if you're trying to do things in 3D, you can't really know how tall they are. And the performance is good, actually, because of the low level of detail. In comparison, you have OpenStreetMap, so many of you have probably heard of OpenStreetMap. It's free, and often it can be more accurate than data from other places because it's hand drawn by people. But then again, the quality is unreliable and inconsistent because of that exact fact. It is however constantly updated, and it does have some heights. So in this example, the red buildings have heights, the blue buildings have floor levels, which you can roughly equate to heights. But it does have performance issues due to that higher level of detail. Another issue is that data is provided in many different unrelated formats. So, for example, a lot of government data is provided in Excel spreadsheets, and geospatial data can be provided in a number of different formats like shape files, arctis files, geojason files, CSV files, or something entirely different. The point being that there's no consistency to that data. And combining all of this data can be a long and arduous process. So when making the 3D London Underground demo, I had to deal with a bunch of different problems. For example, all the data I needed was hidden away. All the data was provided in different formats. I had to manually merge the data because identification conventions between different departments of the same company were different or unreliable. So I was literally spending a whole week of evenings hand converting and combining all of these different Excel spreadsheets. The Transport for London API is unreliable. It's slightly broken, and it's in XML. So it's verbose, it's complex, and it generally doesn't give you good data for real-time analysis of trains. Combined, this made what should have been a relatively simple task if we were using a nice web API. It turned it into many weeks of unnecessary sort of hair pulling. So lesson two, cities are huge. So it turns out cities are far bigger than I expected. And honestly, someone could have told me that before I started. I just kind of didn't realise. Maybe I was too complacent. So I attempted to initially render the entirety of London as a noble goal. It turned out to be a vast area. I mean, I live in London, and it's got a huge number of buildings. We're talking many millions of buildings. And the screenshot here isn't actually all of the London borys, which is what I wanted to originally visualise. And I actually stopped at this point because I just realised what an impossible task this was. So the first solution was to actually crop the area that you're visualising. So you can see one part of the city at a time. And it's sort of like a plinth that updates as you move around. And it worked, and it looked nice. Though it was kind of clunky as an experience, and it introduced another bunch of problems related to how it was rendered as well. So the second solution, and the one that I've settled on, is introducing a method of scrolling around. So it's kind of infinite scrolling. Very much like Google Maps of what you're used to already. And this was the method I settled on in the end. And though it was not as visually impressive, it solved basically all of the problems and meant that I could have a rough idea about how many buildings could possibly be visualised at one point in time. And I'll touch on some of the specifics on how this was implemented in a further lesson. So lesson three, use third party data APIs wherever possible, unless you're made of money. So while using OpenStreetMap is great, the data for the entire planet, which was the Gulf of Izzy Cities, tops 300 gigabytes, and this updates minutely. So it's not ideal for serving and hosting yourself. So is it possible to get this kind of data dynamically without hosting it yourself? So the original solution came by using something called the Overpass API. Now this is an external JSON and XML endpoint to OpenStreetMap data. And Overpass allows you to send a request for a specific OpenStreetMap tag, which is like building or road, that kind of stuff. And then you also give it a bounding box in geography. So you get back a lovely JSON response of all the data you need. And the byproduct of this is that you get worldwide support out of the box and you benefit from the minutely updates of OpenStreetMap. So seriously, if you edit something in OpenStreetMap, you will see it in Izzy Cities within like one or two minutes. It's insane. And the sub-lesson I learned doing this was that spamming this Overpass API, which was kind of meant as a gesture of goodwill to the community, not for businesses to completely over abuse, was that if you send it a ton of XHR requests, it kind of shuts itself down and stops you doing anything else. So that wasn't good for Izzy Cities. So it generally caused delays as those rate limits ended up being like one per user per second. So it took a long time when you're trying to load like 50 tiles in five seconds. So the great thing was that by already using stuff like promises, I managed to, sorry, to manage the XHR requests, I was halfway ready to kind of solving this issue. And the final piece of the puzzle was to use something called froats.js, which was to limit the number of concurrent requests that you're making at one time. So you can take control and load resources as and when you're ready and without abusing the external API. So any rate-limited API could benefit from something like this. So since then, I've moved on to more robust WebE APIs for OpenStreetMaps, stuff like the MapsN vector tile service. But also things like CartODB and MapBox, they offer a variety of solutions as well for geographic data, particularly if you want to store your own and host it externally. So lesson four, accurate height data is rarer than rocking horse poo. If I lived in New York City, I wouldn't really have a problem because they've decided that they want to give away all of their height data for all of their buildings for free. That's fantastic. I don't live in New York. In the UK, it's much harder to find those building heights. As I mentioned, OpenStreetMap has varying levels of detail with height. You get some buildings in the ones in red that have height and others that don't. But you can couple it with an educated guest to see or predict height for other buildings. But actually, if you are from the UK, there is good news. The Environment Agency released LIDAR height data for most of the UK this week. This is huge because it now gives you centimeter-accurate height level data for both buildings and terrain across the majority of the UK, and it's all for free. My hope is that other countries follow suit because this is the hardest data set to get, and it's one that's really bizarre that people don't release it because they have it. So lesson five, converting geographic data into 3D isn't actually that tricky. One of the first problems I encountered was how to turn these geographic coordinates or building outlines into pixel-based coordinates and also 3D. The math involved to achieve this is not that simple, and it gets even more complicated if you're considering different geographic projections, so if anyone knows about that, you know that it can get confusing quite quickly. So fortunately, the Proj for JS library is here to kind of help you convert coordinates, and it's fantastic, and it's quite a small library, and it allows you to convert from practically any geographic projection into 2D screen space. And the second part of the puzzle is to visualise those objects in 3D. Now, I chose free.js, which is a popular and easy-to-use WebGL 3D library, and by combining Proj for and free.js, you can quickly convert these geographic coordinates into 2D shapes in a 3D world, and it's cool, but it's not what we want. The final step is to use a feature in free.js called Extrude Geometry, which lets you stretch these 2D shapes into fully-fledged 3D objects. And it's quite amazing how adding that third dimension suddenly makes that city kind of pop and come to life in front of your eyes. So lesson six, quantity and performance are mortal enemies. So one area that has proven to be a pain time and time again is working out how to visualise as many buildings as possible without causing performance issues. So one early approach I took to improve performance was by splitting the entire world into a tiled grid, much how I mentioned previously. So this is exactly how Google Maps do things. And it allows you to load data in small tiles that are eventually used together to build up a complete image or an entire area of a city. And it solves one part of rendering sort of rendering large numbers of buildings instead of loading an entire city in one go because you only load the tiles that you can actually see. So if you do things like turning around, the tiles behind you disappear. But another problem that existed even with tiles was that complex 2D shapes inherently cause a lot of strain when rendered as 3D objects, particularly on mass. So what's interesting is that you often don't notice this complexity, it's actually unnecessary in the data. So particularly when you're zoomed out as well. So why keep that complexity? And to get around this, I use a library called simplify.js. And it dynamically reduces the complexity of 2D polygons before you render them, basically. And it's a great little tool that allows you to keep the general shape. So it still looks like the original shape, but it just has a lot less complexity. So it means that you reduce the render cost and it allows you to effectively render more objects, or in our case, buildings with little to no change in how those objects look. So lesson seven, don't lock up the browser. So we can now render tens of thousands of buildings. Fantastic. But how do you actually process all of those buildings without locking up the browser? So the solution is to use web workers, which is offloading complex processing to a separate browser thread. And what this means is that you can lock up a web worker as much as you want. And yet the main browser process, your app UI, will basically stay silky smooth and responsive to everything that's going on. So instead of getting sort of dirty with web workers, which actually aren't that complicated anyway, I went with the operative library so I could basically use some of the helpers for handling common situations. And in busy cities, each tile is handled by its own worker. And what this means is that the entire processing pipeline from taking geographic coordinates to constructing 2D shapes to constructing 3D objects is all handled within a web worker and it doesn't block the browser. So instead of having a noticeable difference in frustrating delay in browser responsiveness, you now have this smooth experience where the user can continue to use your application and continue to do things like selecting text or clicking and just kind of actually using the app. So if you don't know what you're doing with web workers, as I didn't when I started, it's easy to introduce a new problem and a new lockup which is passing large quantities of data back from a web worker to the main browser thread. And to get around this, you can use a nifty little feature called transferrable objects. Now, these are basically buffered arrays in JavaScript and they allow you to effectively hand over the keys of the data from the web worker to the browser. So it never has to be copied. It's just the permission is passed across. So it's an instantaneous process rather than a long process when you're copying. So lesson eight, request data once and only once. So this was quite a simple lesson, but one that still proved quite troublesome to solve correctly. And it originates from data being requested multiple times as you're loading all of these different tiles. And also the data sort of being reloaded again as you refresh the page. It's a waste of bandwidth and it's also a waste of the user's time as well waiting for all of this stuff to load. So how do you know what you've already requested and how do you make sure you don't request it again? So the approach I currently use is far too simple and it's basically just an in-memory cache in JavaScript that stores buildings as you move around, but it disappears when you refresh. So if you go back to another tile that you've already downloaded buildings for, then they're pulled from this cache and you don't have to load them again, but only if you don't refresh the page. And what I plan to implement next and what I advise that you do is implement a more robust cache that utilizes local storage functionality in browsers. So things like indexedDB and WebSQL. And there's a great library called local forage by Mozilla that allows you to sort of shim a lot of these APIs. And what this allows you to do is effectively store this data outside of your website or your page. So when you refresh the page you can re-request that data and start from where you were. So it's great for building offline applications. So lesson nine, decouple the architecture. So how do you create a large application that's extensible and easy to maintain? I didn't know how to do this when I started. I'm used to just tinkering with things. So two aspects that really helped me with decoupling the architecture were using the mediator programming pattern but also modular architecture. So modular architecture such as common.js is really useful for breaking apart your application into small components that you can require in as necessary. So if you've used Node.js packages you already know how this works. The mediator pattern is pretty much published and subscribed API which is great for managing communication between these disparate modules without having to pre-program how they should communicate amongst each other. But the problem is that you need to be careful because you can't guarantee what order these messages are going to be received in by the other modules which is particularly important when you're doing things like rendering things in a browser. And I advise that you check out Addy Osmani's excellent book on how to implement pretty much every common programming pattern in JavaScript and there's links in the slide which I will put into Slack as well. And promises are another technique which have proven incredibly valuable particularly the Q library and there are many other promise libraries but this is the one that I use and they allow you to defer the handling of processing until a later date by accepting a promise and that promise is that a predictable action will occur in the future and it encourages an asynchronous approach but it also encourages you to continue working away at something else while you're waiting. And what I've found most useful is that it lets you avoid messy nested callbacks and vastly simplifies the flow of your application and the readability of your code too. So lesson 10, large projects can be a nightmare to manage. So I've never been one to care too much about process or code quality, testing or even making sure things are done right which would be pretty bad if I worked in an agency but I don't. As I said, I'm a tinkerer and I prefer learning and seeing results rather than spending time on what feels like wasted time kind of building these processes and not actually doing the result. And it turns out my tinkering approach doesn't really apply too well to large web applications that require consistency and robustness. Who knew? So it didn't take long for it to become a faff to manually process these files to create new builds every time to generate single JavaScript files with all the modules and all the external dependencies and to serve the examples and the breakthrough for me was adopting just these simple build systems but now I use Grunt in Visicities but I actually use Golpin projects that I've started more recently and Grunt allows you to run a simple command in the terminal to do things like automatically test things and concatenate and minify your files ready for release and I also use it to serve the local build and to auto refresh the examples when you're in the browser. Now I'm sure most of you already know about Grunt but if you're testing it you can look at the Visicities code because the Grunt file is there and it shows you everything that I use it for. Now for automated testing I use Mocha combined with Chai, Sinon.js and Slimer.js which I'll touch on in a minute and each of these serves a slightly different purpose in the automated testing process so Mocha is used for overall testing framework basically sorry, an assertion library that allows you to write more readable tests so it's basically just the words that you're using to construct these tests and Sinon.js is used to fake application logic and basically use these things called Spies to track behaviour and to track processes as they happen and then Slimer.js is used to run client-side tests in a headless browser from the terminal and Slimer.js I chose because it also supports WebGL so theoretically you can basically test graphics at the same time my experience is that it's slightly funky so don't rely on that and then combined using something like TravisCI to make sure I don't break anything when I push all the changes to GitHub it runs all these tests automatically so it performs all the linting and the tests and when the changes are pushed including pull requests for many other contributors it all kicks off and tests everything and what's even better is that it integrates directly with GitHub so it automatically performs these tests whenever a contributor adds a pull request so I know that there's an issue or that there isn't an issue and that it's safe to merge this pull request if I want to So lesson 11 make sure you actually improve things so this was a hard one because you're constantly tinkering away at something but how do you know that you've actually made things better or worse the simplest approach is to just monitor performance in frames per second so one way to do this is with something called FPS meter and it's good for an overview but it's not super reliable the best way to track framerate is to actually use built in frame timers in browser development tools and these timers are useful for getting a general indication of parts of the application or parts of the usage of an application that are locking up the browser or preventing frames from rendering at a fast pace so for example this is how I worked out that I needed web workers because all of a sudden the browser just stopped for like 12 seconds so I was like what's going on but for more in-depth analysis to actually work out exactly what's causing it you want to dig further into the development tools so in my case I've been having a lot of success recently with things like the timeline and profiles tools within Chrome the timeline is particularly useful because it gives you a detailed overview of everything that happens in a single browser frame so it allows you to basically see what's taking up the most amount of time and see how that cascades up the various processes that are calling parts of your application so most recently I used this to confirm and fix an issue where I had an external event amid a library which was causing a dramatic slowdown but only when there were lots of objects in view so it was fine most of the time which is why I didn't see it and when I opened it in the timeline you could see these massive frames so when I switched out the library like literally that's all I did is try a different library and rechecked everything in the timeline each frame now took 17 milliseconds rather than 160 so it's having that assurity if that's a word that something has changed and you've actually improved things that makes a massive difference and these tools are incredibly powerful once you learn how to use them particularly the profiles view as well which gives you an idea about memory usage as well in your application so I highly recommend that you learn about them and I have included a link to some of the really good Chrome documentation on how to use these so overall just don't sacrifice your enjoyment in pursuit of perfection this is more a general life rule but I found that about a year into the project I started taking it far too seriously and I forgot why I was actually doing it and I lost track of what I enjoyed about it and I ultimately felt like it was a chore to continue working on it so really make sure to take the time to do what needs to be done while also ensuring that you continue working on the things that you want to work on the things that you enjoy most about what you're doing so I managed to get back into the swing of things by experimenting with new technology so stuff like virtual reality and putting that into busy cities so I got to strap what looks like an Xbox to my head and just get excited about it again and whatever you find that does that whether that's trying a new library or writing tests or stuff maybe you've been putting off but would sort of prevent you from enjoying the project again then definitely do it because it's amazing how quickly you can get that spark back again if you just give it the chance so I'm afraid that's all I have time for but as I mentioned please grab me later like honestly happy to chat about whatever I've spoken about or anything that I do if you have any questions or just want to chat you can find out more about busy cities at the website which is visitities.com follow the project on Twitter which is at busy cities or like I said just grab me and thank you very much for listening