 Hi there, my name is Jayden, which is the thing that you could actually Google for. These are my things. I'm going to talk about real-time GEO APIs. And I'm definitely coming from the JavaScript side more than the GEO side. I'm a Civic Hacker at Code for America this year, so I work with a lot of government data and a lot of open data, and a lot of that has a geographic component to it. I had to kind of develop any of my GIS sensibilities in a just-in-time fashion, so apologies in advance if I don't know your favorite projection. Webmer, Cater, all the things. So when you're dealing with things, it sometimes happens that they are related to real objects in the real world. And they're located places, oftentimes geographic places, places that you can plot on a map. And they also happen at a certain point in time. When you combine these, we have this sense of the immediate things that are happening in our surrounding environment. In 2014, we had a certain expectation that we're going to be able to have these remote senses by looking at our pocket supercomputers that are going to be able to tell us things like when the space station is passing overhead, or if the bus coming down the street is going to be late, and if you have time to run in and get an extra coffee. So how do you express these in an API? There's a couple of things that I look at as a developer when I'm both creating APIs and consuming APIs are things that I find highly desirable. One is I want them to use common formats. I want them to use simple protocols. I don't want them to be very surprising. In other words, that's me being lazy as a developer and saying it should just work. This should match my intuitive model of everything that I've been using previously on the platform. And the APIs should also understand how people are going to be using the data or functionality exposed by these APIs. So in our case for Geo APIs, a common format, let's use Geo JSON. It's a thing. It's fairly well understood. It's supported by a broad number of tools at this point. In terms of protocols, let's keep things simple. Let's use HTTP. Everyone who develops on the web has at least some understanding of it. And then for real-time components, let's just use the simplest thing possible in a plain text stream. There's a neat protocol for that called server-sent events, which we'll look at in a second. In terms of the principle of least surprise, Wikipedia says that you should exploit users' pre-existing knowledge. And in the case of updating points on a map, there's this pattern called the general update pattern, which you may have seen if you're familiar with D3. I know Mark Oxtalk is talking at Phosphor G, and that'll be a lot of fun. It also applies well towards moving Geo points around. So let's unpack the general update pattern, and maybe you'll recognize it. And if not, then maybe it'll be some cool new learning. But you have some things that you're modeling in your program, and maybe they come into the world. And then maybe some of them leave. And then the ones that remain, maybe something happens to them over time. Maybe they change color, they change position, some of the properties about these objects change. And so you want to be able to update and capture that information in your data model. So with that, we have those three events. We have things entering, things exiting, and things changing. And we could apply those to other things in the world. Now when we're building interactive applications, say in a web browser, we have a couple of well-known ways that we interact with users. It might be a zoom, or a drag, or a click event, and you just register event handlers. So when any of these things happen, and you're wanting to update the map, maybe it's taking care of for you by something like leaflet, or maybe you're adding custom handlers to react to that event. In terms of the other objects that you might want to interact with your program that aren't your user, we could also think in terms of their behaviors as actions that occur, as events. So for a bus driving around the city, it might be something like a bus starting a route and entering service. Then the bus is going to drive around the city for a little bit, go around the block. Maybe some people get on, they get off the bus. Maybe the bus is going to stop at a red light. Someone puts their bike on the bike-craft. Then the driver goes home at the end of the day and the bus leaves service. There are some cases where we might not have the data available in our model. Maybe there aren't sensors capturing this, and so we can't use it in our application. Maybe some things are just too noisy, and we don't really need to care about it. So what we're left with is a couple of events that actually follow very well with the general update pattern. Things entering, things changing, and things exiting. The domain that our application cares about. So how can we get events like this into the browser so that we can do things with them? We want some way to work with them just as easily as we do with click or touch events. We want some kind of event source. So there's a really cool DOM API that is in almost every browser, but very few people use, I know this, and that's a shame. There are polyfills that exist going back to IE8, so you could even use it in the government. But this is how... We had a request by somebody who works for the government to token down our spark. So I promise that person we've made a good faith effort. So speakers this afternoon, please, please be respectful. So you can even use it in your large enterprise GIS shops. This is how easy event source is to use, which is why I love it versus some other technologies that you may think of in terms of real time on the web. Web sockets are great and they have their uses or things like Socket.io. But in a case like this where we literally just want a one way continuous stream of events and we want to treat them in our application as events, I find something like Socket.io to be more complex than is necessary. You do this, you knew up an event source object and you just pointed at the endpoint for your event source API. It's just going to be a regular HTTP service. And then you could use regular DOM, add event listeners and give it whatever handler function you want. So now we have a way to get events into the browser. Cool. Where do they come from? On the API side of things, what is event source doing? Is it making some restful HTTP requests for us? Is it polling? What's going on here? So on the server side, it uses a serialization of framing protocol called server sent events. This is an excerpt from a valid server sent event stream. It's served with a mime type of application event stream. It's just plain text encoded. Please use UTF-8. But the way it works is you have events and then a colon and an event name, data, a colon and then this could be any string. In our case we're going to be using JSON and in particular GEO JSON. Then you just have a blank line in between and then you could keep sending as many of these events as you want. It ends up being a persistent HTTP request. In the event source spec, if for whatever reason the connection is interrupted, it will automatically handle your reconnection and retry logic for you. What's really cool about this is that because it's so simple you can implement it in any language on any platform really easily. You could do it in PHP and just to your output buffer. Whatever you want. You could actually just set up an HTTP server that's listening and tailing a Unix file descriptor if you want and just write to it in C or whatever. In my case, in the example, I'm using Node to spit this out and I think there's like 100 event source libraries on NPM. The way that you make this is not necessarily relevant for the rest of the demo. We're going to just take that you have some API that's going to be treating you nicely and giving you events of GeoJSON features. The rest of my talk is demo, so let's do this. Let's build a bus map. Here are some of my friends from Cascadia.js here in Portland recently. I'm going to be using NPM and just kind of showing my workflow for how I build maps, but it's not... Oh, and I'm also going to duplicate my screen. I build maps because I like to get to code as quickly as possible, but you could do this in a variety of ways. Much like a cooking show, you'd probably rather see the useful bits than watch me type or chop onions, so this is kind of our starting point. This is actually just one screen. There's no vertical scrolling, and this is how the level of code involved when I make maps. Can everyone see the text size? Okay, cool. I'm going to hide this real quick. You're using Linux, it's never going to happen. How? How to computer? All right, all right. So here's a text buffer. We're using leaflet on NPM, which means that we could just use CommaJS require. If you're familiar with Ruby, it's like require. If you're familiar with Python, it's like import. Other things have similar concepts. So we're getting leaflet, which is the library itself, and then a wrapper module called leaflet map, which takes care of some of the boilerplate, and we'll put it into an HTML page for you and just draw a full-screen web map. Here's the documentation for it, which is always great to look at. So what do we do? We get the map, we add just a generic OSM base map tile layer. This is just copy and pasted from the Getting Started with OSM page. We have attribution to make OSM people happy. And then we're just going to set the initial starting view to Chattanooga and add a marker. That doesn't sound too terribly difficult. Oh, no, let's skip that. All right, here we go. So the workflow is using Browserify to compile everything together into one package, and then this is just throwing it up in a web server. So now we can see that we have the beautiful default OpenStreetMap styling of Chattanooga, Tennessee, which is where I currently live. And we have our marker, and we can click on it, and it has the overlay. Chattanooga, wonderful. All right, so that's step one. Step two, this is where we start to add a couple of things. So we have all that stuff at the top still. And now we're going to add our event source. This URL is Chabus, which is the buses in Chattanooga. This is an API that uses kind of everything that we've just been talking about. It uses event source. It's going to return GeoJSON features, one per event. And it calls the events from the general update pattern add, change, and remove. So at any given time, the buses that are driving around the streets have, there's a certain set of vehicles that are driving the different routes. So it's just treating that as a set. Now, the way that this API works, we could debug it just using curl in our shell. So I'm just going to grab this URL. Curl is a Unix utility that will let you make an HTTP request and just watch the results in your console. So when we first connected to it, it replayed the current state of the set using add events so that we could handle everything in a uniform fashion. And at this point, there have been so many other events happening. But as we watch this scroll by initially, you'll see that the event type is add events for the initial buses. And then subsequently, there will be change events as the locations are changing. This could potentially be a little bit more efficient, but for the number of buses in our set, it's not really a problem that we're repeating all of the properties like color and route number. Even though the only thing that we really care about is the coordinates, the geometry of it, we also get some other information which might be useful. For example, we have the heading. So we could say what direction it's pointing in. And so this is it. This is a server-sent event stream. We could log this to a file if we want. We could proxy this anywhere. And it's a lot easier to debug in this case than WebSockets would be, for example. All right, so we've got that. And then we just add a couple of event listeners. And we're just going to log it to the console and see what happens in our web browser. So we have the same map. Nothing's happening because we haven't done anything with the stream yet. But in the console, you notice that we have all of these message events that are being logged. And so this is what the events look like on the DOM side of things that correspond to that server-sent event stream. If you're familiar with looking at raw DOM objects, it looks very similar. You have things like a target, but it's not really meaningful because it's usually just the window. So the important bits are you have the event type, which in this case, the dispatch of that is already being handled for us because we've added up to our on-change event handler. And then we have this data property. And you'll notice that it's still just a string because in server-sent events, it doesn't require it to be JSON or YAML or any other particular serialization. You get to deserialize the JSON yourself. So that's all we're going to do when we're handling this. First, we're just going to do JSON.parse. And then we'll have our objects that we can look at. So in step three, we start to do something interesting with it. All of the setup stuff at the top is still the same. We have our event listeners wired up. And now let's do something in on-add. So the first thing we're doing is parsing, just JSON parsing the data. And then there's this weird kind of, like, quirk going between geo-json and leaflet in the, is it longitude and latitude? Is it latitude and longitude? X and Y, Z, beta, alpha, theta, you know, whatever. So we're just going to convert that and ignore that. All right, so in our on-add handler, we're just parsing the incoming JSON string into a JavaScript object. Then we're creating a bus object in our application. So we're going to grab the ID just off of the geo-json feature ID, just because having identifiers is helpful in programming. Data is just the raw geo-json feature. And then we're also going to create a leaflet marker based on the geometry of the point from the feature. One of the properties that we're getting back is the route, which is just a domain property of what bus line that dot is on. And then we're going to add it to our map. So let's run that. So here's our real-time API. It draws static points in real-time. And it's beautiful. And unfortunately, in 2014, we don't yet have the technology to animate things. So I've been thinking hard in advance of Steve's question, and I hope that in the future, we'll be able to do that and show things moving. Of course, I'm being facetious. So that's the obvious next step. And let's look at what that looks like in our on-change handler. So all of those were plotted from the initial connection when we get the add event. So when it's replaying back the current state of the set of all the buses that are driving around Chattanooga, Tennessee, right now. So the next thing that we want to do is listen to on-change events. So from the top here, we've got all the same setup. We're drawing our map. I'm going to add a collection just called buses. And I'm going to use the JavaScript object here just as a collection indexed by the feature IDs. And I'm going to throw it on the window global objects just so we can debug it in the console and kind of see what we have, what we're hanging onto. On-add is the same, except we're also adding the bus object that we created to our collection. In on-change, all we're doing here is we're parsing the incoming event. We are looking up the existing bus object in our buses collection, because remember that also has the markers that we created on the map. And then we're just updating the position of the marker where it's drawn on the leaflet map with set lat long. And GeoJSON geometry is still the same. So now we have all the markers and every couple of seconds when the underlying data source is updated, we'll see these markers start to move. And it's subtle, so let me zoom in. Actually, that's as much as that'll let me zoom in on this layer. Can anyone see that, maybe? Yeah. All right. So when you're zoomed at, like, zoom level nine or whatever this is, driving at bus speed is pretty slow and you have experienced this. You've experienced this if you've been waiting for a bus to arrive. So this is the simulated version. But now we have dogs moving around on a map. And there are maybe a couple of more things that you could do. How do I...computer? A couple of other things you could do if you have a feed of bus locations or really any other kind of location. This is the unscripted bit. But here's an example of something that I made using this same data feed. This is the bus driver simulator 2000. And so this is where we're using that heading information. It's where we're using the Google Maps API to show street view image corresponding to the current location of the bus. And so that'll update. And so we could switch around to all the different buses that are currently driving around in Chattanooga. Oh, I'm glad you noticed because we built that in. We have this realism slider here. Ladies and gentlemen, not a plant. All right. So that's an example of what we've got and just to jump back to the slides really quick. G-I-S-G-L-V-O. Yeah, but it's PowerPoint version like Windows 8. How to computer? How to scroll on a list? Oh, yeah. That's all we have. Attribution. It's important. It matters. There were...even when your images don't load, in theory there were images from the Noun Project. Squares were designed by like our committees or Pythagoras or someone like that. And then all the material and demos and code and stuff that I did in this talk under the terms of Creative Commons Zero stuff. So I'll take one question exactly. Question for the audience. Our committees and Pythagoras are given a presentation later in the week. First time. Sir. So in source... Yeah. Yeah. So when I first started using it sorry, let me do this, was looking for why that is, right? And so the first thing I thought of was, well, maybe it's not really widely supported in web browsers. Oh, here's my animated GIF. This is the browser support matrix from CanIuse.com. And so we could see that except for in old Android browsers and then this other column on the left-hand side. It's supported everywhere. And in Internet Explorer, like I said, there's actually a really good polyfill going back to IE 8. And I think in theory it would support earlier things. And again, it's just because it's such a simple text stream to parse. And the cool thing is that IE actually has, because they have a different implementation of XML HTTP requests, they have better streaming support. And so someone basically just wrote a JavaScript parser for service and events. So we can use it actually with polyfills. I think this whole thing turns green. I'm not positive about Opera Mini. So it's supported everywhere that we would need it to use. Why don't people use it? I think so much mindshare, as like the word real-time started picking up steam, went to WebSockets that just kind of took the spotlight. WebSockets are hard to implement if you're writing a server. Like no one implements their own WebSocket server. Everyone just uses one that exists, which is fine. But it means that there's less tooling available and if you're adding it to maybe an existing project, that might be more difficult. Or if there's existing hosting infrastructure, for example, because WebSockets use an upgrade from regular HTTP, they use some clever tricks in the protocol. But some corporate firewalls, for example, don't actually play nicely and you have connection issues. And then you end up with libraries like Socket.io that try to do WebSockets and have all these horrible fallbacks. At the end of the day, think about what it is that you actually need streaming for. Sockets are great if you actually need bi-directional communication. If you're writing a game, you want persistent, kind of stateful sockets so that both the server and the client are kind of aware of what's going on on the other end. In this case, and in a lot of cases, where you're just pushing information one way, all you really need is a way to kind of continue sending updates instead of in a request response model. So for that, I would say event stores and service events are great and why don't more people use them? Well, now you all know about them and you all should use them because they're well specified and they're well supported and they're really simple. I like that. Let's give it a hand.