 Welcome back. One quick schedule change announcement. Unfortunately, Brenda Wallace has taken ill. So instead of her talk at 1.30 in conference two, we will be having Sino-Evil, Hino-Evil, Pachino-Evil by Graham Dumpleton instead. Not sure whether the website's being updated, but now you know. And up now we have Tom, who will be teaching us all about Auckland Transport APIs. Let's make him feel welcome. Hi, I'm Tom Slavskonsa. I work for Propelahead, a service company in Auckland that works with Auckland Transport and a few other transport partners. And we have over the years built a set of APIs for Auckland Transport and exposed some of the transport information to the public. So this is what I'll be talking about today. First, signups. If you, as a developer, want to start working with these APIs, it's easy. Just go to the devportalat.gov.nz. You should be able to sign up, get a key, and see all the APIs. See the basic documentation and start off. So it's been a long road. It's been a few years. These APIs have accumulated over time. We have published a few of them using well-known de facto standards for transport. That's the GTFS and GTFS real-time, as you can see, for schedules and real-time information. That's a Google standard. It's been used all across the world, used to record scheduled movements of all kinds of vehicles, mostly used for buses, I believe. But we will be using it for trains and ferries as well. For real-time, so for real-time vehicle movements, for trips, for vehicle locations. That's the second one. Locations, that's right now, managed by Auckland Transport. And that is locations of 80 parking spots. Mostly, I believe, those are the parking buildings around Auckland. Points of interest, anything that shows up on the maps on the website, manually inputted, manually maintained. Tools, various tools used by AT again, and mostly by Journey Planner and Yokozor. So I'm just listing these so far. I'll explain in detail what they do and how they work. Journey Planner is a tool that is used, I believe it's, there's a few other options, but the only supported one is on the Auckland Transport website. Given start and end point, it will plan a journey, give you a list of options, give you a phase. And yeah, well, that's it. It exists as an API, but is currently just restricted to partners. We have received requests for that, but for now it's restricted to partners. Geocoder as well, so if you are, if you know, if you use any site that asks you to type in an address, and then gives you a list of real addresses or shows you that address on a map, like Google Maps, that's a Geocoder. And we have a service running in Auckland Transport that again is a restricted service currently used only by Auckland Transport. Hopefully we will get that out as well. Display, those APIs are used for various screens around the city. I'm trying to remember if they're used in vehicles. I don't think so, not yet. Cameras, there are a few traffic cameras around the city and you can access those as well. Events, those would be the, well, notifications about various roadworks and such. Now, these are all available currently on our public and our public restricted APIs. And if you sign up, you'll have access to most of them. You will be able to look at the scheduled data and you can, you should be able to see all the currently active information about vehicles in Auckland, as well as real-time information about vehicles. If you want to build an application that shows vehicles, that shows, that allows people to plan trips, to do anything with real-time data, maybe something related to industry, maybe something to optimize trips, should be possible using those. We have a few big companies, one of them is MachineZone, that has used these APIs to try to build very large enterprise pieces of software. Hopefully what I'm going to talk about a little bit later about our effort on community engagement will expand this effort to more developers, to include more developers. So I'll just try to quickly go through how our data looks like, because that is the most complicated. I mean, the rest of the APIs are fairly trivial. If you look at the displays, events and locations, those are, you do a request and you will get your information back and that's pretty self-explanatory. If you look at the API documentation, what it does and how to use it. The GTFS, the actual schedule data, the real-time data is a bit more complex, so I'll try to expand on it now. I won't go into the whole spec, of course. You can check it online on the Google sites. There's a list of resources at the end of this presentation so you can read in depth. But this is basically all the information that you get for static data. You get the agency information about various companies that run services across Auckland. You get the calendar dates and stop-downs which are scheduled information about when the service runs, exactly when it should arrive at stops and so forth. Shapes is the geographic shape. So when you go on a map and you, on Google Maps to a trip and you can see those pretty shapes, that's what this is. Specifies exactly how the shapes should be. So you can draw all these routes. All these routes are present in these files. For all Auckland trips that are tracked. Calendar, pretty self-explanatory I think. Feed info is something that can help you keep track of all the data updates that come along. We do update data fairly often. So if you want to keep up, so bus stops move, times change, all of that. If you want to, if you build an application and you want to keep track of all the changes in the transport space, then this will help. Recently with the CRL, with all the tunnels being dug up in the city, they've been moving stops all the time. So weekly was the norm up until a month ago. Now with the new zones and the CRL underway, things have calmed down. About every three, four weeks. So that feed info information present there is also available as an API. So if you want to keep track of it programmatically, you can. Routes, add, describes all the routes set up. This is, there is a bit of terminology here that I'm hesitant to go into. There's very specific definitions for what is a trip, what is a route, what is the relationship between those things. So let's just say that yet another blob of data is in there and you should take a look at the spec to see exactly how it fits together. Stop info is, as in any enterprise system, at some point you reach the enterprise designs that the spec is no longer enough and they need more data in. So instead of just messing with the tables and these existing spec files, we decided to just create another file. So it's 80 specific for 80 use, but if you find the data inside useful, feel free to use it. What it maps to, if I remember correctly, is for big stops that have multiple subtops inside. So it defines the parent, child relationship, boring. Yeah, so these are basically CSV files. It says TXT and that's per spec, but it's basically CSV files. And they can be, they obviously have relationships and they're sorted out on the Google official spec site, but you can easily load them into a database. And that's probably how it's designed to work in a relational database. I missed some. So one thing that I would like to point out that we struggled a lot with is versioning. Since we are updating these things all the time, we have to be versioned and we, this is the version format that we ended up at. Right now it has the timestamp, the timestamp in the actual version and that gets suffixed to every ID in the table. So even though the IDs look like it should, they should have meaningful values. For example, you can find an ID that's 8864 and that conveniently maps to a trip somewhere. Probably you will just better off thinking of it as a justice string. Right now it is 8864 with the time string and the version attached. Tomorrow it could be anything. IDs should be used programmatically as black boxes. Right now we can parse meaning, but tomorrow who knows. That is why this exists. So if you do want to get root or trip number 8864, you take it from the stop code field and you add the suffix as this suffix and then do the relational magic to get your data. By the way, this is our envelope for each message. If you use the APIs and find that the response does not look this way, please let us know because that's a bug. Yeah, I think I see them still more. Cool, so we do plan to have at least one update a month. So even when nothing is happening, they're still updates. So things will change. And I haven't listed all the APIs, all the endpoints because of course you can easily find them online. So just to let you know, there are, I guess, queries available. If you want to get, from the API, you want directly find all the stops in the radius. Like you're building a mobile app and you want to know, give me stops in 100 meter radius you can, there is an API for that, then it works. Not all of them, of course, just the ones that we use the most and then once we went public, that's what we had and that's what we published. GTFS real-time, this one is fairly standard. Once you understand how the data maps inside the main data set, then these are fairly obvious. Once you see the payloads, they are fairly similar but the first one focuses on trips and as I said, there's a specific definition for a trip. That's a vehicle that is on a known schedule. So you do not have unscheduled trips here. All of them are scheduled and if it's a, so there could be vehicle updates, buses going back to the depot, that are not here in trip updates but everything here should be here. Um, these are useful. I believe that what Machine Zone used was the vehicle updates API and they really hammered us. They wanted to do, like they wanted to go twice a second or something, whatever. The current resolution on real-time differs between modes. We get different frequency updates for vehicles that are for trains versus ferries versus buses. Buses get updated every 20-ish seconds in average, I believe. The trains should be more frequent but there's no guarantees and ferries are every 60 seconds. I mean, does anybody really care? It's on a blue, on a blue map there. Anyway, apparently people do care and we will be including ferries in the near future as well as trains. Right now the real-time is only buses. I forgot to say that. It's only buses. Anything else to say about real-time? Oh, I guess I'll remember. I'll leave it here. It's a short talk so I'll leave it here for just explaining, yes. So I can explain a little bit how this message comes to us. So as long as the driver hasn't typed into his device onboard, my route is canceled, we'll still see it as a trip. If the driver enters into his device no longer on a trip, going home, then it will not show up on the trip updates page, on the payloads. So you would have to implement the logic. You'd have to keep track of all the trips that should happen and all the trips that are there and then compare. You would get all the schedule trips from the static data and all the trips that are actually happening from the real-time. Well, no comment. I won't talk much about the relationship with AT. Let's just say I'll come to my last page is on something that relates to that question. So our efforts in that direction. Mashups, so just in case you didn't know, this sort of data is always more useful if you have extra data that you can put together with it. Give it some context, get something interesting going. We have, NZTA have some IPIs available. Motorway events is the one I know of. Should be useful. It does appear on the AT website, but well, again, I will talk about our thoughts on that a little bit later. The AA site has a few APIs, interesting ones. MetService exposes its data as, I believe you can get not quite JSON or not quite JSON APIs. I think they expose their weather data as files or something and you can parse it and then. Yeah, something like that. So still useful in a transport context. Nobody has done anything interesting with that yet, so hopefully somebody does. TomTom has a set of APIs if you want a journey planner service. If you want to start, if you want to build your own journey planner app, that is one of the more tricky parts to do right. There are a few open source projects that do it. They're a bit heavy, Java, not nice because they've been going on for so long. Nothing really solid as far as I can see, new and solid, but the ones out there work, I guess. There's open street maps and there's Google Maps as well. Google Maps is a paid one after a certain threshold, but they're good, so. I would like to hear more if somebody knows of any extra APIs that you know of that are available, just let us know and we'll find a way to use them, I'm sure. So now to the interesting part. Since we've been working for 80 for a long time, there is, as in any enterprise, there's lots of interests, there's lots of conflicts, it's hard to do stuff, it's hard to, especially it's hard to publish stuff. We have managed to work with them to get the current APIs open. Don't get me wrong, like many people inside do want that data open, but there's so many interests involved that it's just hard to get it out sometimes. To witness this difficulty, you just have to install a few mobile apps that have the AT brand on them, and you can see it's obviously designed by committee, it's obviously, they work, and I'll say that, and we've built a few of them, so I can't really distance myself from the projects, but there is no freedom to do anything interesting. Thus, AT Labs, this is something that we've been trying to do for a while, and now we've been given approval from OpenTransport. This is a project that tries to get more people involved to allow us to open source some of the projects that are currently live with the AT logo on them. If you've used TrackMyBus, and I'm guessing nobody has because there's only 10,000 users on Play Store, so if you have used it, you know that it's probably a nice idea, but there's no polish, there's no work on it, it should be open sourced soon, unfortunately, but Python doesn't run on Android as far as I know yet, so it's JavaScript basically, transpiled into iOS and whatever, into iOS and Android. It will be open sourced any day now. We are also opening up the API specs for the public APIs, so if you want to report an issue, or you want a new feature, or you don't like how it looks, you can open up, it's a repo on GitHub, you can open up an issue and talk to us, basically. We will be tracking all issues from the public. Right now, the current Dev Portal, the one I've linked on the first slide, that goes to the API manager, and that was our first effort, but we're trying to move all issues into GitHub because we manage GitHub, and we will be answering questions, and if you had a question and nobody answered, now is the time to ask again. We have backing from Auckland Transport, there's been talk about bounties, publishing bounties for interesting projects, if ATs want something done, we might get the funding to actually publish this. What else? Well, take a look, anyway, it's on atlabs.xyz. We are also, the Getting Started link is written by my colleague, trying to explain all the trickier bits in our APIs. It's been a rush, this all happened after I signed up for this talk, really. So it's been for the last two, three weeks, just a mad rush to get everything out. The website, the GitHub Pages code, clean it up, all of that. We will be using Waffle.io for managing the tasks, if you haven't seen it, and, well, I'm not surprised, the world is full of interesting projects, it's just a GUI, an app that uses GitHub issues and puts them into a Kanban board so you can see what's been worked on. So if you create a bunch of issues, you can see that somebody's actually working on getting that done. We also accept pull requests for anything that's in GitHub. If you find something that you like, if you want to improve something, if you want to contribute, just let us know. GitR is the best, well, we also try to approach people who are not developers. Ideas would be good, just plain English, no tech ideas about improving transport. Those, that's basically a chat, the GitR is basically a chat client so that you can hop on. You can use GitHub or Twitter account, I think, for now. Start talking to us. We will have a chat channel per project. So if anything is specific about, anything generic about APIs, just go to the API channel and we'll sort out the other channels as we go on. API.Portal, that's the one from the first page, that's the API manager one, that's the formal place where you get your credentials and you can create issues there and we will probably just move them into GitHub. It's just hard to manage things in the multiple places and we don't have that many people working on this, so as few systems to manage that, the better. A few other transit wiki is a good nice site to look up general information about gtfs, gtfs real-time. Google transit reference is a nice, really nice extensive in-depth presentation of their spec. We try to keep the spec in all things. We even have Protobuf on gtfs real-time, which is nice. I'm not sure if anybody's using it, but it's there. I think that's pretty much it. So any questions? What's your relationship with AT and the actual real-time system that's giving you the information? Is that a? That is complicated and that is why everything is more, it's always complicated. So we get real-time information from multiple sources. We get them from the train operators, from bus operators, from ferry operators. Multiple systems, multiple sources, and we try to expose them in a single way. So we get to do things after the data has been pushed into AT systems. They get the data from all these devices on vehicles, and then we try to integrate them into the public API. So we receive the data, update it, clean it up if we can, drop fields that are not necessary, try to avoid confusion from working with various systems, that sort of thing. So our role is in integration and the public API. As far as just working with AT, we are just yet another service provider, I guess. Were there other questions? Thanks for the talk. You mentioned, you were talking about ferries, like the data not being high-resolution or not being desired to be high-resolution. Does that sort of standard contain speed information so you could actually just kind of guess where the ferry was going to be at any point in time anyway? For ferries, ferries are a bit special. I'm not familiar with the devices on ferries. I'm not sure if there's bearing and speed in those sources. But for any of the vehicles, does the spec contain speed information or is it just where they are at a point in time? So I just recently heard my colleague mentioning that the bearing is missing from some of the buses. So again, it depends on the devices on the buses. I believe that AT does mandate the same device in all buses, but we are missing. It could be just one provider doesn't have this or doesn't have the newest version of the device or something. Yeah, for ferries, they're all, they're a story on their own. And there's problem with connectivity as well because they have to send data from the middle of the harbor. So that's unless they have really powerful antennas and the weather is not great, then you don't get any signal and you get no data. Just a quick, for the real time feed, are there events or attributes that describe when a bus is no longer picking passengers up? As is often the, well, as can be often the case. So there is one field that's currently not exposed. That's occupancy. So that's the only way you can really tell. So if it's on a trip, it should be picking our passengers. There is some providers to have the capability of giving us the occupancy data. Very roughly though, because we are missing still, oh, it's doable, but what we can do now is guess. We take the average size of a bus in the fleet and they give us the number of customers on board and we just say the transit feed, the transit spec allows for an integer between zero and eight, I think, two or zero is to seven, to say like zero is empty, seven is full. So it's a slider basically, and we say, we try to guess, it's probably full. We do know, we do expose this information, but just in New Zealand bus, because that is, I think first, they're the only agency that's asked. And second, because at the time when we were implementing this, there was the whole renegotiation of contracts and nobody wanted to share their occupancy data with anybody else. So, yeah, that's that. Did I understand you to say that the timetabling data is in a format which is actually a Google standard? Yes, yeah. So I was quite intrigued by that. I had no idea that they were sort of into that sort of thing and I was curious to know if it wasn't for the fact that it sounds like it has become a sort of world standard. Is it a satisfactory standard from your point of view or is it a sense of having it almost forced on you by the G? It's a relief to be honest, to have something. That's something that's used widely or at least more widely than nothing. So is it a good standard? It's okay. It fits most, almost all of our needs. It's a bit, the only gripe I would have about it was that they just want scheduled public transport in there. So if you ask Google about any features that are not scheduled public transport, they get, basically we don't support it. So we asked, I think I was looking for information about airplanes. So it seems like it would fit within the whole narrative, like having takeoffs and landings that would be a nice thing to have in there. They said, no. So theoretically you can put them in, nothing stopping you but as far as support or comments from them, it's like not supported. That said, it's a decent standard and it's used all over the place. I believe that Christchurch and maybe Wellington have a GTFS set of data as well. So it is widely used as well in New Zealand as well. And that's the only standard I'm not aware of. I'm just picking up on that a bit. Dedeen also had produced GTFS files, which of course they applied to Google. So if you use the journey planner or the Google journey planner, you can see when the Dedeen buses come and go. And I know there's been work done to upload. Apparently there's a site somewhere that is not Google but accepts GTFS files from sort of an open source way you can register and upload the GTFS files from your own location. And then that can be used by apps, transport apps that can work worldwide as long as the data's been uploaded. So that's the request that's currently in for Dedeen to put that up there so that some of the other transport apps can work. But what do you expect? I've got two questions. What do you expect? What kind of applications do you expect from making this API public? And what kind of things, what kind of uses do you think it will be put there if you've got anything in mind? Number one and number two, does your company work with any of the other regional councils in New Zealand? Would there be, because to my mind, having an Auckland specific thing is a wasted resource in a way, can't you share the work that you do and apply it to other regional councils either by doing it yourself or by doing similar work with other contractors who contract to the other places? So that is a kind of worms, really. And I'll just slightly open it up. And we have worked with ANSAR TA about the standards with a design for New Zealand-wide version of what AT has. So that has been slightly unsuccessful. So nobody said absolutely no, but it always comes down to the disparity, I guess, and the size of Auckland and needs of Auckland and everybody else. Some of it is easily replicable by anybody, including us. The static GTFS files, that is probably not a huge burden to create and maintain. As long as you already have these schedules somewhere in some format, you can basically, it's a one-time effort to get it into a single format, into the GTFS format, and then you can publish it to Google. And if nothing else, you have Google showing your bus stops on their planner. That is easy, getting real time up, not so much. Everybody's arguing with everybody else, and yeah, nothing's getting done. That is, again, one opportunity for this, since we are publishing GTFS real-time, if somebody can use that to create a valuable product that will be valuable to other agencies across New Zealand, maybe somebody else will be able to publish it because you do not have to publish the whole spec. There is a set of required fields that you have to publish. And if you just have a dozen buses, it's probably not that hard. We have time for one more. Could you, and it's okay if you can, but could you share with us the underlying solution that you're using, like, let's say, Python, Django, or Flask to run your API platform, just so that we have an idea of what you're building upon and how we can do the same. Unfortunately for this talk, I was planning on building a sample project, plus maybe a mini SDK, but real-life intervened. In-house, the APIs are running on Node.js. So that said, there's not much logic in there. So all the systems that those systems just receive the transformed data from the big back-end systems that get all the device information. That's all the time we have, so let's say time again. Yeah.