 Hi. Hi. Well, some of you were probably here, I guess, at my earlier talk on the high car augmented reality, but in case you weren't, I'll just introduce myself, give you a little bit of information about me. My name's Nick Whiteleg. I work at Solent University, which is in Southampton in the UK, and I teach web development and Android programming, Android development. So, I'm also a long-term open street map enthusiast. I've been involved with open street map since the very early days, since 2005, and a general sort of enthusiast of free and open source software. So what I want to talk about now in this talk is another project of mine. It's a bit more immature, really just starting than high car, but nonetheless I think it's quite interesting, and it's OpenTrailView 360. Okay, so what does this project aim to be? Well, it aims to be sort of an equivalent of Street View for footpaths. So obviously you've all heard of Google Street View, that needs no introduction, but obviously Google Street View, well, it's Google, so it's proprietary, you know, very much in the proprietary software world. And also it mostly focuses on streets, so streets, roads and so on. There's a little bit of footpaths off-road on Street View, but it's pretty limited. So I've thought, and in fact, as you'll see in a moment, this is not a brand new project, it's a resurrected project essentially. I've long thought that it would be a very nice idea to have the equivalent of Street View for footpaths, for off-road trails, which allow you to, let's say you're going on holiday, let's say you're going to somewhere in, I don't know, Germany, Austria, Greece, just to give you a few ideas of places I've been to in recent years, and you wanted to find out nice places to walk. You saw some trails on an OpenStreetMap map, so you were looking at OpenStreetMap to plan your holiday, but you didn't know maybe which trails were going to be the most spectacular. So it'd be nice to sort of use something like Street View to preview those trails before you go, and then the ones that look the nicest look the most spectacular, the prettiest, the most interesting historical sites or whatever you might be interested in, you could select them and then actually use those trails when you're on holiday. Or even, let's say, you wanted to see somewhere, virtually see somewhere that you might never get round to visiting, let's say, I don't know, somewhere a bit more far-flung like the Himalayas or the Andes, it'd be nice to go there, but obviously it's a long way, you might never get round to going there, but it'd be nice to actually see what it's like. So that's really the whole aim of this project. Okay, so as I just said, this is not a brand new project. I'll explain the story of it sort of starting and stopping and so on as I go on. This project, I first had the idea of this as long ago as 2010, and I actually gave a talk on the very earliest version, which differs from this version, in the state-of-the-map, open street map state-of-the-map in Girona, in Catalonia, northern Spain in 2010. Now, what this involved is it was interesting, people found it interesting at the time, but the problem was a very high barrier to entry. So at that time, things were a bit further back compared to what we've got now, and what you had to do to contribute was you had to take a regular camera, so stand somewhere and take several shots of standing one direction, then slightly different, slightly different, slightly different, so on. Maybe get about 10 shots, 10 individual regular photo shots, and then you had to manually stitch them together using some sort of third-party stitching tool. The standard in this area still is the standard by the looks of things is Hug-In, which is an open source, basically photo manipulation software, which allows you to do things like stitching. So you had to use Hug-In, and at the time, it didn't really seem to auto-stitch very well. You had to manually adjust the stitching, and it was all quite a long-winded process. It could be done, but it might take half an hour to just prepare your panorama, so it was a cylindrical panorama in this version for upload. So interesting idea, but didn't get taken up that much, and I got a bit fed up of the long-windedness of contributing photos, so I kind of left it at that stage. So that was my reflections, which I've just said. So the project saw no further work for a while, but then a second version came sort of on stream in late 2013, which I called OpenTrailView2. So this was around the time that photo spheres became popular. So as you may know, the stock Android camera app allows you to take these 360-degree photo spheres, which is basically essentially a sphere of the entire view around you. So what it allows you to do is basically take one... It's all done in the camera app. It takes one shot, then you move around again, move around again. So you take several shots, but the advantage over the original approach in OpenTrailView1 is that this is all stitched together automatically. And it did gain some interest at the time, and it did get a few contributors, a few sort of photo spheres were uploaded, some guys in Germany. I think Germany was the most popular country at the time. So that was nice. It was easier to contribute. You just uploaded your photo sphere, and that was fine. The disadvantages, though, was that obviously not everyone has got necessarily an Android phone, and it's still somewhat long-winded to take a photo sphere. You've got to basically stand there, you know, rotate round, take one load of photos about here, another load of photos about here, a few sort of viewing the sky, and then a few round the bottom like this, and you might get your feet in the photo sphere if you're unlucky. So a bit nicer, a bit easier to contribute, but still a bit long-winded. So taking photos, it's fun for a while, but if you go out and want to take 20 or 30, which is what's probably necessary to contribute well to this project, then it again gets a bit long-winded. Okay, so again, that all took place in late 2013. Now, that brings us to today. So this talk was originally scheduled. I did put it in as a lightning talk. It got accepted as a full talk. Luckily, I do have enough stuff to talk about, and I've managed to get a working basic prototype up, so there is something to show for this. But my interest in restarting this project now, it was first of all actually through one of my students. I've got a postgraduate student who's wanting to do something vaguely similar, but for streets and using 360 video rather than photos. So 360 cameras have become more popular in recent years. So cameras which basically have got these two lenses facing one direction and another. So to take a panoramic shot, you just need to take the photo by clicking a shot. And that gives you these two sort of half-stheres using these fisheye lenses. And then you can use stitching software to join those together into a photosphere like an equiractangular panorama. So the work necessary to create a 360 panorama using these 360 photos is a bit less. And there are some models out there for reasonable prices at the moment. Samsung Gear 360 and Ricoh Theta SC seem to be the leading models. So the stitching, I believe, I'm kind of looking at getting a Ricoh Theta. The only problem is that I've seen some slightly variable reviews regarding the reliability of its accompanying apps. So for the moment, I'm just borrowing a Samsung Gear 360. But if anyone has used the Ricoh with success, I'd be interested to get that feedback. So the actual camera is fine, but I've heard problems with the app, the app which is used to basically stitch the photos together. So if anyone has used that camera, I'd be interested in finding out more. Okay, so what's the consequence? Well, the whole barrier to entry is less now because you can use these 360 cameras. And I've started developing the project again. Before I talk about the technical details of how it's now working, I just want to cover some of the other alternatives that have come on stream since I originally started the project. So Google Street View is the obvious one, but obviously that's not for software at all and limited off-road coverage. There's Mapillary and OpenStreetCam. So I'll just quickly discuss those two alternatives. Mapillary, first of all, that's an interesting platform. Basically it's aiming to capture street level imagery and it's aiming to basically connect the imagery together so you can navigate from one image to another. So it's quite a nice project. The imagery itself is available under an open license, but the platform isn't. So it's not FOSS software. The actual website or the apps are not FOSS. There is an API available for third-party projects as commonly as in these sorts of fairly proprietary things, but the actual platform itself isn't FOSS. So I think that's enough given that we're looking, given I'm looking for a completely FOSS solution to this, that's enough to essentially consider that Mapillary isn't doing, fulfilling everything we need. And the other thing about Mapillary is it's not particularly focused on walking routes. It's basically general imagery, whether it be street or whether it be trails. Okay, the other alternative that I can see at the moment is OpenStreetCam. That's in many ways got several advantages over Mapillary. The entire platform from what I can make out is FOSS, so both the apps used to contribute and also the actual websites. However, its focus is not really at producing an open source street view. Its focus is mostly, and this is a very good focus, but it's slightly different to what I'm aiming to do. Its focus is to capture street-level imagery to then use AI techniques to actually analyze that street-level imagery for things like street signs, shop signs and so on so that that data can be contributed to OpenStreetMap. So obviously it's a very good project with a very good aim, but slightly different to what I'm doing. Okay, so is there still a need for OpenTrail views? So I think I would say yes, because it's doing something slightly different to both Mapillary and OpenStreetCam. And another thing that I thought would be really nice is obviously in street view, you can navigate from one photo to another. That's obviously one of the key things about street view. What would be really nice is to automate that in OpenTrail view by using underlying OpenStreetMap data. In other words, your panoramas essentially snap to a nearby OpenStreetMap way and then we can route from one panorama to all the nearby panoramas and therefore auto-connect the panoramas together. Now, the alternatives don't appear to be doing quite that. The original iteration of OpenTrail view required panoramas to be manually connected, which worked but again was a bit long-winded. So I'm going to talk about how exactly I've done that so far. So I've already implemented a demo of that and it essentially works to a limited extent. So I'll talk about that. So my latest iteration of OpenTrail view is called I've entitled OpenTrailView 360 because obviously of the advent of 360 cameras and users can upload so you can already upload panoramas as you'll see later on. So basically 360 panoramas in equirectangular projections, so essentially the same projection as photo spheres. So photo spheres still work. You can still upload a photo sphere, an Android photo sphere and the idea is also you can upload a stitched 360 photo from a camera as well. So what of the stitching? Well again I believe the Ricoh will do that automatically, the Ricoh Theta. The Samsung gear appears not to but you can use either the Samsung application to do that but unfortunately that's Windows only which if you're coming from a Linux world like I am is a bit problematic. But there are some third-party tools. So the one I've given you there on GitHub, Gear 360 Pano, that is a script, basically a shell script that will automate the stitching of the two images from the Gear 360 into a complete panorama using hugging. So it actually uses hugging to do the work but the end user doesn't have to actually use hugging directly, they can just use that tool. And it does seem to work with a couple of caveats that I'll come on to later. Okay, so this is what the focus of the work I've done so far. This idea of being able to connect the panoramas automatically using underlying open street map data. So how do I do that? Well, basically, again it's going to be using routing. So routing's come up quite a lot today. I used it in Hi-Car and then obviously Peter was talking about graph hopper earlier on. So what the aim is to do is to create a graph of the open street map data. So again, junction nodes in open street map become the vertices in our graph and then they're linked together using edges. And what I've also done is I've created a graph and I'm going to talk about how I've created the graph shortly because that's interesting and the actual panoramas themselves get inserted into the graph as vertices. So the graph we produce contains junction nodes from open street map but also the panoramas themselves. So in that way you can route directly from one panorama to another over the graph. So what approach have I used for creating the graph and doing the routing? Possibly the most well-known is PG Routing which is a Postgres-based routing algorithm which works on a Postgres database with PostGIS enabled. I did consider that initially as the approach to use but then I discovered something really nice, something very powerful and also certainly for my use case pretty efficient in terms of time, something called GeoJSON Pathfinder. I just want to talk about that. So what's GeoJSON Pathfinder? Well it was written by a guy called Pear Leedman and what it does is client-side in-browser routing using JavaScript. And if that sounds slow, it's not slow, certainly not for small areas. So what Pear Leedman did was he produced a demo for his home city of, I might get the pronunciation wrong but this is what my geography teacher from school called it Yurtaburger in Sweden. I hope I got that right. My geography teacher told me that anyway. But basically he produced a graph for his home city and it's very efficient, very fast. You can route from one point to another and it takes like sub-second essentially. So I thought based on that demo I thought I'd investigate using this myself. So how does it work? Well basically it takes as input GeoJSON line strings. It's a very standard low barrier to entry input. No fancy formats, just plain old GeoJSON. And from that GeoJSON Pathfinder then generates a graph basically using junction nodes. And also it contains an onboard implementation of the Dijkstra algorithm so really it's meeting everything we need. So I thought well this will be the way to go about it. And particularly as the routing in my use case is really over a very short area, very small area we only need to route from one panorama to adjacent ones. And if there are several panoramas along a particular path then obviously that area is very small. So this describes the algorithm that I use how I'm using GeoJSON Pathfinder. But I thought I would... It might be best if I go through the diagram because I think it's probably better explained visually. So basically the application, the OpenTrailView application is a mixture of PHP on the server side and JavaScript on the client side. It's got a PostGIS database but the PostGIS database is very lightweight. It just contains the panoramas, their locations and their heading. In other words what's the bearing of the centre point of the panorama. So we start by basically having a latitude and longitude. So on the server side we then find the nearest panorama to that lat and lon. The server then sends back some JSON so JSON containing the details of the nearest panorama. In other words it's ID, it's lat, it's lon and it's heading bearing. What we then do is we retrieve the OpenStreetMap GeoJSON corresponding to our lat and lon and again that's retrieved as XYZ tiles. It actually uses the same web service, the same web API as hikar uses. So the same web API that I talked about earlier on today. What we then do having got our panorama nearest that lat and lon is we then request nearby panoramas. So what are the panoramas within say 500 metres? So we get all the nearby panoramas, again they're returned as JSON so all the nearby panoramas are described as JSON and returned to the client. Having received all the nearby panoramas what we then do is root to each nearby panorama. So that's using GeoJSON Pathfinder. So having prepared our graph which we do from our GeoJSON we use GeoJSON Pathfinder to root to each nearby panorama. And then having calculated the root we actually get the panorama image itself off the server so in other words the panorama nearest that lat and lon and then we return that to the client. I'll talk in a minute about how the panorama is rendered and then we add the bearings to all the nearby panoramas to the rendered panorama on the client. And in that way it will show a panorama and it will also connect it to the nearby ones. Okay, right now there are a couple of things we need to bear in mind when rooting to nearby panoramas but it could be that we find two in the same direction. So in this example we've got our current panorama in the middle and we've got four nearby panoramas within range A, B, C and D but the thing is C and D to root to C and D we root along the same way, the same OSM way. So clearly we don't want B, we don't want the current one, sorry not B, the current one to be connected to D because it's not connected directly to D it's connected through C. So what we do is we filter out if we have more than one panorama along the current along the way in a particular direction we only take the nearest and we filter out any further ones in that direction. Okay, so what about showing the panorama in the browser? So again what did I do originally in the early versions over from Trail View? Well basically there were a few sort of rough and ready demos available on the internet at that time for showing panoramas in a browser, basically JavaScript based demos but they weren't particularly stable and they weren't seeing a lot of active development. However since then again things have matured and there's now a very nice panorama viewer called Penelom written by a guy called Matthew Petrov and this nicely displays these equirectangular sort of photosphere style panoramas in the browser and also it has a very nice tour feature in that its API will automatically allow you to connect one panorama to nearby ones so that suits our needs very well. Okay, so what do we do about positioning the panorama? Now what we can do hopefully is to actually extract the lat and lon from our panorama directly. If you're using a photosphere you can certainly do this because photospheres include the GPS latitude and longitude as exif tags. So the JPEG format can store metadata in the form of exif tags which describe different properties of the photo such as its latitude, its longitude, the time it was taken and so on. So there are two exif tags of interest here GPS latitude and GPS longitude which we can extract to get our lat and lon and then there's a further extension called XMP tags which I think were actually invented by Google originally but I might have got that wrong. They were certainly adopted early on by photospheres and from that you can get a very useful piece of metadata called pose heading degrees which is basically the centre point of the panorama. So if you want to link the panorama to nearby panoramas you need to know the bearing of the centre point of the panorama so that you can put your links to nearby panoramas appropriate places. Okay, so what will it do currently? As I said it's in the very early stages at the moment it's not as far developed as high-carb by any means but nonetheless it is up there. It's available at www.opentrailview.org so what you can do, I'll probably give a quick demo at the end finds the nearest panorama to a latitude and longitude now at the moment there are only four test panoramas up there because it's just gone live so not much up there at the moment and you can navigate from a panorama to adjacent panoramas and back that works but again the same caveat as high-car the navigation feature will only work if you're in Britain, Ireland or Greece. Apologies for that but again it's the cost of hosting like high-car any offers of hosting or any pointers to cheap hosting would be most welcome because I really want to make both this and high-car certainly cover all of Europe, certainly. Okay, now uploading panoramas is possible so even if you're in a country which doesn't support the auto connection yet by all means upload your panoramas because then once I do have data for your country they will work so if you're in Germany, Belgium, Netherlands, France, Romania wherever else you might be do upload your photos anyway there's no login system yet at the moment anyone can upload a panorama but they have to be moderated before they go live for pretty obvious reasons so anyone who uploads an inappropriate panorama it will just be deleted Okay, so what won't it do yet? Well obviously at the moment no navigation in other parts of Europe or the world now if your panorama doesn't have the EXIF metadata at the moment that's all that as far as it goes what the next thing I want I'm going to do is I'm going to add a web map interface so basically a leaflet interface to allow panoramas to be positioned and that will probably involve essentially positioning your panorama on a map and furthermore you'll be able to rotate a photo icon so that it's pointing in the correct direction so that's what I'm aiming to do in cases where your panorama doesn't have the required metadata you'll be able to add that manually via a web map interface later and it doesn't offer a complete walk through just yet from panorama to panorama so you can start at one panorama and go to the adjacent ones but what you can't do yet just yet as I said this in very early stages is go from panorama A then to B then to C then to D then to E that's not quite working yet but it shouldn't be too difficult OK, that's more or less it but I'll just give you a very quick demo of the site because I think we've got three minutes before questions is that correct? Yeah, that's correct So I just need to provide a latitude and longitude so at the moment the Wi-Fi seems to have gone a bit slow but at the moment it will it's normally faster than this and here we go so what it does at the moment is it finds the nearest panorama to your Latin lawn even if it's say 100, 200 kilometres away this is one of the test ones you may well recognise this here I was trying to find somewhere where there were no people around but this is basically the edge of the campus so you can kind of see that and see that OK if we change that to take an IED and then just to quickly demo OK yeah, a bit slow but never mind OK, so while it's loading OK, here we go so this is somewhere where you've got the link together this is somewhere near Southampton you've got your panorama here and then you can link to an adjacent one like that and again it's a bit slow but OK, any questions in the short time we've got left? yeah yeah yes that's well that's fine so photos could be tagged with seasons so you could just add tags as this winter as this summer as it's spring or autumn and then furthermore the user could actually either select what season they want to view a tap or they could even detect based on the current the current date and time so that's something I have thought of and it's something that I is one of the sort of to-do lists yep, yeah