 Rwy'n cael ei wneud, Nick Whiteleg. Rydyn yn bai o'r cyfnod o'r cyfnod, rydyn ni'n ddweud y ddweud y Ddweud i amser a yw'r ddweud o'r ddweud o'r Ddweud i gyfnodol oedd yn Southampton, yna yng Nghymru. Mae'r ddweud i'r ddweud o'r ddweud o'r ddweud o'r ddweud i'r ddweud. Rydyn ni'n ddweud o'r projec sy'n 2005, ac yn gyda'r cyfnod rŵan i'r cyfnod o'r ddweud. Oed, yn gwaith o dyma dros gyfnod o'r proiect이 ar bobl cyffredin a gyrdd. Rydyn ni'n ddweud o'r cyfnod oherwydd gobl o gyrdd cyffredin a gyrdd cyffredin, da wedi bod yn y ffordd ar hyn oherwydd oherwydd oherwydd ddweud o'r cyfnod o'r ddweud on a mobile device such as a phone or a tablet with computer generated data. So probably something like Pokemon Go is the most mass market example. Now there are variants of AR. I'm not going to talk about these in any great detail because they're not really on topic, but we have marker based AR and that is that basically uses computer vision algorithms to draw 3D computer models on paper markers. And a very good example of that is augmented reality chess where you've got a chess board with chess pieces and computer vision then detects the real chess pieces and draws computer generated models of chess pieces and also obviously can then work out how the chess game is being played and whether anyone's won yet. We also have marker less AR, which is again all about using computer vision algorithms to detect things like planes. Planes on the ground like a obviously a floor or a piece of grass or something like that, and that's been put to use through various AR apps such as Ikea have done one basically where you can model piece of furniture being placed in your living room and see what it likes. See what it looks like. But we're going to focus on geographic AR and that's all about using your GPS location and the sensors on your device. So the the compass, the accelerometer and so on to develop an AR app. OK, so what is the state of geographic AR? Well, quite a lot of work's been done on it. There are a lot of what I call points of interest AR apps where you can, for example, point your phone at a building like a famous building like, you know, let's say the EU Parliament or some of the buildings on the ground plus to think of some local examples and the AR app will give you some information about that building. So there are a lot of apps out there that will do that. Wicked huge is one of the earlier ones. AR nav is another one. Another another take on AR is show me hills, which is quite a nice app which shows you basically a point to a view from the top of a mountain. And it will actually tell you what the various other hills and mountains you can see are. So there's quite a lot of work out there, but a lot of it's closed source and proprietary. So that's not so good. I do get the impression that while there's a lot of work on AR at the moment, most of it is probably proprietary and closed source, which is not so good. Show me hills is an honourable exception that that is GPL. OK, but one little explored area of AR, which what I'm going to talk about is all about countryside navigation. So all about walkers and hikers, potentially cyclists as well, perhaps. Where basically you can use your phone or your tablet to navigate through the countryside. So I don't know what what your experience of navigating the countryside is, but often find in in the UK where signposting isn't always that good and some of the landowners aren't very friendly. Sometimes you're walking across a field or some woods and it's very unclear where the path goes. In some parts, the signposting is good in other parts. It's terrible. So it would be nice to actually have an augmented reality app where basically the course of the footpath, the route of the footpath is overlaid on the camera feed of the phone. And furthermore, it would be even nicer if we could have virtual signposts. So whether or no real signposts, we can instead have a virtual signpost showing you the direction and the distance to nearby points of interest, whether that be hills, mountains, pubs, towns, railway stations or whatever. A my project Hicar is aiming to do precisely this. OK, before I start talking about the project in depth, here are a few screenshots. So there are four screenshots here with basically the latest version or an iteration of the latest version that was in the autumn. So these are from the autumn, as you can probably tell from the state of the nature. So you can see that three of these actually show these virtual signposts. So the top left one, for example. Now Craig's probably going to tell me off for getting the pronunciation horribly wrong. These are in Northern Ireland. Sleave Donnerd is the highest peak in Northern Ireland. So it's telling you that it's 3.79 kilometres that way. So I actually use this. I actually use my app to use Hicar to navigate to the top of Sleave Donnerd, which I've never actually visited before. So it's telling me it's 3.79 kilometres on the left and then another peak, which I'll probably get the pronunciation horribly wrong, Sleave Commoda. OK, that's good. 4.03 kilometres that way. The one on the top right, it doesn't show any signposts, but it does show you and it gives you a hint of some of the issues of the app, namely that the placement is pretty good, but it's not perfect unfortunately just yet. I will return to that later. So you can see there is a real path there. Hopefully that's a dirt path and then that's a cyan line is the open street map because this is all from open street map. As I'll explain in a minute, the open street map way corresponds to that path. So it's a bit off, but it's probably good enough to navigate with. Bottom left is almost at the top of Sleave Donnerd. So it's another example of a virtual signpost. And the bottom right is in Newcastle, which is the town below Sleave Donnerd. So that just gives you an idea of what it looks like at the moment and the sort of information that is shown. OK, so we're now going to start covering a bit more in terms of the technical detail of how it was implemented. So it's Android. It's Android only. I do want to make that clear at this point. It is not available on iOS just yet, but again, I'll return to that later. So just to give you an idea of some of the APIs that it uses, some of the Android APIs it uses. So a lot of them are really standard Android APIs. It used a limited amount of third party APIs, but mostly it's just standard Android. So the Camera 2 API for basically getting the camera feed off the camera device. The Location API for getting your GPS position. The Sensor API for detecting the orientation of the device, which is critical. So are you facing north, south, east, or west? And also, how is the device oriented? Is it standing straight or is it at a tilt? So the Sensor API basically gives you a rotation matrix that represents the current orientation of the device in relation to the default. And that then that leads on to the rendering because that orientation matrix can then be used directly with a slight transformation because we're working in landscape mode in OpenGL. So we use OpenGL ES to do the rendering. So the data, as I've already indicated, is from OpenStreetMap. So obviously this is an open source project. So OpenStreetMap is what it's got to use. And obviously I'm a community member anyway. So that all makes sense. So that data is all from OpenStreetMap. And also some freely available elevation data. So that is pretty critical because obviously if you're walking in the hills, you want your routes, you want your paths to be superimposed accurately on the hills without appearing to burrow into them or appearing way above you or way below you. So elevation is critical. OK, and that's really what I just talked about. I've already gone through that slide. I think I will talk about the elevation data sources in a minute. So where do we get this elevation data from? There are a number of places I've used before settling on a final source, but I will talk about those elevation data sources subsequently. So basically just to clarify that, the rendered data, the rendered footpaths that you see have got to be in 3D because if we didn't have a z-coordinate, then let's say we're at say 300 metres above sea level. If all the z-coordinates were at zero, then the data wouldn't be at your level. It would appear to be some way down there and look very small and very distant. Likewise, if the z-coordinate was always a thousand metres, let's say in most places, it would be hovering above you like that. OK, and then that's what I'm aiming to do. It obviously doesn't look quite look that nice just yet, but in an ideal world, once I've got a bit further with it, that's the sort of effect I'd like to have. It's not that good yet. That's more a desired end product in terms of showing you the route up mountains. OK, so a bit of sort of history because what I'm talking about today is not the first version of Haikar. It's a project that's been around for about five years now. I've been developing it on and off as you often do with open source development. You know, I do have a day job as well, which is very demanding. So I work on this. It is now research for my day job, so I am allowed to do it at work when I'm not busy with other things. But obviously, you know, I worked on it quite a bit. Then I sort of stopped working on it for a while, but then because AR has become quite, you know, has had quite a lot of interest in the last couple of years, I've started developing it regularly once again, and I'm quite passionate about keeping that development going. So it was originally developed in 2013, and the original version, and I do want to emphasise the original version, is on Google Play. It will not show signposts. It's the original old version, but it is there if you want to see the older one. And the elevation sources that the original version used included Ordnance Survey, which is the UK's National Mapping Agency open data and also NASA SRTM, which probably you all know about it out of thought. OK, so how do I prepare the data? So it does not talk to OpenStreetMap directly. Obviously, the main OpenStreetMap servers are aimed at editing, not consumption of data. So it's basically using a very standard tool chain, really, using Osmosis, first of all. So I download data from Geofabrik in Germany, so the Geofabrik site by Jochen Toth and Frederick Ram. So I download country extracts from there. I then run it through Osmosis, the OSM filtering tool to select only the things I'm interested in, namely roads, footpaths and a small subset of points of interest. And then I import into a post-gist database. So again, it's the standard tool chain, really, using OSM2-PGSQL. So the post-gist database on my server is the standard one that you would also use for Mapnic rendering. And then I've written my own PHP-based web API to serve the data. So basically that's running on my own server, so you send it a query and then it returns the data back as GeoJSON. And then that data is then cached on the device so that once you've downloaded data for a particular area, you don't need internet connection again. Right now, this is what I'd say the number one issue with the app at the moment. And that's the availability of data. And this is where if I could get either volunteers or people who can direct me at suitable sort of hosting providers or storage, that would be great. My own server is something I pay for out of my own pocket. I pay 20-odd pounds for it per month. And due to the constraints of the server, it only covers certain parts of Europe, unfortunately. So that is Britain, Ireland and Greece. Reason why I've got Greece in there is that I regularly go to Greece for personal reasons. So that's why I've chosen those three countries. I'm very much not a Brexiteer, so I very much like it to be the whole of Europe. That would be great. But unfortunately I don't have the money to pay for the hosting to cover the whole of Europe, unfortunately. So if anyone can let me know and I'll give you my contact details later, any affordable hosting providers for that, that would be great. Or even if anyone has got any spare servers lying around that I could possibly use if you're particularly interested in this project. OK, so what I'm looking for in terms of somewhere to host it, really, is just somewhere OSM post just database, only needs to store highways and selected points of interest. And my API uses PHP, so that's what would be needed. Or if anyone knows of any existing APIs that someone else has written where I can get GeoJSON data of OSM. OK, so that's a bit about my own web API, just giving the GitLab link. So I am on GitLab rather than GitHub. I think I was a little worried about Microsoft taking over there, but just a personal thing. But yeah, bear in mind, all my source codes on GitLab, so that's something to bear in mind. So basically the web service, the API now takes XYZ format, so the standard XYZ tiles. So you send a request specifying XY and Z, just the standard Google Maps, the OpenStreetMap, everything else style, XYZ web micata sort of thing. And then it returns GeoJSON in the 3857 web micata projection. So that's what the API does. In terms of terrain, basically I'm now using the MapZen terrain tiles. So one of the ex MapZen employees, let me know about that. So terrain elevation is not a problem at all. I've got that access to that globally. So basically it's now provided as a free Amazon AWS service. So that's where I get the data from, the elevation data, and it's in the terrarium or terrarium ping format where elevation data is encoded in the RGB channels of the ping image. So that's where I get my terrain from now, following a few earlier solutions which I've now abandoned. So I'm all good with that. So basically it was something that MapZen made and then they put on a free Amazon cloud service. So that's all good. Right, a bit now on the virtual signposts because that's the really new thing that I've added, as you can see from the screenshots. And I would like to give credit here because I am not a 3D modeller. Those signposts models that you saw were generated by my colleague in the computer games animation group at my university Neil Bruwys. So many, many, many thanks to him for doing those. So how has this actually done? Well, this actually follows on fairly nicely from the previous talk, but on Graphhopper. I generate a graph. Now I did actually use Graphhopper for a while but I had to go through a, because Graphhopper uses OSM as its input source, I had to go through a rather laborious manual procedure to generate static graphs, whereas what I really want to generate is graphs on the fly to reduce the maintenance needed. So what I now do is I take the GeoJSON, the same GeoJSON used to render the data and from that I generate a graph. So what I do, similar, I'd imagine to what Peter's doing, I find the junction node. So I work out where there are junctions and then each junction node then becomes basically a vertex on the graph. And then obviously the edges of the graph become the links between the nodes. And for routing it's for basically, for graph management, I use another open source library called JgraphT. That's basically an open source library which allows you to efficiently store a graph. And it also comes with implementations of Dijkstra and A Star so I can do my actual routing using JgraphT. So that's just a quick diagram. So the things in black are the underlying open street map network. The blue is the structure of the graph so you can see that the graph is a simplified version of the network where each vertex on the graph corresponds to a junction node in OSM. And then my edges store the distance, the actual geographic distance between the nodes, between the vertices. And that distance is the full distance along the OSM way, obviously not as the crow flies distance between the two vertices. Then I do my junction detection. So what I do is when I get a GPS position, when I get a new GPS position, I then look at the graph and see if there is a graph vertex within a certain distance, I think it's 10 meters if I remember right, a certain distance of my current GPS position. JgraphT does actually have a number of useful API calls to do that so that's relatively easy. And then if I am at a junction what I do is I calculate the bearing, basically the azimuth, the bearing of each way radiating from the junction, each edge radiating from the junction and I use that to place my signpost arms. Now what I do there is I filter the edges because my signposts only show footpaths so I check that each edge, because each edge basically contains all the various OSM tags associated with that way, I filter the list of edges so that I only take those which actually represent footpaths, so things like highway equals footway, highway equals track, et cetera, et cetera and access not private. So I select those ways which are potentially walking routes. Okay, and then having done that what I then do is I find out, I basically work out, I search for points of interest within five kilometres of my current position. So that gives me a list of points of interest and then I use J graph T to route to each point of interest and once I've got the route I then basically compare the bearing of the first segment of the route with the bearing of each OSM way heading from that junction. So for example if the route to, let's say, I don't know, the Grand Place, let's say, was 270 degrees, not sure exactly what it says, 270 degrees and one of my ways from my current point was at 270 degrees then I would know that the route to the Grand Place would be along that way. So basically I add that point of interest to that way and then the arm, the signpost arm pointing along that way will then contain Grand Place. OK, so that's basically like that. So we've got a graph here, POIA is there, POIB is there, POIC is there. So basically by comparing the bearings we can work out that POIA and POIB are along that signpost arm and POIC is there. Right, a bit on weighting, different points of interest types get different weightings, so places are weighted heaviest, Hamlets, pubs and cafes and then lastly localities. That ballpark figures at the moment really what I consider the most important things when out walking. It's not extensively tested so those figures probably need a bit more refining. So basically the higher something's weighted the more likely it is to appear on a signpost arm. So I only take the lowest weighted essentially two points of interest along a particular route and add those two to that arm. OK, a bit of an enhancement would be exact routing to a point of interest because it may be a point of interest is not at a junction node. In other words it's not at a vertex on the graph. So in those cases what I do is I first of all route to the nearest junction node I think you can see there and then I walk along each OSM way leading from that junction node and then if I find a node along that OSM way which is closer to the point of interest than the junction node I essentially add the route from the junction to the node nearest the point of interest to my route because that is a potential problem points of interest might not be at a junction node. Signpost rendering, that's all done with OpenGL the actual letters the characters are done using an OpenGL texture and an interesting point is that I do support Greek and Cyrillic character sets as well so if you used in Greece it will work because obviously my data covers Greece so it does support Greek character sets and if someone wanted to set up data for say Bulgaria or Serbia let's say then it would work there as well. OK, remaining issues, battery life it's always a big thing in AR apps battery life often sucks the battery inaccuracies in obtaining GPS position holding the device steady so AR glasses is possibly the future maybe. Went next talk by the way is it 30 or 35? OK fine. Holding the device steady is another thing so AR glasses are an interesting new development and technology where obviously you can wear glasses like that Google Glass was an early somewhat failed experiment but there are newer models out there as well they still all look a bit sort of sci-fi but nonetheless I can see possibly in the future where we can get AR glasses that just look like regular glasses then that might be the best device to actually do this sort of augmented reality because you don't have to hold your phone steady. The other issue as you probably saw from the screenshots is that the computer generated objects float a bit which might look a little bit unrealistic it would be nice a future plan to use computer vision algorithms such as using OpenCV for example to actually overlay those on the ground more accurately so yeah that's the state of play 0.1 as I said is the old version 0.2 is what's been presented today it's available on GitLab under that URL it's working but I'd say it needs a bit more further testing so I haven't put some Google Play just yet I do want to test it a bit more before I do that there is however an APK available which I'll leave open at the end and 0.3 is a planned future version which will explore computer vision for example using OpenCV maybe to more accurately place the paths and the signposts hi Carl Libb I haven't run out of time a bit but my plan is really to make a lot of this code a bit more generic so rather than just focusing on walking focusing on any open source application of AR because I think at the moment a lot of the solutions out there are proprietary that is a big problem so I think I would like to start this hi Carl Libb project where I try and make the code a bit more generic so it could be used in other use cases such as city guides virtual tours of historical sites and so on and so forth so interesting contributing obviously a successful open source project really needs more than one contributor I think so who am I particularly after anyone with experience in OpenCV and computer vision I think will be really great to have any form of AR development experience just sharing experiences sharing knowledge particularly making efficient use of the battery volunteers if there are any volunteers who could help host the web service and increase coverage because I want to cover certainly the whole of Europe as a starting point you know the world at one point but Europe would be great and then the other possible contributor is an IOS developer because obviously it's Android only it would be great to get it working on IOS as well where well the source code is on GitLab so gitlab.com slash nickw1 slash hi car that's the source code I've got a preview APK for downloading that will work on Android 6 upwards so if you are in the UK, Ireland or Greece and you want to test it feel free to use the APK and as I said the old versions on Google Play right okay thanks so that is it so there my contact details I'm a bit of a newcomer to Twitter so there's not a lot on there at the moment but I am tending to use a lot more to keep you all up to date with updates okay thanks I don't have an estimation at the moment but I could probably work that out why do you ask? yes okay fine okay if you give me your contact details I'll try and work that out and I'll let you know yeah that's a good idea yes I'm sure I could do that yes I'm sure that would be possible probably open street map really in terms of points of interest open street map is really the main source yes I'm not planning to do that myself but I do have a colleague who is interested in using AR for climbing routes so yeah that would be it that's a great idea