 We evaluated PocketParker in a couple of different ways. One of the first things that we did is we built a system simulator. And that allowed us to experiment with different parameter settings and with also with different types of parking lots because we wanted to evaluate the effectiveness of our modeling approach when faced with parking lots with different features. So we actually looked at five different types of parking lots. So for example, the fast fill parking lot is a type of lot that suddenly experiences very high arrival rates and so it fills very quickly and then some point later in the day everybody kind of leaves at once and the lot empties rapidly. And this might be associated with a place of work where employees have pretty set hours. On the other hand, this low churn lot over here, this lot actually never fills and there's constant inputs and outputs to that lot. So we suspected that the low churn lot and the high churn lot were going to be the most challenging for our approach. Partly because the low churn lot makes it very difficult to estimate the monitored fraction because we never see a full zero to capacity swing in the capacity of the lot. On the other hand, a lot that never fills like this low churn lot may not be a good candidate for our technique anyway because if it's never full, you'll never have a hard time finding the spot. So we looked at the relationship and this is a graph showing as the monitored fraction increases for the different types of lot, whether it's a high churn lot or a fast fill lot, the accuracy increases. And there's a little bit of weirdness here towards zero because in certain cases zero essentially will overestimate the amount of time the lot is free and that can also, that can improve the accuracy a little bit under certain conditions. But in general what we see is this nice trend where at some point, you know, without having to monitor a really high number of drivers, maybe only 20%, I still get pretty reasonable accuracy. And what this is computed over is we ask the model questions about whether or not the lot has a spot available in it or not at that moment and we compare it with the ground truth maintained by the simulator. So 90% correct predictions means that 90% of the time when we asked the model whether there was a spot available, it gave us the right answer, either that there was not a spot available or if there was not or there was if there was. But, you know, we're systems people and so we are not satisfied with simulation results. We want to look at how things work in the real world. And so we deployed Pocket Parker, we built an app out of it. We experimented with a number of different features of the app in terms of the activity recognition and other things and there's results in the paper describing those. And we deployed this app for 45 days in several parking lots surrounding our building. There were 105 people that eventually used the app at some point during that time and the events that we studied fell into a four-day period where we did some camera instrumentation of the lot. So here are 217 parking events that we recorded during this four-day interval and the reason why we chose four days out of that longer interval was because we actually needed to ground truth the system. So this may be hard to see but this is a picture showing that we added some cameras on the building and then a couple of graduate students looked through, I think these cameras are pictures every minute and they looked through the lots to determine whether or not there were spots available and they did this for thousands of images. So this was a fairly labor intensive process to ground truth the system. And what we found was that the system could actually be fairly accurate. So on the fourth day you can see here that we have a correct prediction rate of about 94%. And in this case we break down the errors in terms of false negative meaning that there was an opportunity to have a parking spot and we missed it because the system claimed that there were no spots in the lot and also what we call waste or false positives and these may be a little bit more problematic because what this will mean is that some will actually go to the lot, look for a spot even if there's not a spot available. But on several days, not sure exactly what happened on day one but on the third three days of the study we had between 81 and 94% accuracy which we think is quite good. The monitor fraction was computed in the same way that we would do normally using swings of the lot and given the number of users that were using the system, the capacity of these lots and the positions of those users in terms of what they do with the school we think that those monitor fractions are pretty reasonable actually. And this work got picked up in a couple of places. This paper was published at Ubicom 2014. MIT Tech Review did a little write up of the project and since then we've actually been contacted repeatedly by companies that have been looking to commercialize this. I think that this idea would actually work very well in practice. I think the biggest trick though, and this is a challenge that many of these types of crowdsourcing systems face is getting up to a higher monitored fraction. So if we could get this in the half of the phones out there in the world then the modeling and the prediction challenges start to go away. And so I think there's actually a really interesting lesson in here for people that are building crowdsourcing systems because there's this sort of virtuous cycle where as you get more users in many cases for a system like this that's relying on crowdsensing the system gets more accurate. But how do you convince people to use a system until it is accurate? And so I think there's interesting challenges for the mobile systems community to look at and also in the IOT spaces. We start to have more sensors out there in the world. How can we somehow give apps a boost so that they can get over that initial curve and how also can we detect which apps are actually going to succeed in certain cases certain approaches may not work even if they're widely deployed. And so it can be hard to tell the difference at the beginning but telling the difference at the beginning is really critical because we want to support the apps that are going to work when they're really deployed and not the ones that won't. So overall we were fairly happy with the performance of Pocket Barker. We think this is a simple system as the accelerometers and the GPS on these smartphones have gotten better and as smartphones have seen wider adoption a system like this should actually become way more practical.