 All right, well, good morning, good afternoon, good evening, depending on where you're joining us from today. Welcome to Engineering for Change, or E-for-C for short. Today we're very pleased to bring you a special segment in E-for-C's 2016 webinar series focusing on mobile data collection. My name is Yana Aranda and I'm the Director of Programs here at Engineering for Change. Now I'll be your moderator for today's webinar. I'd like to take a moment now to tell you a bit about the Mobile Data Collection series. The widespread availability of mobile communications offers international development researchers, practitioners, and students new tools and techniques for collecting field data and determining success of projects. So, we've partnered with the Development Impact Lab at UC Berkeley for a series of six webinars to introduce a sample of survey software tools and demonstrate how to implement each tool in practice. For a recorded introduction to the series, please visit the E-for-C homepage. Today's webinar is the fifth in the series featuring premise and introduced by Joe Reisinger, the CTO and co-founder of premise. Our final webinar in the series will be with open data kit or ODK presented by Weylon Burnett on April 13th at 3 p.m. Eastern Standard Time. If you would like to make a recommendation for a specific platform, future topics, and speakers, we invite you to contact the series team via the email addresses that you see on the slide now. Now, before we move to our presenters, I'd like to tell you a bit about E-for-C and who we are. E-for-C is a knowledge exchange platform and global community of over one million designers, engineers, development practitioners, and social scientists leveraging technology to solve quality of life challenges faced by underserved communities worldwide. We invite you to join E-for-C by becoming a member. Members enjoy access to relevant and current news, professional development resources, including jobs and fellowship, and a growing database of hundreds of poverty alleviating products in our solutions library. E-for-C delivers a unique user experience based on user site behavior and engagement. Essentially, the more you interact with our site, the better we are able to serve you resources that meet your needs and interests. We invite you to join our passionate global community and contribute to making people's lives better across the world. Check out our website to learn more and sign up. We're excited to collaborate with Phil on this and future webinars. Phil is an international consortium of universities, research institutes, NGOs, and industry partners addressing global poverty through advances in science and engineering. Phil is headquartered at the University of California, Berkeley, and was launched in 2012 with support from the U.S. Agency for International Development through the U.S. Global Development Lab. This leverages the innovative capacity of world-class universities to design development solutions which help with new technologies, with novel economic and behavioral interventions. Phil calls this approach development engineering. Now the webinar you're participating in today is part of our professional development offerings. Our webinar series is a real-time and on-demand resource showcasing the best practices and thinking of development practitioners. Information on upcoming installments in the series, as well as archive videos of past presentations, can be found on our webinars page, as well as on YouTube. If you're following us on Twitter, I'd also like to invite you to join the conversation with our dedicated hashtag, hashtag E4C webinars. Now a few housekeeping items before we get started. Let's see where everyone is from today. So in the chat window, which is located to the bottom right of your screen, please type in your location. And I'll start us off with my location. I'm joining everybody here from New York this morning or this afternoon. So if you don't see the chat icon, just click on the top right-hand corner. Any technical questions or administrative queries should go into the chat window. In the webinar, please use the Q&A window located below the chat to type in your questions for the presenter. Again, if you don't see this, you can just click the icon at the top right-hand corner of the screen. And I see we have folks joining us from the UK, from Washington, D.C., from Jersey. Thank you so much, everyone. Now for the following webinar to request a certificate of completion showing one professional development hour, PDAH, please follow the instructions on the top of the E4C professional development page and the URLs listed. If you're listening to the audio broadcast and you have any trouble, try hitting stop and then start. You may also want to try opening WebEx in a different browser. Now with that, I'd like to take a moment to introduce today's presenter, Joe Reisinger, who is the CTO and co-founder of Premise, a mobile technology platform orchestrating and crowdsourcing the collection of microdata for precise and robust economic measurement, which we'll hear more about in a few. Joe holds a Ph.D. in computer science from the University of Texas and spent his academic career building natural language understanding systems at Google Research and IBM T.J. Watson. Prior to co-founding Premise, he was chief scientist at MetaMarkets. So with that, I'd like to welcome Joe and turn it over to him. Great. Thanks, Yana. I'm super excited to be here. My name is Joe. I co-founded Premise. Today I wanted to share with you all some ideas about modernizing statistical data collection, particularly targeting local expertise in the field via mobile smartphones. So the world and human society more generally is becoming increasingly more complex. The drivers of this ranging from things like increasing interconnectedness and span of international trade to the rise of extreme weather events because of systematic climate change. To keep up with this relentless rise in complexity, the way that we collect data and track global economic trends and understand the world has also evolved significantly over time. The technology platforms that we use are getting incrementally more sophisticated. The U.S. Census started with door-to-door surveys in 1790, more than 226 years ago, and this was backed by a constitutional mandate that continued in more or less the same form for nearly 200 years until the first mass paper surveys that people filled out themselves was mailed out by the 1960s. This was a key technological innovation in the 60s was that there was this free response. Around the same time during the span between the Great Depression and World War II and after saw the rise and adoption of telephone surveys as the telecommunications networks started to span the country from coast to coast and made sense to call people up directly and gave a very interesting and useful sampling frame for doing that. These were things of measuring inflation expectations or unemployment. The first attempts to call mobile phones and online line phones for economic surveys happened in the 2000s. One of the biggest and arguably most successful attempts was the current population survey for started experimenting with calling mobile phones in 2004. Now, 12 years later, we're basically living in the future. We're in this period of extraordinary technological innovation and there's this profound proliferation of new data sources, the web, mobile apps, satellites, drones, Internet of Things, and other embedded sensors. All of this can help us derive an even richer understanding of the world. This technological sea change, this rise in mobile phones and the web and social networks has reduced the friction of point-to-point information dissemination and it really knits together entire populations, irrespective of borders. Premise is situated as part of this conversation. We're building technology to orchestrate a network of people, local citizens, and to orchestrate them collecting structured, verifiable information about the world using inexpensive and ubiquitous Android smart phones. We're doing this now in 34 different countries. The data that we collect helps us better understand the world, track economic and human development trends in real time and these include things like price, pricing and market statistics, inflation, resource security measurements, or electrification patterns in the development world. The workflow on this slide probably looks familiar from a survey development, from survey development at one end through data capture and validation to presenting composite, you know, aggregate analytics and insights. We use this platform to distribute research tasks directly to data contributors to collect and analyze the data and then as we're able to sort of preview analytics and patterns of the data as it arrives, we're able to adjust the questions or survey instruments or even the sampling frame itself really at any time with a very, you know, only a very small amount of effort. Nothing is fixed definitional in this model and there's a very, in my opinion, revolutionary dynamic feedback cycle here. The underlying technology platform basically links machine intelligence to, you know, local human intelligence by a robust, highly instrumented, articulated channel. The work that we do is made possible because of a single key technology and that's the mobile smartphone. This is a photo of a data contributor on the premise network, performing a pricing task in Abuja in Nigeria. A simple gesture, this Instagram gesture of taking a photo with your phone is immensely important and it's something that's fundamentally new and transformative. You know, this really didn't exist even 10 years ago, so in less than a decade, you know, smartphones have become ubiquitous, each has, you know, more computing power than the Apollo project. You know, today there are 4 billion smartphones in use by 2021. The estimate is that over 80% of the globe, 6.4 billion people will have mobile smartphones. And this device, you know, really is their tether, their lifeline, their transmission cable to the outside world. Not, you know, not just purely for communication but, you know, for so many on the premise network, you know, using their phone for the first time to earn meaningful incomes and participate in the emerging information economy. So I wanted to take a look at some of the information that we can derive from using this data collection channel. So these are, these are just a small sample of photos from some recent submissions by CMS contributors capturing food stable prices in Brazil. This rolls up into a much larger consumer price index, consumer price index. Likewise, you know, the technique, the technology applied equally well to much more complex survey models. These are example submissions which show homes in, you know, regional city in Kenya that are either connected to the grid, electric grid or not. We use this model to build a map of electrification patterns across the town. And the data contributors went door to door, you know, verifying information and gathering feedback from residents, understanding kind of their willingness to get on the grid. We're able to design and deploy this project via the Android app in only a few days. I'm going to try to switch to a demo. Hopefully, hopefully everyone can see this. So this is just sort of, this is a mapping tool that we use internally, you know, the town that we did one of the electrification surveys for. And, you know, here is just on the map kind of house by house, whether the, you know, whether the houses are connected to the grid or not. We can actually kind of drill in and understand the determinants of, you know, what drives this adoption, you know, the, we can look at specific types of structures and see, like, how, see how the electrification patterns correlate, how the actual layout of the town itself correlates with, for example, the wall type of the buildings or, you know, the size of the structure. And along the bottom here, you see essentially the evidence. These are just the actual photos of the structures and the data contributors providing evidence that there is, for example, there is a drop connection from the grid to the particular residence or places. What does the survey look like? This is sort of what the app, the in-app experience looks like for a permanent data contributor for this particular task. You know, you're faced with a list of a list of survey items, you know, scouting buildings, the electricity, the electricity, et cetera. And then as you walk through the task itself, you're asked to provide photographic evidence and then provide us that a metadata about, about what you, what you found. Another sort of, you know, complex and to my, to my mind, extremely interesting example, you know, last year we received a grant from the Bill and Melinda Gates Foundation to do financial inclusion mapping in Nigeria. And so I'm sure everyone, webinar is aware, like more than 2.5 billion adults don't have access to a bank account or formal financial services, you know. And so in addition to verifying a set of existing financial access points, BMG wanted us, wanted to basically understand, you know, what was new, what had changed since the last time they had run this survey. You know, the goal is to help them understand the best places, like physical locations in which to invest in financial access infrastructure. Of course, the, and of course the idea being that by increasing access, you can help bring people out of poverty. Our contributors discovered 12,000 new financial access points. And then we took this new kind of more comprehensive map, a layer of it with transportation patterns, income levels and basic demographic data to create kind of a more holistic map and model of financial inclusion. So I'm going to try to show that one. What that looks like, a lot of financial access points coded, color coded by type, you know, banks, mobile money, kiosks, microfinance institutions, et cetera in, and this is in Abuja, Nigeria. And what's, you know, and so, you know, these are, you know, essentially, each one of these is essentially a, you know, an actual surveyed location. And the cool thing is, you know, we can use this aggregate data to kind of build better maps of, you know, what are areas to target, you know, what essentially are the areas which have, you know, good access to these institutions, and then what are the, you know, sort of population, what are the demographics of those areas. So you basically build these geospatial models using the data that we've collected. And, you know, what this sort of looks like is, again, so looking at the evidence, you know, you have different kiosks, here's a mobile network operator that was identified. And, you know, we have, we know exactly where that is on the map, and then Promise, of course, collects, you know, a huge amount of metadata about the survey itself. That's super useful for validation, verification, but also understanding the contextualization of these places. One, we did in the Philippines for the World Bank, and they, you know, the Philippine Finance Ministry wanted, was looking to monitor compliance on a fairly major tax policy implementation around cigarettes and spirits. Basically, they rolled out a, basically, they're, you know, they're trying to capture tax revenue from these so-called SIN items, and this is really critical to ensuring the sustainable funding of their universal healthcare program. You know, we're looking at, they're basically looking at the, they're basically implementing, it's basically implemented by holographic tax stamps, and the estimate is that every one percent of tax stamps, every one percent of leakage, you know, leads to roughly $20 million in additional revenues that's lost. So data that we captured here is really critical in identifying compliance and designing better programs to increase revenue capture. You know, this initiative is sort of intended to empower, catalyze, empower the Philippine government, civil society with the necessary tools to quickly and scaleably and transparently monitor compliance. And, you know, towards that end, essentially, we built not only the monitoring, not only the monitor during implementation on our, on our platform, but also the, but also the aggregate analytics and dashboarding required to make sense of, you know, to actually make policy decisions on the back of the data coming in. So this is the dashboard that's actually live on the Philippine Finance Ministry website. And you can see, for example, over the course of the last year, so you can see the rise in compliance with the mandatory tax stamps kind of rising from something like 50 percent up to near 100 percent penetration. And, you know, we can do this across, we can very easily kind of break this down by different, by different brands and try to understand, like, how different manufacturers, how different, like, cigarette manufacturers are actually kind of how they're actually acting on, on this requirement. The interesting thing, I think, to my mind, like, one of the most interesting things here is not only can you see this over time, but you can actually, you know, we actually build maps of, of this as well. See if it loads on the slow Wi-Fi. If not, so here's, here's the live, you know, here's a live map just of cigarette prices in the Philippines by, you know, coded by, by, by, by state, essentially, looking at the average price per individual cigarette. And then we can break this down farther in some of the metropolitan, some of the major metropolitan areas. The cool thing is kind of when you combine, you know, the longitudinal and cross-sectional monitoring, you can kind of make these cool, you know, videos where we can monitor tax compliance by region over time and you sort of see the, it's a little slow, a little fast. You can sort of see how this, how this, how this spread out region by region. So where, you know, so what's next, right? Like where, where can we go from here? You know, I can offer sort of like one, like small glance, you know, throughout this talk, we've been looking at, you know, just a massive, we've been looking at a plethora of images from the, from the from parasitic contributors. This, you know, this, and we probably can't see very well, but like this misshapen blob is a two-dimensional representation of all, all images of rice taken over the last 14 days, as seen by premise contributors around the world. This is the, you know, this is like the rice, this is the rice image manifold, if you will. So we're using, you know, deep learning technology, you know, the technology that so recently, you know, became the world champion at, at, at, at go and powers, you know, the visual recognition algorithms faced on Google, you know, can be used to extract instructor information from these photos directly in real time to help us understand the physical world. You know, the, the patterns and characteristics in this 2D representation are discernible even to the human eye. Imagine, you know, just imagine if you could visualize the full 2,000 dimensional manifold, you know, what, what types of even richer attributes and characteristics could we automatically detect using, using visual information extraction? You know, for price statistics, as one example, you know, as products and service availability becomes more homogenous, but also highly variegated, we are increasingly relying on key docs and characteristics assessment for understanding what drives pricing. We could actually use the technology to extract these types of attributes directly. And likewise, you know, for venue classification, you know, for the mobile, you know, for the financial inclusion example, you know, actually automatically, accurately classifying and characterizing venues and physical infrastructure for helping contextualize the evidence that we're pulling back from the world. So we're still, it's still very early days for us. We're exploring kind of the combination of a bunch of new technologies, but you know, we're super, you know, we're super passionate about, we're hoping that the technology that we are building will fundamentally upgrade that understanding. Thanks very much. Thank you, Joe. This was really fantastic and such a rich presentation in a short period of time. And at this time, I'd like to invite our listeners to go ahead and submit their questions via the Q&A. We already have a question that's come in, and that's actually one that I was thinking as well, which is, you mentioned working with organizations such as the World Bank and the Bill and Melinda Gates Foundation. How might smaller NGOs or researchers at universities work with premise? Yeah, that's a great question. And we've, you know, it's definitely one that we're keenly interested in and part of our medium and long-term strategy is around opening up the platforms such that you don't need to be a large institution in order to work with us. You can just, you can have access to the platform more directly, design your surveys, implement and execute them in areas where we operate. We've piloted a couple of, we have a couple of successful pilots with academics in working with food security researchers or working with folks in international development, implementing a pretty wide range of monitoring activities. And it's something that we're definitely super committed to. There's a lot of additional infrastructure and technology rollout, I think before we can really scale that up. But even now, I mean, if there are folks that see premise kind of fitting into their activities, you know, we would love to talk further for sure. That's fantastic. I guess I can serve as an invitation to our attendees to consider that offer. So in terms of some of the findings that you've had to date with the organizations that you're working with, are most of them published or available freely to anybody who is listening to this podcast or webinar right now, or are they often just directly for institutions or some components of them available? Yeah, so it's a mix. Many of our customers are extremely committed to open data. So there's a fair amount of stuff that is freely available and downloadable. And we like also kind of company-wise, we're very interested in, a lot of this data has more value. Chef data is sort of like a, in general, a set of data is a many ways of public good. And we're pretty committed to making it available. The issues of course, like on the dark side of open access data, of course, is privacy. And for us, because we operate a human network, we're very sensitive to issues about contributor identity and protecting their privacy. And so that's sort of the complicating factor, but in general, we're working, actually kind of working with some, we're working with some interesting researchers on particularly in differential privacy, looking for ways to make kind of as much data as available as possible, while still, you know, while still protecting the identities and livelihoods of our data contributors. But it's something that we're super committed to, for sure. Okay, good. And we have another question that's come in. How do you verify differences in food commodity varieties of the same commodity, for example, employed rice versus local rice and so forth, as this impacts price differentials? Yeah, that's a great question. So we, so specifically for, I mean, I can talk to some work we're doing, measuring purchasing power parodies. In those cases, you know, we take a specification-based approach where the contributors are asked to price very specifically different types of rice, you know, different grades of rice, essentially. So broken rice versus standard grade, medium-grade rice, like depending on the country. You know, and of course the characteristics of those specifications drive pricing pretty immensely. It's super important to have, you know, so training and kind of understanding the problem are super important. And then, you know, we actually can use, we can also use the photographic evidence that we received back in order to contextualize and understand those types of things as well. But it's sort of, you know, it's a mix of both, you know, kind of training the contributors to recognize and understand what actually matters in the survey and then also kind of verifying and understanding what comes back directly. Gotcha, and there's a kind of a degree of specificity that's increasing for the specific question regarding the differentiation and we're gonna kind of layer onto this. How do you accommodate for alternate measurement units or weights, for example, by pile or, sorry, lost the question. Your photo in Nigeria shows commodities in a pile. Oftentimes these weights very arbitrarily are by type of commodity. So I think you already started speaking to the notion of training your contributors, but do you want to maybe add a little bit more color to that? Yeah, for sure. So in general, you know, we don't, we're sort of approaching this problem like Tabula Russell. Like we don't actually know a priori how even markets function in many of the countries that we enter. We actually rely on multiple phases of discovery and tuning to, so one, just finding where are the markets? When are they open? How do they operate? Who are the traders? Finding the set of products. What are the important characteristics of products that drive price? How do we refine the surveys to make sure we're actually accurately capturing indicators for those drivers? And so that process is very iterative and it's automatable to some degree and we're getting actually much better at automating it. But in general, what it looks like is you collect a large number of prices and you collect a large number of prices that you are relatively sure are accurate and you find, for example, bimodalities in that distribution. So then that hints at the existence of additional attributive characteristics that aren't necessarily being captured. So then we have to go in and kind of refine the questions in order to account for that additional dispersion. But that whole process is very iterative and I think there are very interesting ways to automate it and we've done some amount of that but a lot of it's just like relying on having a good active channel to the data contributors, having them tell us. Like, hey, you asked this question but I can buy things in all sorts of different sizes and which one do you want me to actually measure? So it requires having that open dialogue. So if we can pull in that thread a little bit and with respect to those contributors that active tremendously large network, how are you engaging contributors and in terms of compensation, what is the model that you have for that? Yeah, for sure. The basic model is you, anyone can download the app and on board onto the platform. We're limited to 34 countries but generally anyone in those countries that can on board. Once you're on the platform, you'll have access to these types of micro work tasks and you'll typically be paid per capture. And the actual payouts of those captures are variable and they depend on the whole slew of factors. Typically the biggest driver of cost is how far you're gonna have to travel yourself to perform the task. And so depending, like if I'm asking you to travel to the store next door versus I'm asking you to travel like 50 miles away, like that's gonna really clearly determine kind of the value proposition. And so a lot of the platform is really geared towards understanding what are the factors and drivers of task uptake? Like what do I need to do? What do I need to offer? What's like a fair price for work and how do I do that across so many different places? As the contributor gets better, as the contributor kind of graduates up through levels, we do have, we maintain a core of community managers and other individuals who help us kind of smooth over a lot of the fundamental issue of course, like running this type of company from our perch in Silicon Valley is that we don't know, we're not the experts on the day-to-day experiences of the vast majority of people on our platform. So we rely a whole lot on having that type of open communication channel and having very smart individuals help us, like explain to us why the things that we're asking for don't make sense in their context. So really the platform itself is a mechanism for kind of vacuuming up that understanding of the world. Thank you, that's very insightful and we've had a few more questions come in so I'll make sure to address those. So this is a really exciting field and this is something that the question asker has noted and I completely agree. And what the individual is interested in is whether you utilize your powerful contributor network to undertake surveys on more human-centered economic development, particularly this listener is interested in it from the position of program evaluation. Yeah, so I guess it's definitely worthwhile to think a little bit about the limitations of what we do. One of the, so the easy things for us are operating in public spaces in a more or less like unobtrusive way, like documenting passively. Those types of tasks are generally the easiest and have the highest uptake. On the other end of the spectrum, you have highly invasive tasks where you want to do a, you wanna do a consumption, expenditure survey or household survey and so you're going door to door and you're asking, you have a very complex survey instrument. That type of stuff, I would argue that this platform is less well-suited towards cases where you need highly, highly trained enumerators who have experience working directly with other people. At least the 2016 version of premise, we're not confident enough in, there's many dimensions to this, but we're not confident in building a platform that sends random strangers to your house. That's a, it's not a, without more vetting and understanding of the individuals on the network, it would not be a quality experience for anyone. That's, we do operate, we do operate a few surveys where we have operational partners who are experts in, who are expert enumerators. And then in those cases, you know, you, basically you have, well, you know, you're implementing in GEO or whoever it is, training, you know, training and vetting your workforce and then you kind of, you just onboard them out of the premise platform and then have them perform the survey task. And then the benefits of that are, you know, you don't get, you don't get access to a large crowd, you still have to do the training, but the benefits are having standardization and having kind of the platform and app experience. Absolutely. So, and one more question, speaking more largely to the strategy for premise, what countries has premise worked into date and what countries do you expect to be working in over the course of the next year? So we operate in about 34 different countries now. It's a mix of, you know, it's a mix of Sub-Saharan Africa, Southeast Asia, Latin America, you know, we operate in a few countries in the developed world as well, you know, the United States and sort of bits of, bits and pieces of Europe. The, you know, in terms of expansion, in terms of expansion, the plans are essentially to, you know, really, you know, we're really kind of, I think, generally very, very focused on those three areas, Latin America, Sub-Saharan Africa and Southeast Asia. So probably the bulk of our expansion over the next, the next year or so we'll be targeting those areas. And we're looking to expand probably from 34 to maybe 60 or 70 by the end of the year. Wow. It's a fantastic one. I'm sure delights many of the listeners on this webinar as they're like, we're interested in engaging with you. So then I think we've covered the bulk of our, let me see, I don't see others at this time. So if we haven't covered your question or you have a question out there that you'd like to ask specifically, please do feel free to follow up with us via the webinar's email address listed on the slide. For those of you who are interested. With that, I would like to thank Joe for joining us today. Thank you so much for sharing information about Promise. And I'd like to thank all of our attendees for joining us today as well. For those of you who are interested in having the PH for this session, please use the code listed on the slide. And with that, I'd like to close out the webinar and invite you all to join us for the next one. Thank you so much and have a good afternoon or good evening or good morning depending where you are. Take care.