 Hi, my name is Anurag, and I'm a product manager at Google. And today, I'll be talking to you about my experiences building a product that somewhat defies intuition. What did we learn in the process, and how can you apply it to some of the product development that you do? Now, over the last five or six years, I have been dealing with the area of volunteer crowdsourcing. Now, this is an area where people come together to solve a content problem or some other problem. For example, Waze has got all its maps crowdsourced. All the content on Waze is crowdsourced. Likewise, Wikipedia is a great example of crowdsourcing. A lot of people's intelligence comes together to build something that's useful to everybody. The important part over here is the volunteer part of it, which is what I've been focusing on. That means we're not talking about the likes of Mechanical Turk, where you get paid to do what you are doing, but where you come in because of your interest or because of your passion or so. Let me tell you about some of the products that we built and our journey through it. The first product that we built about six years ago or so was the sync of fan captions. And we built it in collaboration with YouTube. This is where anybody can caption any YouTube video in any language in the world. The great words is you go to a certain section in the community part of YouTube, and you get this UI where you can draw boxes that span the length of the caption, and then you can type in the caption in whatever language you understand, ensuring that a number of people look at a particular caption before it goes live, and those people have to understand that language and so on. And so we built a nice algorithm that takes into account people's credibility and skill and whatnot, and we have checks and balances to make sure that that stuff doesn't go live. So this system was made live, and it was working nicely. And the question that came to everybody's mind that we talked to about it was, well, why would anybody do this? Like when you go to YouTube, why would you caption a video spending several minutes or hours captioning a video because it can be a tedious work? Why won't you just enjoy YouTube, right? And we have, obviously, an answer to that, but in order to really understand this, you have to understand the story of some of our users. So Sofia, her story comes to mind. She's from Bulgaria, and we talked to her because she had spent time contributing eight caption tracks. That means captions on eight different videos. It took her a few hours to do that, and here's the bad part. None of our captions have gone live. And when we talked to her about it, we understood her frustration, of course, but she was actually pretty joyful about it. And she was like, look, I contribute because I really care for these captions becoming available in Bulgarian. And I know this problem will be solved eventually, so I'm going to continue doing what I'm doing because I really care about people in my country who speak my language but do not speak English to be able to watch those videos. Now, that's a different kind of enthusiasm altogether. But I'm not as good a storyteller as this guy. He is destined family. He runs an amazing channel on YouTube called Smarter Everyday. You should definitely watch it. And here he is talking about how he enabled fan captions on his channel and then how his community is helping with that. So let's listen to him. So selfless that it doesn't really compute. I want to tell you about a guy that's done something for me and other people. His name is Eden. And for the past year, he has been translating the captions from Smarter Everyday into his own native language. And now, all of the videos, like every single video of Smarter Everyday is now available with Hebrew captions. And I think that's incredible. I've asked him to shoot a video, turn the camera on himself, and just tell you why he did it because I think that's fascinating. That is a significant amount of time that he spent. I want to know why on earth he did this. Hey everyone, my name is Eden. Or Eden, I suppose, if you happen to speak English. And I guess you're wondering what these are. Well, I spent some time translating Smarter Everyday to Hebrew. And these are there because Smarter Everyday is now actually a multilingual channel. I got the idea for this project from Andrew Jackson, who? So the important thing to note over here is that Destin is not actually explaining himself why these videos need to be captioned. He's letting somebody else who is bilingual and who has himself put in the effort to caption those videos, explain what motivates them and what drives them and why they think this is important, and so on. We'll come back to that part later in this presentation. I want to talk about another product that we built. It's called Translate Community. And this is where anybody can go in and translate simple phrases into their language or from their language into English. The data from this helps Google make translate better in those languages. As you know, the quality of translate varies depending upon the language. And that's a direct function of product data that we have available. And what people are doing over here is providing more data in their language so that translate can improve the quality of Google Translate in that language. Now, once again, the question comes, like, why would somebody take time out of their day and contribute those to, let's say, playing your favorite video game or chatting with your friends or whatnot? So this is where, like, when you travel around, you talk to some of the people who are doing this, you find out that language means a lot to people. In Bangladesh, for example, language is patriotism. And when we organized the Translate upon in Bangladesh, people showed up with the data of the national flag and contributed thousands of translations. In Thailand, there was a month that showed up and there was this old gentleman that showed up. He didn't even know that he's supposed to enter them in a computer, so he actually wrote down a whole bunch of translations and brought them in and saying, like, here, this is how I would like to help I get better in Google Translate. So this is the kind of passion that you only find out when you talk to a lot of people and really understand how they think about these problems. So as we found out more and more about this, we found that, well, there is this common passion about my language, my region, my culture. And people wanna make things better for that. And Google products may not always work better, may not always work as well for people in these regions as they do for people in the United States or Europe. So rather than have these scattered experiences through YouTube and Translate and so on, would it make sense for us to just kinda collect them all and launch an app that allows people to improve the quality of their language for several different technologies that they touch and use every day? So that's where we launched the CrowdSource app. And in the CrowdSource app, there are a bunch of these delightful little UIs where you can help with things like verify labels on an image or capture images of things around you or help us understand the sentiment in a sentence or help us understand writing. So all of these things are tied to understanding of your language or understanding of your culture or understanding of your environment and so on, and is catered towards the diversity of the world. Now, when we look at the passion of users that motivates them, it's not always about the big picture. It's not always about, I wanna do this thing for my language or for my region or for my culture or for my country. Sometimes it's very, very personal. Hossan over here tweeted that his motivation for using Google CrowdSource is watching his father use Google Docs. His father speaks Farsi and he uses the voice interface to use Google Docs and the quality of voice recognition or speech recognition in Farsi is not the best. And so he uses CrowdSource to improve the quality of speech recognition so that his father can be better at using Google Docs in Farsi. Now that is very, very personal and that motivates Hossan. And likewise, there are different motivations for different people around the world. Some of them care about the big picture, some of them care about something that are very, very personal to them. So back to that question, which we get asked several times when we're going into meetings with execs and so on, right? And you will face this question every time you're building something that is counterintuitive. Why would anybody use it? What is the motivation? What about quality, right? And will we get what we're looking for and so on and so forth. Now as product managers, we know to rely on data, right? We dabble in metrics, we love metrics and we like to take our sheets of metrics with us. But no matter how many metrics you throw at people, there will be that lingering doubt in their mind, right? It's like, ah, but I still don't see and people doing this much. Now what tells a better story than metrics is, well, stories, right? So we let the users tell the story and the story over here, the passion that our users have for their language, their region. It's very hard to tell that story with metrics. It's much more significant when a user themselves stand up there and tell that story. This is what Destin did with his Hebrew speaking community person and this is what we did with some of our community members. We brought them in front of our leadership and let them talk about why they love crowdsource so much and why they care about their language and so on. So the result of that was we were able to convince our leadership that crowdsource is a great product to invest in because people really care about this and they want this. We started off crowdsource as a very bare bones product and it had just a few tasks in it and you could do those tasks but there was like really nothing much beyond that. Then what we did was we started talking to our users and found out what is it that they like or not. And what we learned was that while this is something that they would like to do and they're driven by passion but after a while it becomes very monotonous and it would be nice if it had some fun game mechanisms. So we built some gamification into it and we were careful and thoughtful about what kind of gamification to build into it. So we built some gamification into it, stopped and looked at what the user's reaction was and at that point of time we found that our users, well, they were delighted by the gamification and our metrics showed that but the thing that our metrics did not show that was super important for us to find out was that our users had started talking to other people about this and in their colleges or at their work or in their communities, they were showing this to other people and saying, look, Google is doing something about our language and we should download this app and use it. And they were bending into these little local communities. Now that doesn't show up on any of our dashboards. For this you have to go talk to the people and you have to understand what's really going on in the community. So each other, not just locally, but also around the world and learn from each other's experiences. And we found that those community interactions were more delightful for people. They learned from each other and they shared their questions and they shared their answers and so on. And very soon they started organizing these little events, these little walk-a-thons for contributing images or these little translated fonts for translating content and so on. So we helped them with Shrag and so on so that they can get more joy out of these events. And that's how the whole community grew. And today on Facebook, this is a very thriving community with over 200 cities having local communities. Now, talking to our users, we also found out that they had this lingering doubt in their mind. Like they could not quite make the connection in terms of how their contributions and crowdsource result in better products. So we started telling them about how Machine Learning works, how it manifests itself in their experiences in daily life and how it relies upon data. And why is it that diversity in data matters and how we have sparse in data that if they contribute, then we can make things better. So when all of this picture was complete in their mind, light bulb went off and they were like, ah, now I get it, right? And at that point of time, they became even more motivated and they spread the word of crowdsource even farther. So these are a lot of things that we learned by talking to a wide range of our users. We talked not only to the most passionate users, but we talked also to the users who were kind of lukewarm about it. They were lukewarm about it because the whole picture wasn't very clear to them. But once the picture became clear to them, they became very, very motivated. So fast forward to today, crowdsource has collected close to a billion answers over the last four years from 10 million plus users who span pretty much every country in the world. They find it really delight user 200 cities. So back to the point of how did we get it from a product that everybody questioned to a product that's successful? Product that defines intuition, how did we make it successful? What are the three lessons that I learned from it? Lesson number one, tell a good story. People will doubt you when you come in with a counterintuitive product idea. You know it, but people don't understand it. Now you can bring metrics, metrics are great, but people make decisions based upon emotions. So before you bring in your metrics, learn how to tell a good story, become a masterful storyteller, appeal to people's emotions, help them understand the feelings of your users, the people that you are building the product. And once you're able to do that, you can back it up with metrics and make the decision so much more easier. I've provided a link to an amazing talk by Karen Eber on TED where she talks about how your brain responds to stories and why they are super important. I would highly recommend every product manager to watch this talk and take learnings from it and incorporate it into your own storytelling. My second lesson is to talk to a diverse set of users. Well, obviously US product managers know how to focus on users. The diverse part is important. The stories, the features that we built would not have been possible if we hadn't talked to the users of the world who were getting frustrated with our products. We wouldn't have figured it out if we hadn't talked to users around the world. So talking to a diverse set of users, not just your most passionate users, not just the ones who are always giving you feedback, but also the ones who are dropping off. And it's really tricky to get them. So we have to figure out how. But talk to those users and talk to them directly. Do not delegate it to a certain function in your organization. As your product manager, you have to be of user research and use it widely. And finally, the third lesson, it takes time. The journey of crowdsource happened over several years. And during these years, we went through a lots of ups and downs and we pivoted several times. Our team started with 12 people. It whittled down to just three and then we threw it back up to about 20 people. And this will happen in every product. You will have the ups, you will have the downs. So whether through the storms, right the waves. Now the storms will come, the waves will come. When you're riding the wave, make the most of the time that you have. Build relationships, encourage people, make sure their experience is enriching and do not overwork or burn out your team. So these were the things that helped us get successful at crowdsource. And I'm happy to share these experiences with you. My name is Anurag Bhattra. I'm a product manager at Google. Thank you for listening to me today. Please connect with me on LinkedIn and I would love to continue this conversation offline. Thank you.