 I love to welcome our speaker for tonight, Joris. Give him a round of applause. Thank you. Everyone, I'm Joris. I work at Google as a product manager. I've been at Google for a while for about six years. Done a number of different things there. I've worked in sales and marketing for two years. I worked in finance for two years. And now a bit over two years I work in product management. And specifically I work in what we call the next billion users organization. Which is where we try to build better products for people in emerging markets. Out of curiosity, I wonder what people do in this room where they come from. Engineers in here? Quite a few students. Quite a few as well. Now I've got to guess. Any other product managers? Quite a few as well. What companies? Where are you from? It's called Elevate. It's a big audit firm. Awesome. What other roles do people have? What are people doing? Data analyst. Data analyst. Okay, awesome. Cool. Okay, so a nice mix of different kind of tech oriented people. Working with the... Yeah, some reason it's like... Sorry. No, that's fine. So yeah, I've been with Google for some time. I studied back in Amsterdam where I'm from. Did a master's in economics. Studying taking courses at Stanford. Graduate courses in artificial intelligence. Which is pretty interesting. Very different type of thing to be doing. Extremely technical, but a ton of fun. But I'm going to mainly be talking about what I do here at Google right now. And how we at Google go about trying to build great products for people in emerging markets. Which I think is a pretty tough challenge given that we are this U.S.-based company. We all kind of are in our little bubble in California and then you try to build products for an audience that is entirely different from your surroundings. Sorry, I'm blocking the screen. Is this better? That's good. So it's a pretty tricky challenge really. And I'm just going to tell you about what have we learned from doing this over the last couple of years. How are we going about it? How do we do research? Which is an absolutely critical component of it. Is this not working? No. Sam, the Chromecast is not working. Yes, exactly. We're passing on that to you. So I don't want to screw it up anymore. So this slides have very little information on them. The text is not going to get any smaller than this. So I think we should be good with this screen. But maybe if people really can't see it, maybe you should come. There's some chairs here as well. Yes, so I'm going to be talking about that mainly. So a couple components that I'm going to be talking about. First, about where the users are. And you'll quickly notice why I'm going to be talking about that. I'm going to be talking about what we've learned by building, trying to build better product for emerging markets over the last couple of years. I'm going to provide some tips from Google to build for these audiences. And some tips on research, which are more my own experiences in doing research and what I learned from that, what worked well. And feel free to interrupt me anytime, by the way, if you have any questions or whatever. So internet growth is happening everywhere. It's definitely not just the US. Actually, there's not that much happening in the US. It's happening on a very global level in a lot of different places in the world. So if you look at the total number of internet users, so this is the nominal amount of internet users by country, back in 2014, you see a couple places light up. US is one of the places. China is another place. And some other places you see some darker shades. But it's pretty much those two places that have the bulk of the users. If you go forward to 2016, you can see one other country particularly pronounced there as well, which is India. You see Brazil popping up a little bit as well. Nigeria in Africa is coming out a bit. So that's kind of the current number of users that we see in those countries. I think a more interesting view than this is to look at the new internet users. If you look back in 2014, it's already very clear US is not where that is happening, right? That's not where people are coming online. It's happening in China. It's happening in India. There in Africa, Brazil. This is a little distorted because it looks at the total number of users. So there's a ton of other smaller countries where you also see a lot of people coming online. But the crux of the message is that it's particularly happening in these kind of emerging markets that we talk about. Scroll forward to 2016 and you can very quickly see why there is one country that we focus on primarily. It's India. Last year, more than 100 million people in 2016, more than 100 million people came online for the first time in India. Those people did not come online on computers like we did. They came online on smartphones. That is their, as we say, primary computing device. That is the sole way that they access the internet right now. So 100 million is a big number, but there was a second time 100 million new internet users came online in India. The year before, we saw the same number of people coming online in India. So these kind of numbers, they're pretty mind-blowing. Forecasts say that by 2020, we might reach a billion unique mobile subscribers in India. That is, again, to me, a pretty mind-blowing number. That is kind of the whole population of Europe, the whole population of the US together and then still you're not at one billion. And that is just this one country, right? That is just India. So if you hear me talk about India a lot, you can kind of, I hope you understand why that is. That's the place where the huge number of new users are coming online. So this is kind of a clear consequence of this. We see hundreds of millions of people on Chrome, the web browser, as well as Android, the operating system, coming from these next billion user countries. And again, with next billion user, we mean kind of any country that roughly meets the emerging market criteria. Three typical ones, Brazil, Indonesia and India. And they're all right now in the top 10 countries. If you look at the search volume that we see on Google Search, so the total number of searches, these three countries are already in the top 10. And as you can imagine, they're going upwards as well, right there. As you saw with these numbers of internet users in India, more and more are coming online. So more and more are going to be searching as well. These are huge, huge numbers. Also interesting, by the way, is sometimes it feels a little wrong almost to be talking about countries as diverse as Brazil, India, and Indonesia as kind of one block of countries. And there definitely is an extreme amount of diversity between all these emerging markets. However, and I'll touch on that later as well, there's also a bunch of things that they definitely have in common, particularly technology-wise. And even within India, you could argue it's not really one country. The cultures are extremely diverse. Different regions have different languages, different languages, different customs. It's a very diverse country. So what have we learned over the last couple of years trying to build great products for people in those markets? Now, this is one of those things that we see kind of as a common thing among people in these markets. If you look at the technology in particular, is they have very low-end smartphones. Because again, we're building, I'm not sure if I should have made it clear, we're building software for smartphones, right? Internet software, apps, those type of things. They have very low-end smartphones. You can buy new smartphones in India for $50. That's probably more than ten times cheaper than the smartphones that we have. So you can imagine that the kind of specifications that these smartphones have, they're a lot lower than what we are used to. We saw this survey done by Western Digital and it just found that a third of smartphone users in India, they have daily storage issues. So I don't know how many of you have issues on your phone with storage running out. I've had it now on my phone after like a year of usage. In India, a third of people have it every day. So it's kind of on a whole new level of how pressing these type of issues are. And I think this is a great example of something that does span across these countries, these type of things in Brazil. Actually, yesterday we ran a study on this in Brazil and we saw over 20% for this as well. Another thing is people are on slow internet. Wi-Fi is not very widely available in general. So mobile connection is the primary means that people go online and that they use internet. Right now, still about half of people in India are on 2G. And to give a comparison of what 2G is, because I don't think I ever actually used 2G myself, this is pretty much the internet. I looked it up just today. It's pretty much the internet I had 20 years ago when I first used the internet. It's that kind of speed. You can imagine that 20 years ago the internet was kind of built for having that type of speed, right? But given that the current mobile internet is very much built for people in the U.S. that are connected always, this is a really slow speed to be getting all your stuff from the internet. And with that it's also good to... Yes? Or the device memory that they use is lower than average. What's the reason? It's both. So we studied this quite a bit actually. You see that they definitely have less storage than we would have on our phones. But they also, they save a lot more on their phones. We are kind of very used to having things in the cloud and things in our email, et cetera. For them, they make sure they save it on their phone because they're not always connected. So they have a ton of media on their phone. I'm generalizing, of course, but in large trends that's true. Also, I think those are the primary two reasons. So lower end hardware and saving a lot more stuff offline on your own phone. And again you have to realize that for them their smartphone is pretty much what for us is our computer. This is their only way, their only piece of technology to access the internet and to save their files. So they use it in a much more, in a way in which we would use our computers. Yeah, sorry, not a question. To add to that, I mean, they're basically running a four or eight case of memory. And they are constantly sharing a lot of things. Yes. A lot of files. That happens, yeah. One more thing to add. So I've used smartphones from a very low range for 3,000 to 4,000 rupees. Two, I'm spending now for 3,000 to 3,000 on one phone. That's the main reason that in India that they use most memories that the phone has to offer because we don't have, as you said, the internet quality is not good. The internet speed is not that good. So if we want to access our files very little we have to store them in our phone so that we can access them offline. That's the primary reason why we run out of storage. Rather than some files you already know. Okay. Just for the record. Okay. Sorry. Yeah, I completely agree. I apologize in advance. I'm going to make a ton of generalizations. Okay. Yeah, completely agree. Yeah, just for people that didn't hear it, like the diversity, particular urban versus rural is huge in terms of productivity, phones, et cetera. Sorry. Are any of these issues similar in China or completely different China? I don't actually know that much about China, to be honest. The way, I've read about China quite a bit. The way sometimes kind of as a rule of thumb people say you can look at China as kind of five years ahead of India in terms of or five or ten years ahead of India in terms of development. So you see the same kind of trends, but they're further ahead. Just one more thing to add of the internet. At the current stage, India has better internet than the United States. We get 4G internet. Yep. One big burden. Yeah, with Geo, right? With Geo. Yeah, yeah, yeah. No, it's true. You have this mobile carrier Geo which is changing the landscape super rapidly in India. That's definitely true. To be seen where it goes over the next year, the next two years is a sustainable, et cetera. But that's definitely true in India. That's the thing that's at place, yeah. Okay, I'm going to continue. So this one other thing is also when I talk about India, you can kind of see it as a... You can kind of apply this to a lot of other countries, right? So where some of these issues in some areas in India are less pressing right now, than in many other countries there at this same level. So another thing you got to think about then is what we consider to be kind of a free app for many people is not really considered free. If you look at popular apps such as Facebook or CleanMaster or other apps that are super popular in India and elsewhere, these are like 40 to 50 megs, I think. If you pay for every megabyte that you download, which a lot of people do, I think, that's not really free anymore. In particular, the data costs, they differ very much over countries and over regions, but you got to imagine that if your income is, let's say, at one-tenth of what our incomes are here, but the data costs approximately the same, then all of a sudden you're looking at... If you'd have the same amount of usage as we do, you'd be looking at several hundred dollars worth of monthly mobile bills, right? So you're going to be a lot more careful about what you spend your money on. So downloading anything, in general, people tend to be a lot more careful about it and they tend to avoid large apps, large files, etc. So yeah, that's one kind of advice that we give and that we take ourselves, is try to remove that barrier of downloading big things. There's a couple ways to do this. There's a couple ways to do this. One way is build a light app. That's what you see a lot of companies do, right? They have one app which is 50 megs and have one app which is 5 megs. That's a pretty typical solution to this. Other solution is build a very lightweight website. There's other solutions as well, like one of the called progressive web apps. There's a bunch of technical solutions to this, but I think the important thing to keep in mind is anything that's lots of megabytes is going to be pretty tricky. It's a bit of a barrier for people to be using it. So very closely related to that is optimized for speed. What for us loads in 5 seconds for people on 2G might very well load in 2 minutes. So 2-minute waiting time is a lot of time. People are a little more used to it in those areas, but obviously if you can trim that down to half or a quarter of the loading time, that's going to help a lot. And any improvement you make in loading time is obviously also going to benefit everybody in America and in other Western countries. So this is a very important one, built for intermittent connectivity. So you can see that I think also Google products in the past and many other products that were built in the US and in other Western countries, they very much assumed that you were online and that you were online with a decent connection. And once there was not the case, things didn't work well at all. However, the reality in many of the next billion countries is that there can be gaps in connectivity and these gaps can be very different amounts of time. Sometimes people don't have connectivity for a couple seconds, right? That happens a lot. Or sometimes people don't have connectivity for a minute. Sometimes they don't have connectivity during the day. They only have connectivity during night. Sometimes they only have connectivity every other couple of days because they don't top up their data. So they have a couple of days without data. So that kind of intermittent connectivity is extremely important, I think, to take into account. And so you really got to start thinking about, or at least for us, we really had to start thinking about not just an offline state and an online state but also kind of a bad connectivity state, kind of these intermediate kind of areas of connectivity that are extremely prevalent in those countries. I don't know. I noticed that Google Maps, for example, started doing this recently where in some situations it says, oh, you have a bad connection. It doesn't say you're offline. It says you have a bad connection. And then it kind of alters your experience to make sure that it works well, even if there's no data for 20 seconds, those type of situations. And again, just to highlight how used we are to having, to being always online, to me it's very telling that when I'm in an elevator, I'm like a little bit annoyed that I can't use the internet. I'm like, oh, it's like 20 seconds without internet and this is such a short period. Or on a flight, you really, at least I, I really, really noticed that I don't have internet for a couple of hours at a time. So just to mention, that's kind of a everyday experience for most of these people in those regions. So these are two pretty cool Google solutions that we have. One is on Chrome. You get this, if you're offline at that point, you can say, I want to download this page later. And we have a great little dinosaur game. I don't know if people know that. It's pretty awesome. This dinosaur and your jump runs. I spend a lot of time when I don't have connectivity doing that. And in Google search, we have this option nowadays that it will, if you're offline, it will push you the search results at a later stage. Which is also pretty useful. So I think these are, these are two, just kind of random examples, but these are two interesting solutions to dealing with this on off connectivity type of situation. So another thing, we kind of touched on it with the diversity in India, but languages. So if you just look at that one country, India, you folks probably know better than I do, but I think there's 22 or more languages that are... 26? 26. But then in the 26 languages, we have different regional languages, so it goes beyond how many languages. Okay. All right. That's anywhere between 22 and more than a thousand. A lot of languages in India. And that's just one country, right? So obviously the diversity in languages, if you want to target all of these users that are coming online in different places, it's just huge. So yeah, you got to make sure you do that. Because in India, I think it's also, it's about 10% of people that speak English and they speak it to different degrees. Correct me if I'm... Okay. Okay. Yes. Okay. So English users is going up a lot. There are 22 to 26 different languages we have in India. So they all don't speak the common language, which is... Even people consider Hindi as a national language. It is not. And they don't speak Hindi. So you have to communicate. We always use English language. We have some of my friends from the south of India. Hindi and I, we always communicate in English. Oh, great. That's great. Yeah, it's interesting. We have... Actually, the majority of my team has roots in India. And when we go out in the field and we do research, it's like, oh, great. We got somebody from India with us. He can speak to the people on the street. But it's hardly ever the case that it actually works out because of the languages that are different. So yeah, speak many languages. One interesting thing that we find also, just kind of a random insight, is that a lot of people, they want to have their software, such as their phone in English. But a lot don't actually understand the English very well. This is... I'm just saying what we saw in... Our research is a different... Yeah. Yeah. We choose to change it from English to Hindi or English to Gujarati, whichever the language that they are familiar with. So that happens a lot. Yeah. Yeah, we see that happen. We see a lot of people... What we're also experimenting with is seeing if you can do some kind of mix of some pieces in English, in particular the words that people are familiar with, such as, let's say, internet or download, et cetera, and some other parts in local languages. It's pretty tricky to get this right because some people choose to have their app in English but don't actually understand it. Some people choose to have it in another language, but then the translations don't work for them. So this is a really difficult one to get right. Yeah, yeah, we do. So it gets... The apps and the software that we built, it gets automatically translated. Obviously, there's translators doing it, but that's true. Yeah, there needs to be quite a bit of quality control on it. And it kind of depends... So I would argue our software in general, the local translations are pretty decent. I'm glad you said yes. But I think that's not the case for every app and all the software out there. Good question. So you said there are 22 plus languages that you're targeting. Yep. So wouldn't this increase the size of the app that... Yeah, yeah. ... how do you address that? Do you localize it for a region? Or do you ask them what language they want and then just come look at that part? Yeah, no, that's a great question. It's a really good question. It's something... There are solutions to it. Currently in some of the software that I'm involved in right now, we're not actually using any of the solutions, but you're right, in the size of, for example, an app that I'm working on, about 20% of that is just all these languages. It's a lot of the size. There are technical solutions to it. You can, for example, make sure that phones in different languages get different apps, different, definitely localized apps. You can indeed do regional ones. You can let users download a specific language in the app. So there are solutions to it, but you're right, this is a problem, and there's different ways to go about it. So there isn't one unified golden way to do this? If there is, I'm not using it. I'd love to use it, because this is definitely an issue for us. So other issues that you're going to work into is that you don't really know what language the user is going to want. It may very well be... It's very likely that his phone is in English, but then what language are you going to push? English definitely got to be there, but then you probably want to have a range of other options if he actually wants to have Bengali or Hindi or whatever. This is another thing that Google Search did around languages. I think if you search specific terms or specific regions, it starts showing your regular results in English, but it also shows the second tab where you get all your results in Hindi. According to the team, it got a 50% increase in the amount of Hindi that people were searching for. And this is another one. This one's pretty interesting to me. So... I'm sorry. Is there a way for me to simulate that in Hindi? Ooh, that's a good question. Maybe through a VPN? Yeah. This is another one that's interesting. When we build software normally for people in California or in the US or in the Western world, we take a pretty minimalist approach where everything's white and there's very few buttons and that's kind of the... If you just think about the Google Search page, right, it's extremely minimalist. What we do find when building software for countries such as India is you've got to do quite a bit more effort to guide the user certain ways to help them understand what they need to do, those type of things. So I think there's kind of two components to this. One is, to me, the most interesting, I think just general life in particular urban India is so much busier than general life is in, for example, the Bay Area, and that is reflected not just in kind of the day-to-day life on the streets and then going about your life, but very much reflected in the software that they use and the apps they have on their phone, those type of things. They tend to get way more notifications. There's way more animations happening in apps and it's kind of, to my eyes, a super busy experience but to a lot of people in India, that's the experience they're used to and that's kind of the experience they want as well. So that's something that we really got to adapt to. So there's one component to it. It's that you just got to be a little more clear about, try this. Press this button. Do a bit more effort to guide people a certain way. That can be done by bigger icons, brighter colors, those type of things. It's not necessarily sample size. So we do, we have a huge research team which does a lot of, we do all types of different research. So for something like this, so yeah, again, this is not, this is not one piece of research. We do, sometimes we do kind of survey bay research over many thousands of people. Sometimes we do infield research which I'm going to talk a little bit about later where due to the constraints that you have, you don't reach more than like two dozen people. Sometimes we do kind of very intensive one-on-one interviews and then you have a very small sample size. So it really depends on the type of research you're doing. So it's more of an observation of what we see people using and apps that are built in India and for India and the phone experience that we see people in India having reflect that type of experience. So it's a way more busy experience. I think it's a deal. You know we're doing more. Yeah, yeah. So rather than specializing it, it's generalizing. We're raising 30 people. Yeah, no, that's a very... Fair, no, that could very well be. I think there's a bunch of elements in here but I think your statement is definitely right in the sense that a lot of the apps that are popular in markets such as India but also China and other places are extremely feature-rich as in like they're kind of five apps in one and we're like, what is this app? What does it do? Oh, it does antivirus. It does email. It does checking my phone. So you were right. I think there's kind of also a tendency to have more functionality in one piece of software than what we are used to. So it's part of it. So the ads definitely make the experience busier but we definitely... So look at an app such as CleanMaster, for example. I don't know if you've ever used that but that's like hundreds of millions of users in India as CleanMaster. You've used it. Maybe I've used it when I was in India. You've used it in India? CleanMaster? Yeah. Cash cleaning. Yeah. It's pretty famous in India. It's very famous in India? Yeah. Yeah, it's one of the absolute top apps in India. But that has like a lot of animations and there's so much happening and like I get lost but it's an experience that works very well. For sure. Yeah, yeah, yeah. Great. Like you are going to specifically do, for example, make Gmail more feature rich about that or... You mean like expand the feature set? Yeah, like a Google app. So it's interesting. It is something that we kind of talk about like how feature rich should your app be and there are kind of different opinions on it. Some people are very much like, you do one thing, you do really well, nothing besides that. Other people are more like, oh you do a bit of that, a bit of that and you kind of go that way. There's different opinions on it. I think the general consensus at Google would be do one thing and do it really well. But obviously, to your point as well, as we see apps in India and in China being extremely popular, that definitely don't do one thing but do at least five things. It does also make us ask the question is this the right approach to take for countries such as India? I think this is kind of a very minimal example, but this shows this is a type of experience that we have in general on Chrome opening a new tab. It's very minimalist, gray, and you can start wherever you want. Then what an experience in India could look like is you give people a few more options, you guide them a little more towards certain directions. You agree with this? Great. So to repeat the things that I went over, I'm afraid if you have any questions on those because this is the end of that part, remove download barriers because downloads, they're not free for people. People really think about downloading. So as much as you can remove the megabytes, people are going to really like that. Optimize for speed. So just think about what your app would do on how your app would load or your website would load on a 2G connection and make sure that's at least somewhat decent for that type of experience. Address intermittent connectivity. Again, this is one I think is extremely important. Don't just think about online offline, but start thinking about all these intermediate states of having some connectivity every now and then. It's a very important one. Multilingual, we talked about a lot with a thousand plus languages in India. And guide new users more than you might do for software otherwise. So a question. Why go through the trouble? I mean, why does Google want to invest so much for the next 20? They mostly come from rural areas and there's many powers to this. So yeah, for us it's... So I wouldn't necessarily be able to comment much on the kind of larger business goals, but it's very clear that if you have a billion people coming online, that's important, right? Yeah, data, but I think more importantly... I think more importantly just the size of the groups of these people, and they're going to use search, they're going to use the internet, they're going to see ads. I think those type of things are important. Can you elaborate more on the value of search? What are the key criteria that you are looking to when you are on that aspect? Yeah, sure, yeah. I don't know if I have much more to say about it than what I said previously, but it's basically don't go for the minimalist approach that you might want to go for in a product that you built for here. So give users more options, give users... give them clearer visual cues to what to press, where to go, what to try out. I think those would be the most important. So an example of this would be, like, when you go to a Google, even, like, on the Google search bar, you tend to see some new sort of way to use Google search, some mathematics search or whether... check my remainder or whatever. That's a good example. Do you have any questions? Yeah, can you talk a little bit about how you can research local competitors or have you ever experienced that? I don't know if there's any major competitors in local companies. Yeah. How do you deal with exploring just local, either, you know, based on policy or policy or just business competition? That's a good question. When we build products at Google, we have this kind of philosophy that we don't necessarily look at competition as much. Obviously, we're not, like, oblivious to competition. We realize what's there in the market. But we try to have kind of this laser focus on building something great that works really well for users. And if you just kind of... the philosophy is if you focus enough on that, things are going to work out. So that's, again, that's not to say we're oblivious to competition. We do research and we map out what other players in the markets that we're active in are doing. But, no, like, you need lessons on researching competitors in large markets versus established Western markets or any anecdotes from that. Yeah, it's a good question. But, no, like, if you, for example, compare our effort in researching users versus researching competitors, we do at least ten times the amount of effort in researching users than we do in competitors. It's really user-focused. I got a section coming up on research. If you have a question after that, please ask. Any more questions on these? Yeah. How would a product manager really converge into a product? That would be a mind-boggling question. How would... I can't be able to converge all this into a product because it's very tough for any product manager. And people out here who are product managers would probably like to know how Yeah, it's a very broad question. I think we have two types of efforts. One effort is we look at our current products and we apply a lot of these lessons to them and see can we make those better for people in countries such as India. And we have different efforts where we try to build kind of new products grown up, specifically for India. Yeah, both kind of we take these same type of principles into account. You do a lot of research. You try to understand what is working, what is not. You try to understand what big needs are that users have. You try to very deeply understand user behavior. And then you come up with ideas. You start testing those ideas in the market. It's a long... This is kind of my whole job. So it's a long process with a lot of aspects to it. I'd be very glad to talk more about it later. Particularly if you have more specific questions that I could go into. I'd love to talk about it more. How big is the next billion users group? I wouldn't know from the top of my head. If I would know, I wouldn't know if I should comment on it. Sorry? Is there like Google plus or Facebook? If I like Google plus or Facebook? Facebook is very popular, yes. I think Facebook is very popular. So then I'm going to talk about research. These are kind of more personal types of anecdotes. I'm not a researcher. We have a bunch of researchers in our team. They could do much better talk on this than I can. But I'll tell you from my own experience what we've been doing and how that's been going. So one very important thing is getting your team out there. We work with a bunch of our team are actually from India, but a bunch of them are not as well. And in particular for those that are not or for those that are from India but haven't been there for 20 years, it's pretty important to get them out to the markets and get them to experience a lot of different things in India and in other countries. Take everybody. Go with engineering, go with UX, go with product managers, go with marketing, everybody get in an airplane, go to India and do research there. That's really extremely helpful because even if you just go with a small research group and you report back about what you've learned, the impact of that is just a fraction of having people actually experience it themselves. There's more to it than just going into an airplane and flying to India. You've got to set very clear research goals for yourself. You've got to set out what do I want to learn, make it very clear, make out the questions that you want answered and make out the type of questions that you should be asking to answer those questions. Very often, I won't go too deep into research methodologies, but very often you can't really directly probe too much into the things that you want to know, but you've got to get to it by asking a bunch of questions around people's behavior and from that you can understand where opportunities and needs might be. Sorry, that was a little vague, I realized, but I'm not a researcher. And then it is definitely very important to, after you do this kind of research trip, communicate back to the people that stayed at home and do it in a very clear and very exciting manner where you use videos, use a lot of photos, and make sure that people actually want to see it and want to go through it. Field interviews is kind of one very important aspect that I probably like most of all the different types of research that we do. You literally go out with a clipboard and with questions on it, and you go to malls, you go to universities, you go to markets, kind of wherever, you just go to random places and you start interviewing people. You get extremely, by just doing a couple of these interviews you already get extremely valuable insights and things you would have never thought of yourself about how people use their phones, how people use the internet, how they go about their lives. It's very interesting. I think one very important thing to keep in mind when you do field interviews is that you get a diverse demographic. This is, it kind of sounds pretty obvious, but it's pretty difficult to do. For example, what I find a lot in India is that I have to do a lot more efforts to get a decent representation of women in the sample because it's very often men are fine talking to you, but women, particularly women are alone or in small groups. They don't always want to talk to men. So you really got to do extra effort to be able to get a diverse demographic. Even in big cities, yeah. It depends what you call big, but like tier three cities, for example. I mean, it's not like people don't want to talk to you at all, but I mean, still this year, I've experienced this, that it's just like men, you get a 100% hit rate if you talk to them. They grab their friends and they're like, hey, we're going to interview you. This is cool. Women, it's kind of hit and miss. Sometimes they only want to do one minute and it's just you got to do more effort. So when we, one trip is, actually it depends, but it depends. That's the answer. A lot of different research teams and research initiatives go to different parts of India. We tend to do, in one research trip, we tend to do like one or two cities, it's kind of, but it really depends on a research trip, right? I think at all times, there's probably a couple research trips going on for different products. And they may be in different countries. They may be in rural areas. They may be in tier one cities. They may be in tier three cities. It's just, it really depends on the trip. Yeah. Sorry, that's kind of the same point that I was also making on the previous slide. So after a day of doing these type of interviews, I make sure that we grab the team together or maybe the next day we grab the team together and we go through the huge pages of notes that we took from the interviews. We, we summarize them together because different people might have heard different things in the answers as well. We make sure that we have kind of a crisp summary. We expand that with visuals, such as photos, videos, et cetera, to make it engaging. And we share that back to the team, back in the U.S. to make sure that they learn similar things to some extent as what we've learned on those days. You're a researcher, so please correct me if I'm... I don't do too much field research. Okay. So for field interviews, field interviews is one of the types of research that we do. But just like you're bounded by, you're bounded by the actual time that the interviews take, by the time that it takes to find people on the street, and by the time it takes to get to different places. A lot of these cities, they're pretty congested, so it can take like two hours to get to one place and two hours to get to another. So, in general, I would say if on a day with a small group of people we interview 20 different people or 20 different groups of people, that would be a pretty decent result. What can we do in the university is we have these human subjects, so they have to agree that, you know, they understand that this is being a research proposal. Is that something that Google needs to do as well? Yeah, we do. We have NDAs and those type of things as well. It depends on the type of research, right? When you, when we ask a couple of questions on the street to someone, it's probably not necessary to go through kind of a formal process, but it also gets me to my next point. When we do interviews with the recruiting participants, there's definitely a whole part of the process which is, I actually know very little about this, but there's documents they need to sign and agreements. Yeah, so that's a different type of research that we do. Also super insightful. It's a little bit less being out there, but what we do here is we ask an agency to recruit people for us, and you can tell an agency we want a diverse set of demographics, these type of income levels, just a different mix of people, and you invite them into the office. And in general, we do one person at a time or small groups at a time. A general interview would be about an hour, and this would be a pretty typical setting. This is not the research I was on, but this is kind of what it tends to look like. You interview them, you ask a lot of questions. The main benefit of this would be that you... Actually, there's two benefits to this. One is you can do way longer interviews, so you can be interviewing them for an hour, whereas on the streets, nobody's going to give you an hour of their time. They're going to be walking away after five minutes. So you get like a lot of time from people. And the second is they're on their NDA, right? You can have them sign in the NDA document, so you can test your products with them. That's probably the most valuable part, where you ask them to use your software, your website, whatever. You film them while doing it. You notice that everything goes wrong. They press everywhere. They don't understand the text. It's always very enlightening to do this type of research. One benefit, one other benefit of this is this can potentially also be done remotely. We've done this a number of times remotely. In my experience, it's definitely not as insightful as being there yourself. But there's a big benefit to it, is that you don't have to travel that far, and so you can do it more easily. Yeah, it's also like our researchers would be able to tell it better, but there are incentives, I think, for interviews. And one thing that I would say if you do this remotely is you've got to have somebody on the ground that is very knowledgeable about research and can really understand what you're trying to do. So it should be a very good agency or one of your own researchers, for example. Because otherwise, I think this can get pretty tricky. So better to be there live, but if not, there's ways to do remote as well. There's actually some websites that I know that facilitate doing these type of interviews remotely as well. Yeah, question. Why such colors? Sorry? Why such colors? Why such colors? I think that's Ghana, right? I think that's Ghani's color. Yep. Yeah, I guess that's true. At the same time, I don't think this has to be that expensive. I think every type of research you want to do is going to cost money. I mean, the expensive part of this is flying to India, right? That's where the money is. Other than that, sorry? I'm staying there. Well, staying in India doesn't necessarily have to be very expensive. It depends on where you stay. It depends on where you stay, yeah. So what type of research would you suggest if you don't do it like this? Yeah. So there's two things. One, you're assuming there's already a product that people are using, right? We're not necessarily, for a lot of things that we're building, we're not necessarily at the stage yet that it's a product that's out there and people are actively using it, then we can get those statistics. Once we are at the stage, we definitely do that type of research as well. Actually, that's kind of a pretty critical part of my job as well is analyzing that data, trying to understand what's going on. Unfortunately, that is a very different type of insight you get than by being on the streets and interviewing people. So I would argue kind of this type of research, in particular the field research that you do, is very, you do that in an earlier stage where you really try to develop the idea of your product. And like here, like when you have an early product that you can test and a type of data analysis from the logs that you have and the statistics that you look at, I would say that's slightly later when people are using your product. Can you describe or talk about just the per baseness of smart phones in general and how those compare to maybe like with phones? I wasn't actually hearing Donna. This is a photo I have for another reason. So in general, with the research that you do in like different places. Yeah. It's kind of the numbers you see with the cell phone connectivity, but not smart phones. Yeah, that's a good question. We actually, I wouldn't be able to answer it. That's a short answer. The thing is that we don't, as Google, we don't really look at feature phones that much. Like we have very limited ability to do things on feature phones and to help people on feature phones. So we are like 99% focused on smart phones. So yeah, unfortunately, I can't tell you much more about it. Yeah. I'm saying this, but in the city, I've seen a lot of foods where they give out free smart phones and stuff like that. Have you guys ever thought about doing the same in emerging markets and give out some type of disclosure like, hey, we're going to track your usage data and stuff like that? No. So giving our free smart? Sorry? Or is that a possibility that something you can do? So I think people overvalue what the value is of the data to us, but also giving out the free smart phones is obviously something that you wouldn't be able to do at scale. So I don't think that would be a feasible strategy for us to do. Yep. Yeah, that's just going to agree. Usage statistics all need to keep you so far. After that, you really have to follow up with people to find out the why. Yep. Also, developing your product, like you need to do some contextual research. And just to respond to the person who, I think most of us don't have the luxury of Google and the budget and everything, but then that becomes your job as a product manager is to advocate for why that's important, like the business case for it, what the retail investment is for it. And yeah, I mean, if you're a valuable product manager, you can do that. Yep. I also think like, so one thing that I've also thought about, like doing this in a cheaper manner, and again, I think the thing that makes it expensive is traveling to India because the distance is just so large. It's pretty much the other side of the world from here. Like for example, rural areas in Mexico might be a pretty good substitute. A lot cheaper to get to. You do need translators, obviously, but that might be... I'm pretty sure that that might be a decent alternative as well. Yep. Question. Is the development cost higher because of all these things that you're saying for emerging markets, or does it get balanced out when you're developing for countries like the US with other things? It's a good question. Is the development cost higher? I've been in finance, so I can give you kind of my financial answer to it. I would say that people are probably, by far, your most expensive piece of building a product. So the efforts that you do to reach users in India, for example, are probably not that significant part of the total cost in comparison to your engineers, your product managers, your designers, et cetera. Sorry? Sorry, the light is a little... So we have engine... We do. We have product teams in India. I currently don't really work with product teams in India, but we have. What makes working with India extremely difficult is the time zone difference. It's always at a bedtime for someone. We have, for example, we have a researcher in India, which is actually great to have a researcher in India, but even that, it's either for her, it's late night or very early morning, or for us, it's late night or very early morning. For a little bit of time in the team that I work on, we had part of our team where the engineers sitting in India. It was very difficult for us to make that work well. I'm going to talk about people's work in India, which is in America, I don't know their research schools. But in India, when you ask people that, how do you like this product? Or are we going to go for this product? There's always stuff that's wrong with the product. So it gives you the positive aspect because you're a researcher, so you get to know what people are expecting out of you. So when you are thinking of developing a new product, it gives you lots of information on what you can work on and what already is in the market. So it gives a positive aspect to that, but it's in the company world where it's in the research study. Yes, this is something that's not in the market. So this is something that you can work on. So I think face-to-face, the research that is done in person, it helps rather than analyzing some of the exclusive data because they're going to use what is available to you and they're going to provide you with information which they want you to work on. Yeah, that's a great comment. Also one more thing on that point if you want to try and do this at kind of a lower budget. And what we've also done is we've worked with websites that do a lot of this process for you while you don't have to be there. So there's websites that help you. They get participants, they film them, they ask them questions, they let them use software. So they take over a large part of that whole process for you and you don't actually have to travel there. It's a little less insightful, I would say, and a little trickier to do very well than going out there yourself. But there's definitely an honor option. User-testing.com, I think, is one that does it. There's probably more. I don't want to advocate one. There's probably a bunch of them that are great. I don't think we do. Yeah, yeah, yeah. How does that affect the end result and what kind of trade-offs do you make for those differences? Yeah, I think that's a very good question. Again, I'm talking about next billion markets as if there's one thing, but the diversity is just absolutely huge. So I think it kind of comes back. I think it comes back a little bit to what I said about trying to find these things that they have in common and a couple of these things are pretty technical. They're kind of boring aspects of what they're, kind of the phone specifications that people have, the low internet connectivity, high cost of downloading, all those kind of general things that you would see in any type of emerging market. And then, yeah, so to account for, in India, for example, rural India is just a different world from tier one city India. It's like, it's as different probably as US and tier one city India. So obviously you can build a product for everybody and that's perfect for everybody, but at least you can build a product that kind of sits in the middle of what India users might want versus a product that just sits in the middle of what US users might want. Does that kind of answer? We tend to focus on big cities and I think that's primarily because that's where the majority of the users are. But, yeah, so I would guess that our products are probably skewed towards people in big cities. Yeah. So that goes under the assumption that the majority of the potential new users would be plastered towards big cities rather than rural area, right? Yeah, so, yeah, I would have to look at the numbers. My assumption right now would be that the numbers of new internet users as well as current internet users in big cities are way higher than in rural areas. That's not so wise. It takes time to propagate the networks of connectivities. I have a question about, it seems like billions of new users and we can't get enough potential information or knowledge just by a few tips. Yeah, that's true. Do you guys have a permanent full-time researcher on-site or in India? Oh, yeah, yeah. I guess your strategy as a product manager is to go through basically working in collaboration with the full-time researcher. Yeah. And then you also do intermittent trips to gain by specialized, like what kind of thing that you need to basically get there to gain, like what kind of knowledge? So your question is if you have a researcher in market, why do you need to go out there yourself as well? Well, honestly, like the researcher is out there like she does 10 times more. The researcher that I work with closely, for example, is out there all the time. So she does 10 times the amount of research in India that I do. However, like for me, getting a research report from her versus me being there and interviewing people, even if the same insights are in there, just kind of how you internalize the knowledge is very different if you spend the full time and you actually interact with people there. So I think it's a great question, but it's kind of a soft answer. You learn a lot more by interacting and by spending a lot of time in the market. I had another slide which, I guess I can't get through anymore, but it was a slide where I said it's also good to... Okay. Yeah, great. So this is one other thing that helps you as well. At least for me and particularly my first couple of trips for India, for example, really helped is what we call immerse in the cities there. So just trying to stay at more local type of places. So you can meet locals, you eat local. So guest houses might be a good choice. If this is different areas, it kind of sounds very obvious, but at least when I travel normally, I'm used to being a tourist. So you go to the main square of a city and you go to the beautiful church that's there or something like that. You can do that, but that's not what you should be doing probably for your research. You should go to this random neighborhood, this random mall here. Just travel to all kinds of pretty much random places in the city to get a real feel for this is the everyday life that's going on. Then use local technology. So for example, what we would do is we try and get a SIM card in India, which actually extremely covers them to get a SIM card in India. But you get a SIM card, so you get the same speeds of internet that everybody else is using, the same coverage, et cetera. I often go and buy, or not often, but I've bought a second-hand phone for $30 to get the experience of what it is like to be on a very low-end phone. Talk to shop owners in India about what apps they use, what they sell, et cetera. So to really get a feel for... Obviously you just scratch the surface, right? But to get a bit of a real feel for the everyday life and the everyday technology experience that people have there. So there's never enough research, I would say. We're kind of always on short supply of research. There's just so many things that you want to know. So I go every couple months, but as I said, on our team there's a researcher that is there continuously, and in India sometimes travels to other places to do research there. So... Because at different stages of the product cycle, there's also just a different need for research. When you're just trying to understand a certain area, you want to go there to understand what apps people are using, what their current behaviors are. When you have your own app, you go there to test the app. So there's kind of very different stages in which you do different research. The need for research is almost never diminishing. So yeah, that's a good question. A researcher would be able to answer better, but so for example, our researcher is located in Hyderabad, so I go to Hyderabad every now and then because it's just easy to coordinate it. She often suggests different cities to go to as well, just for the diversity of it. They have different demographics, cities that are more student-heavy, richer, poorer, tier one, tier three, rural, south, north. So we try to get a diversity of different places. Well, it depends if you're building something new or you already have something, but in general as a product manager, you're always building something new, right? You don't rest on your laurels and like, our app is great, we're done. If your app is out there and it's doing great, you try new features, you try like new approaches to things. So I think you're always testing a new thing. Sorry. Yep? Yep? Sorry for... It's extremely influential. We make, I would say, research is a... I don't know how to phrase it, but I would say we make all our decisions based on research, pretty much. Yeah, sorry. How does the product design change based on the insights you've got? So, I mean, we define the product. We define what the product is based on the research. So it's very difficult to give a concrete example because everything in a product, like front to back, the idea, the design. So you're specifically asking about kind of the visual design. For example, there's no network connectivity or intermittent connectivity. The search result would be given to you later on. Yeah. So that's one of the design or the product decision that is taken on dates on the inside. Yes. So, like, and a specific question of mine would be because the network connectivity is too... How do you optimize the products to work on slower networks and take less data? Timon. So we... Yeah, it's kind of what I tried to touch on previously. We... For low data in particular, we try to have three different states. One is an offline state where we are pretty confident users offline. It's not going to get... It's not going to go online, like, very soon, probably. We have the online state where we have pretty decent connectivity and we can work with it. Then we have the intermittent state where sometimes we're getting data, then we're not getting data, then something comes in again. So we very consciously designed to have those three states. No data, online, and intermittent. Does that kind of answer your question? So what specific product decisions were taken? Can you take an example of any Google service that incorporates such a feature? Yeah, sure. So for example, in YouTube Go, which is a YouTube app that we have for India that we're testing out right now, one thing that we do is we make it very easy for people to download videos to save offline so that offline state is kind of a very rich state where you can watch your videos, you can do a lot. We also let people... I believe we let people time when videos are being downloaded so they can download it at times when data is cheaper or faster. We also allow people to share the videos at the half offline with other people while being offline so they don't have to use their data to be able to share videos with each other. So those are kind of... That's very offline focus, not as much intermittent focus, but those are, as you can see, we do a ton of effort to make that offline state a really powerful, really rich state. Yep. So I think you... So were they contradicted your intuition? I think kind of all the time, you kind of forget your intuition in a sense because you just don't know. You kind of got to assume that you don't know and if you start from that basis, you might get less surprised because you actually learn from what you see. Sorry, it's not a very concrete example, but... Yep. So we have some... We have some published results, like a number of these slides, for example, are published from IO, I believe. Others are not... We have some published reports, but it's mostly for our internal teams. That's a good question. Sorry, I should have known, but I don't actually... Search for it. I think if we... So the IO resources that we have, they're pretty good. We have these two hour sessions on this topic. And we probably have some other reports out there as well. Sorry, yep. So I just recently heard about the Google Station project where they're doing free Wi-Fi hotspots and train stations in India. I'm wondering if you worked on that project at all or if you know anything about how this type of research informed that? So I worked... I know a bunch of people on that project and they sit very close to me, a couple on his feet, if less. How did research inform that? I haven't been as exposed to their research to be able to tell you, but that is definitely... That is definitely a research-based product where people went out to India a lot. They noticed this kind of... This is also something we've noticed in our research. You notice people kind of bundling around Wi-Fi hotspots. We have great photos of it where you see a group of 50 people in one corner and you're like, why is everybody there? There's a great Wi-Fi hotspot there. So I think it comes from those types of insights. So I think that Google Station is considered... is going great, I understand. I mean, if a product is successful, it's going to be there forever, right? It goes to all the product managers. How important is it to start your product development with zero assumptions? So I would guess... It's a good question. It's a broad question. I would guess having zero assumptions is going to slow you down because you don't know... You have to get everything from data and research. But in particular, when you do the type of thing we do, you build for people that are in very different places and have very different cultures than our own. Then that is kind of the place where you've got to start. Zero assumptions and everything is based on the research that you do. Merging market. Research is kind of an insurance to that risk. People are a big risk? Knowing what people want in the new market. So what are some other risks involved in entering the new market? Would you say, and what are some insurances that you take to... Interesting question. The fact that you don't understand the people is a risk and your research is an insurance against... That's what you're saying, right? It's a very different kind of frame from how I normally think about it. And also we don't really think about entering new markets as much because we have some usage of most of our products in a lot of markets. It's more that we kind of try and make our products better for those markets. So I don't really know if I can answer that question very well. So first let's say you look at a product like YouTube, right? It's being extremely widely used already in India. And then what we try to do is we try to make it a better experience for people there. So it's not as much kind of entering a market as more... This is what we currently do in the market. It works so-so for people. Let's make it great for people. So I don't know if I can give much advice about entering markets. As a follow-up to that, earlier you mentioned about these five considerations they have for altering the existing product. But is there any product that's more country-specific? Let's just say, for example, P&M's or even, for example, WeChat has all these different boxes. That might be one or maybe the white one, but it will be one that is just country-specific. Google Station, for example, is one that's country-specific right now. So in general, Google tries to build as much of a scale as possible. And even though India is like this huge market, the world's pretty big. So we try to build products that perhaps you start building them for one market and then make sure that they work in a lot of markets as well. But we tend to not limit ourselves to a single market for a product. So countries with restrictions. I mean China is an obvious example of a country with restrictions. I know very little about China. And my focus is primarily on these three countries, actually, that we mentioned earlier, India, Indonesia, Brazil. Restrictions are very limited. So we're going to take two more questions. Two more, okay. Don't feel pressured. Yes, so I think that definitely requires a certain approach, but I don't know very much about it. Okay, I think that's it. We have time for two more questions. Okay, we have time for more questions if anybody has one more question. I don't know. Okay, yep. What type of features would be interesting? Oh, interesting. So I think what's most important is not really about the questions you ask. It's more about understanding their behavior. And you do have to ask it through questions, but are you referring specifically to when you already have a product? Yeah. Yeah, those kind of questions would be decent. Again, there's a whole, like, recent methodology. I hardly dare touch the subject because it's such a huge subject and there's very specific methods to it. But yeah, you can, what is great, for example, is have them show you things on their phones. So you say, oh, how are you using this right now, and they show you? Like, you learn a lot from that. Ask them, have you used this feature? What is your general impression of this website or this app? Just kind of very... I would say try and ask very broad questions because then you will probably learn most. Okay. We can talk later. How do I apply for becoming a product manager at Google? How do I begin preparation? So, I mean, applying itself is just through... We have jobs at Google.com where you apply. But, I mean, there's... I mentioned products through themselves. Mainly, you need to build good experience in some technical understanding, some kind of general business understanding. And then, if it all possible, experience having built a couple of products. It can be small products, but... I'm an engineer, so the way you do, I just deal with code. How do I approach... Yeah. So, I think expending on your business side would probably be, for an engineer, would be good advice. You see a lot of engineers with MBAs, for example, that become product managers. That's probably the most typical product manager background. And then build a couple of products yourself where you don't just... You actually build it end-to-end. You try to understand what to build. You try to get users. In particular, if you get a bunch of users on it, that's kind of the whole goal of being a product manager. So, then you've done it right.