 Hey guys, so we're talking about data-driven product development, and I want to jump in right. And I want to start off with how product development actually happens in general. When I say general, I don't mean a large-scale company, maybe like something like Google, but something like a traditional startup or maybe even a mid-scale startup. And how do these companies do product development in general, right? So I've been working with a few startups and what I've seen is that it's to a large extent it's hunch-based, where let's say a project has a very prolific product manager, right? He's been working for some 10, 15 years, so he gets to call the shots on what gets added to the project and what doesn't get added, right? Or maybe it's a developer who has been at the company for a long time and then gets to do the same, or maybe the designer, right? You get the drift. So it's probably someone in the team who is probably the most senior and supposedly has the most amount of experience. They say that this has to be done this way, it's going to be done that way, right? So there's another point that I want to specifically call out, founder-first. Hunches can also be founder-first because every founder likes and wants to be Steve Jobs, right? Where they call the shots and they can define what the users are going to use. They want to do that, but there's a very high chance that you guys sitting in the office have a disconnect with the audience. Maybe if you go out, have field trips, talk to your users, that's the thing what you're supposed to do and then take a call, right? But what if you're sitting in your office and defining that this is how the users are supposed to be using the app? That doesn't really work. And founders, they love to do this. They want to be seen as these product gurus, right? So that's a lot of this. It's also reactive. Somebody before my talk spoke about this also, that maybe you launch a feature and you get a lot of angry responses on Twitter screaming, hey, this is not working the right way of expecting this or you have a lot of people shouting at you, maybe even writing to your support tickets and everything. And then you take a step back and like, okay, what just happened? And then you probably have a look, make changes and make another release, right? But there is a high chance that your product doesn't work the right way and nobody told you about it. Your users are not that engaged and they didn't get you to get back to you on this, right? So product development tends to be reactive in the early stages a lot. And when they do look at data, most of the companies tend to just end up adding something like a catch all system, which is like a Google analytics, which what it does is like a magical script that you put into your HTML page and it starts tracking your users, how they're going from one page to another, how many users dropping off in between, it gives you all of that data, right? So a lot of people start thinking, okay, we have data. We can always look at it whenever we need to. And we are done. Let's just stop at Google analytics. Then if somebody realizes, okay, probably we need to be doing a bit more in terms of analytics, we need to put in a lot of custom tracking. It gets pushed off because of lack of bandwidth. Most startups tend to work in a crunch mode. As a product manager, you always think that there are not enough developers available for all the things that you want to be doing, right? So lack of bandwidth defines to a large extent how the products are going to be run. So yep. So this talk is about data driven product development and how you can go from these stages to a more data driven approach where you have the data talking and data analysis being the major way of making things. And who am I? So I'm Shashank Mehta. I take a lot of roles at Razor Pay. It's an early stage startup. I was employee number seven. So I'm primarily now an Aalakartian manager where I'm pushed to handle whatever requires handling. I've done a lot of other things which helps me in taking a very holistic approach to the product. And that has helped me. But at the same time, it's difficult for me to tell you guys what exactly I do on a day to day basis. Apart from this, it's relevant to this talk. I've worked at a company called Zomato two years back where I made a product called digest, which was entirely about collecting statistics from users, what was happening and then comparing it to the past. So this tool was also made by me over there. And yeah, so some of our friends are pretty excited about this and they've their own hashtag. Feel free to give me feedback via this hashtag. I'll be monitoring. You don't need to remember my Twitter idea or anything. Just to eat a Mehta refresh to the extent that I think folks, I think have a different timeline for Mehta refresh also now. But anyways, yeah. Okay, so now you've decided that, yes, let's let's bring in data. Let's make this entire approach more scientific. Let's. So yeah, you will be like, let's define data points. Right. First step is very simple. Define data points. Nope. Uh, today in the morning, the first talk was by Sovik and he mentioned how before making anything, making a content website and the entire IA thing, he said that you should be asking the right questions. And what I believe in analytics and analytics also write questions are the most important thing because the data point is just a metric. You need to be asking questions on how the users are going to be using your product. You need to be asking questions on if let's say people have a problem, what data do would I already need to have to solve those issues? Right. And you need to also ask questions. Okay. Will this feature work as expected? And you need to define the expectations. For example, how do you call a feature successful after you've released and people started using it? What is the metric to define a feature as successful? Or the corollary is how do you define it? The product has been an absolute fail that we released it and turns out it's a flop. We were absolutely wrong. Right. So you need to have definitions with these things and can these be quantified? Uh, just saying that, okay, if people are happy using it, it's not enough. You need to have some numbers and that's very important. Going forward, right? The question that I just set out loud over here. These are questions which anybody can ask about any product. The very generic, but there are a lot of questions which can be specific to your own product. For example, with Uber, Uber might want to track the time that it takes to open an app and then book a ride. Where are the dealers? Right. It's specific to their particular app. So you should be asking questions at all points of time. And before making any particular thing, any changes always ask questions along the lines of these, but at the same time, whatever is required for your product, ask those questions too. So now that you're done with the questions, then you probably move to define a data points. You could be tracking the events, user clicks, you could be, for example, in Facebook, you could be tracking the likes and how many people open their own profile page, et cetera. You collect data. There are a bunch of different tools out there. I've mentioned Google analytics. There's mixed panel, there's segment, there's key, there are a lot of different things that you can use. Then you analyze. Maybe you have a product manager or maybe if you have very copious amounts of data, you have a data analysis team which looks into it and then gives you meaningful insights into it. Then you iterate. Products are made with iterations, right? So you iterate, you track some more, then you go back and iterate, right? And finally, what are you supposed to do profit? This is all too abstract and it's way too high level. So I'm going to be looking at an example. Instagram stories. So how many of you know what is an Instagram story, the feature? Okay. So I knew there would be some people who would not know. So Instagram stories is basically Facebook's take at Snapchat. Basically they decided that, okay, maybe teenagers don't want to be posting photos which last for a long time. So decided we'll have a system where you can post an ephemeral photo or a video. It stays up for 24 hours. People who are following you can see it and then it goes away. It's not available anymore at all. So how it works is on the top that you can see the plus button on the top allows you to, the plus button on the top allows you to post photos or videos. That button is specifically for the Instagram stories. The button at the bottom that you have, it's for the normal Instagram account. You can keep posting throughout the day. It gets added to your 24-hour timeline. And on the top that you see over here are the stories from your friends. So they're listed over here. All the friends who've been posting stories, you can see it over here. And when you go inside a story, the white bar on the top, these four bars, these show you how many posts are there in a story. You can actually draw on a post. You can add emojis. You can actually send messages for a particular, like, I like this photo. So I can reply back, hey, a nice photo, really creative, maybe. And you can, there are a lot of gestures. You can actually tap left to go to the previous shot in the story. You can tap right to go to the next. You can swipe. A swipe gesture skips the entire user. So there's a lot of gestures. It's like a standard mobile product where you have gestures, you have actions being taken, you have consumption. So that's why I'm taking this example because it has a lot of the general thing that you see in an app. Right? So let's start asking the right questions. Same questions. Success, failure, define what is expected in everything. And, yep, let's start with defining success and failure. Right? So I took a lot of examples, especially for this talk and at work, I take a lot of products. The way we define success and failure, I've noticed that there's a trend. It can vary from product to product again. But the trend is basically first you define the number and type of users. The number of type of users who are going to be affected by this feature or this product. It could be your daily active users. It could be your monthly active users. It could be a segment. For example, only your power users or let's say it's only for teenagers from the age of 13 to 18. You define a segment, right? And then you quantify what is the effect or the effect on these users. How are they going to use it? So you define all of those things. So I took a shot at this for Instagram. You can vary, but yep, this is a definition that I want to take for Instagram that success on a normal day, if let's say 85% of active users post at least three shots and view at least four stories. It's a success. It's a proper success. Your product works exactly like how you would want it to work and everything is fine. The other side is the failure where on a day less than 50% of active users post less than three shots, you less than three stories, right? How I came to this is I so since I am not working with the product I took a guess that for Instagram, they want stories to be the next big thing. So they want at least 85% of active users to start using it and between these two definitions, your success and failure, there's a lot of space and there's a very high chance that your product might actually lie in this space. But that's a good thing. If you have already defined this and you know that it's lying in between, this is where analytics serves you the best because now you can know that, okay, my metrics are at this level. What can I do to push it up? The usage, the events and everything is telling you that it's at this level and now you have to push it up. That's your next role. So let's see some data points for Instagram, right? We have already defined success and failure, right? So for success and failure, all I need to track is the number of stories viewed, the number of posts in the user's story. So I'll know how many stories he has seen and I'll know the number of posts in the user's own story, right? So this would help me in judging success and failure entirely. Pretty simple and straightforward. But that's not the kind of analytics I'm focusing on while it is relevant and to the point, but that's not the analytics I'm focusing on. And my entire focus on this is that a PM or maybe even a designer and developer, when they think about the product, they have to step out of this data points and move to something that I would like to define as ephemeral data points versus the database. So when I was working at the Matto, the tool, for example, what it did was it looked at the number of likes that a user's post has got, a number of photos that are user on average post. This was a metric, right? This is in your database. It's already available and stored in your database. Right now, if I make this product, which I did for the Matto, I could actually add the last five, six years of data without requiring to make any changes. I already had historical data available and that's key because you have all of that data available. You can actually start it whenever you have the time to do it. But what you're missing out on is something that I want to define as ephemeral data which exists somewhere inside your ecosystem but it's not in a database at all. For example, at Zomato the number of times the number of pages maybe I scroll that data was not available in the database, right? Or maybe inside the Instagram stories the number of times a user gesture it's not available. It's there inside the application. My Android app knows that the user took a gesture but it never got added to the database and this is the data that I can never look at afterwards. Right? So these are the main points of focus that you should always look at and see that how do I start tracking these things right now so I don't miss on this data. So what are the ephemeral data points that can be there for Instagram? Time taken to post a photo in a story. So your feature, so you want it to be the main feature but if there are a lot of huddles in the way of the user before he can post a story and post a particular thing, there are going to be issues. So you can track the time it takes to go from an app opening to the user posting. Then there are multiple things in this. You can look at how much delay was added due to the interface itself, the interface that we have designed so carefully. How much was the delay? If it's too much, it's an issue. Right? And then there are delays which maybe are not in your control or maybe to an extent like the network delays. So let's say the network delays tend to be a bit too much and especially let's say in India because the internet is not very good, right? Maybe Instagram needs to focus on the entire process of minifying the photos better, maybe making them lower quality or do something that can push the number to the success levels, right? But you need to be tracking all of this and these are some points which are not easily available to you guys. Then the days before posting first have a story. You roll out a feature on a particular day. If you just look at the database you would know that the person posted a story on let's say the 10th day after the release but you don't know when did he start seeing the feature. He could have seen the feature on the 6th day after the release, right? This data again is not available in a database may not be, can be depending on how you have structured it but it may not be available, right? And this matters because this for example in this entire thing, the amount of time I need to spend or a user has to spend before he gets convinced to use a feature is important because this is something that you would want to control. If a feature takes about 30 days for someone to be convinced to start using it, it's an issue, right? Again, this is a point which is not available to you in the database. The number of stories I view before I am convinced that yes, this feature is right, I need to start using it. Again, this may not be available in a database. Some more points. I spoke about gestures, right? On mobile especially gestures are the way of how people interact with the apps. No longer are these guys crawling a lot. There are a lot of gestures involved especially with these high consumption media applications. Gestures are the main thing. You look at a video app, you look at a photo Instagram app or Snapchat. Gestures are the main thing. To be very honest, I have been using Instagram stories on a daily basis and to me it was very intuitive. But my friend Nemo, when I was speaking to him about these slides, he told me these gestures exist. I had no idea. Right? So that's another point that we have not been tracking and has been missed. So how can you track it? The number of times the gesture is used, you can track the total stories viewed before I did use the first gesture, right? These can help you in then analyzing it and we'll get to the analysis part. Then for example, did the person use the drawing feature? Did he use the writing feature? Did he send a message? So all of these points should be tracked and they help you in understanding how is the user really using the feature? Now, these are not covered by your high level metrics and that is why these tend to be a bit more important also. So you start collecting data. Now that you have defined data points, you have to start collecting data and while again, I don't work for Instagram so I'm not sure how exactly they would do it. But we have our own entire pipeline at Recipe and I'm going to just run you through briefly because my focus is not on the tools. There are various tools you can use, whatever is more convenient for you or what your company is already using maybe. My key point is to tell you that there are a lot of things that you can do. For example, Google Analytics is used by us for the general catch all what we don't track ourselves. Mixpanel is for a lot for segmentation, for custom events, for creating funnels, all of those things. Mixpanel helps me identify what a single user has done. Whereas Google Analytics works on the aggregate data. Keynotio is a special one. Not a lot of folks might know about this. Keynotio is to a large extent similar to Mixpanel but there's one key difference that they give to people like me who are also developers. I can use the graphical interface to create an entire query where I'm defining a segment and I'm defining a metric and I'm asking about the historical data. Keynotio, what it does is it also gives me an API endpoint complete the token everything that can copy paste to my application and I can query that data whenever I want in a programmatic way. Keynotio is used for that aspect in our company. I can give you an example also where we have a checkout form where a merchant so RazerPay is an online payment gateway so we help merchants collect money. If you use Hasgeek and bought your tickets, you bought it via RazerPay. Over there we have a merchant icon in between when we were changing the designs how we were displaying that icon was being changed and we could have easily broken it for our older customers who were using it in a very different way. So what we did is we went back to our Keynotio data ran a query which got us the unique images, got an endpoint and then ran it through the new design to see how is it working. That's how we did data driven development and the last one is special one Lumberjack. It's a custom service that we've made very simple nothing very complicated. All it does is it takes the data from all of these different sources and puts it in a normal database so that they can run our MySQL queries and other querying whatever you want to do so that we're not just curtailed by what features these other applications provide us. And how all of this is wrapped up into one is segment.io. Segment.io is a pipe which allows me to send data to all of this. So analysis right. Let's look at the look at the gestures example again. Gestures supposed to be useful and they're supposed to be intuitive. You can look at the data how how frequently is a gesture being used. If it's not being used at all by a particular user it may mean that it's not intuitive right. And this is when you can take a product call based on the data that okay it's not intuitive. We thought it would be intuitive. We need to do something probably we can call it out specifically or we can probably you know if the user does not use it till 6 post then we show it up. You can take various calls and that is possible if you're tracking it and it's again on the ephemeral data we would not have been working on this if you were not tracking this ephemeral data point right. This is just an example. Your analysis depends on how your product actually works itself. And then the many steps are the same. You iterate, you track, iterate, profit right. We're going to move on to some other examples now from Rezape. So as I said we're an online payment gateway company the most important factor for us is success rate. So this is an example live data for a particular merchant. This merchant is getting about 88.05 percent success rate at an order level that if a customer is trying to pay the odds of successful transaction are 88.05 percent. Big Spanel helps me to check it at a merchant level at a merchant city level. All of those things. It allows me to look at historical data how the trend has been in the sense that I can look out there some days when it was as good as 100 percent some days it went down. We can have a look at why that happened. Then there are other cases for example over here what I'm tracking is that for a payment there are two end scenarios success failure and we had programmatic callbacks given to merchant on success and failure but in Android what we observe was it wasn't happening 100 percent time. And in hindsight it was obvious user can press the back button and probably just press the home button quit the app not taking any other step right. So we started tracking these the first part that you see over here is the number of people have submitted the data to make a payment to be a car details then the second one is the number of people who press the back button the third is how so we now show an alert that hey are you sure you want to cancel a payment. So the third one is when the person said yes I'm sure I want to cancel it and fourth is when they went back to the checkout right. So this helps me in analyzing a lot of things and I can now test on this for example if I change the alert message what are the implications. Does the number in between change those things can be done via this. I'll take another example so auto to be filling is a feature where it's on Android where if you're making a payment to read your SMS show it to you over here you click on use OTP and it's done. So let's look at the number of users in this for us it's everyone on Android the usage and effect the success rate and the time to pay right. These are the metrics that are going to be affected and this is the metrics that we took we have to reduce the time taken to complete a payment by 30% and increase success rate by 5%. This is not a critical feature in terms of the payment itself if this did not exist you can still make a payment. So this is an add-on right so for us an increase even in 5% success rate is a big number and failure over here means any decrease any decrease at all in success rate or increase in time to pay even milliseconds becomes a failure because this is not a critical feature right and this is what we found the success rate increase was actually 8% during the time and we did the analysis and the decrease in time to pay was as much as 45% that after the 3D secure page open initially it used to take people 30 seconds to switch to another app look at the OTP put it in and then submit it after this feature it went down to 16 seconds right that was a success for us but there were issues if you look at just the success and failure metric there were issues because that clearly said it's successful what we observed is that in the feature what we had is we had an overlay on top because you wanted the focus should be at the bottom and we were expecting that if let's say something happens the OTP is not delivered user would just tap on the overlay go to the bank page and take an action like ask the bank to send the OTP again and answer the payment turns out 10% of people did not realize it's clickable and you could find this out because at a payment level what we saw that the back presses had increased suddenly and that's not one of the things which we were adding to a success because overall it was looking like the increase has been good enough we had 8% success rate but turns out it could have been even more than 8% because of this issue so we went to this where we removed the overlay completely and tried to bring prominence via shadow so this is another example of an iteration that you can take if you're calculating and you're tracking ephemeral data points so you have your entire system in place what are the new possibilities that are now open to you when you've done all of this you can now tell precisely if your feature is working how is it working what are the actions that the user is taking but this opens up a lot of new possibilities in terms of the experiments so in the instagram thing on the top what you see is the stories of friends right it's in some particular order it has to be an order right can we actually play around with the order to see how the metrics are being affected does the order of friends in that list matter what if I have a look at ok this person likes why person's photos more so let's put why on the top does that affect the time it takes for a person to convert to an instagram story user does that how does it affect all of these metrics right so you create a base and then you open up your application to a lot of experiments like this if you have the entire base in place a lot of other things that instagram could probably do like notifications if one of my close friends starts posting does this affect my adaptation of the entire feature and when I'm clicking a normal photo so as I told you right there's a different button for normal instagram a different button whatever I'm clicking from the normal button I show a small overlay saying hey do you want to add this to your instagram story does that really engage users so you can do a lot of these experiments and it's only possible if you're tracking all of these data points I just want to call out unsplash.com all my images background images are from there they are creative comm and zero license requiring no attribution but thank you and yep that's the end of my talk I'm going to leave this slide over here because this talks about how PMs generally work a lot of misconception about how what product managers do and yeah I'm open up to questions now thank you yeah great talk thank you you were specifying that lumberjack is used for collecting data of all the other analytics tools into one place segment yeah so how is it different from segment warehouse I've not used the tool so I can just explain to you what segment does what segment so the major advantage of segment for us is that on android where you cannot keep on changing the services right so segment becomes the one endpoint and I don't need to change anything on the application to add another service so I'm not sure the other two probably that works the same way but segment allowed me to I can remove keynote I have put another third party service ahead of segment without requiring any change in the android application that I've released in the market so that's the major advantage for me alright I guess either I was clear or I have a question so I mean all the experiments you did how important or how much did you AB test versus you know just pushing out and testing the metrics so AB testing is a different setup in itself AB testing what it means is you just roll it out to 50% of people and 50% of people whereas what I was talking about over here is a conscious change in the product that you're making and then you're looking at what are the changes in metrics AB testing can come after this that if you have all the metrics in place now you can the next step would be to set up an AB testing platform where you can now serve users different views based on the segment that they are in and then track it so the entire process remains the same AB testing is an add-on to this entire feature so yeah the uh full-on to that is so once you do this right there can be another random event that can affect your metrics when you're measuring at different time points rather than through an AB test so that does it yes AB testing captures all of those things because uh it looks at the general like this entire thing what it does over here what you've done is we have added one particular feature and then looked at okay there's one particular feature that would affect it AB testing helps you in that uh so yeah that helps but over here this becomes the main base because without this I can't get AB testing in place and that's the next step for me also in uh at raise the pay that we want to put in AB testing in our main checkout form see at the larger scale what changes that's the next step for us yeah all right folks thank you is more question it's fine I can hear you I'll repeat out the question for the audience it's a pretty good question uh so what she wants to know is how do we come up with the numbers in the success metric and the failure metric itself so what we've been doing and how I came up with my own product is basically so we've been developing the feature we use it internally right uh you've rolled it out basically to our internal folks so we have like two versions of our application one is for the external people to try other is an internal lease we do we roll it out over there people can have a look and then we see what are the numbers looking like now this is a controlled environment because the people that I've rolled it out to are all people of my office with very good internet and everything so that tells me what probably the upper limit can be that it's been coming up with the metrics the second thing that a lot of people can do and we also want to be doing is uh I think one person before me spoke about this coffee house testing so you you basically go out give out a voucher and ask the person to use it and see how what happens over there this helps you in placing at a very small level that okay this is the change that can happen then you define this entire success rate and then you roll it out to everyone see okay to the masses how is it working so you have an expectation which is why a controlled environment which is your old colleagues and everything and then you move on to the entire audience and see okay how is it working at a at a large level thank you