 Hello, everyone, and welcome to this week's Product School Webinar. Thanks for joining us today. Just in case you didn't know, Product School teaches product management, coding, data analytics, digital marketing, UX design, and product leadership courses online and our 16 campuses worldwide. On top of that, every week we offer some amazing local product management events and host online webinars, live streams, and ask me anything sessions. Head over to productschool.com after this webinar to check them out. Hello, and welcome to How to Measure the Right Things. This is Drew Dillon, and I've prepared this document for this presentation with. Hello, and welcome to How to Measure the Right Things. This is Drew Dillon, and I've prepared this presentation for Product School. Today we're going to be talking about web applications, mobile applications, how to make sure that you're collecting all the right data that you need to become a data-driven organization. This is me, Drew Dillon. I'm a freelance product leader currently. In previous roles, I've been Chief Product Officer at a company called Schedulo that does mobile workforce management, VP of product at a company called Anyperk, which is a YC company that does employee perks, discounts, and rewards, and then prior to that, Director of Product at Yammer, which is where I learned a lot of what you're going to be hearing about today. First, we're going to talk a little bit about the character of data, what data can do for us, what it can't do, so that we can understand a little bit more about what we're working with before we decide to figure out how we're going to collect it. Now, when I talk about data, what I'm really talking about is this projection that we see when companies go from a Series A startup from three individuals working out of a garage, all the way to become these data juggernauts out in the public markets, collecting and using this data to inform product decisions, build better AI to better adapt the product towards users, and really just understand it more and more about their users. What I'm not talking about is big data. When you hear the concept of big data, what people typically are talking about is data that's been collected over a number of years, put into a database. It's not really well understood what that data can do and how it's been collected. That's one of the biggest things you need to think about when you're looking at data. When somebody hands you a database full of tracking data, the way to think about that is you can't really understand the data because a lot of opinions go into gathering data. Do I measure this or measure that? So data that you get is basically a set of opinions that you can't really trust and you can't be sure that the opinions are going to be ones that you would have made on your own. So when we're talking about data, what we really want to talk about is actionable data. Most likely when you're talking about web and mobile applications, this is data that you've collected in the context of the user using your application and any metadata around that. When they logged in, what browser they were using, what mobile phone they were using, not any of the actions that actually took place on your site to drive further usage and engagement. What we won't be talking about is what's known as shadow profiles. So shadow profiles in the social media context are typically data sources outside of your application that can be used to gather data about your users. We're not going to be going too deep into that today, but I think there's a lot of ethical soul searching that the product community should be thinking about when we talk about shadow profiles and these large scale data brokers. Now, when you look at data, the way to think about it best is actually as a history book. Data can't tell you exactly what's going to happen in the future, but it can tell you what happened in the past. By collecting the data about your users, you're storing that information of what happened in the hopes that that fits some kind of pattern that you can use to apply to the future and drive better and more useful engagements with your users. We were thinking about that history book. Here's Archduke, Franz Ferdinand. You can think about basically, you can look at the history of World War I and say that the murder of Archduke Franz Ferdinand actually led and caused to the outcome of World War I. And with a certain amount of tunnel vision, you could look back in the history books and say, okay, killing a world leader might actually lead to World Wars. Now, the challenge with both of those statements is they're both backward looking. If we were able to go back and reverse the assassination of Archduke Franz Ferdinand, we might know for a fact that that would have prevented World War I or not, except we would have had to run that experiment both times. In order for science to work, it has to be run kind of in a clean environment. You can look back in the past and say, okay, assassination very often leads to a world war with a certain amount of tunnel vision that might be true, but you can't really know for sure looking into the future whether that will happen again because you again didn't have a clean environment. You couldn't run it like a scientific experiment. A lot of other things going on obviously in the context of World War I. Now, what you've looked back and seen is a correlation, and that correlation needs to be tested in the future to make sure that it actually was causal, that those two things were really related in the way that you thought they were. Here's a funny website. It's called Spurious Correlations. You can see basically graphs and things that match up reasonably well. In this case, it's divorce rates in Maine against the per capita consumption of margarine. So if you really want to help couples in Maine, this graph would tell you that you should cut back on the margarine. But similarly to assassination and World Wars, they might look very close together, but you can't really know for sure that the two are really related unless you are able to test in an isolated environment. So with that being said, what can data tell us? Well, data can tell you what areas of the product are getting the most use and how. People vote with their clicks if they are looking at a certain page more than other parts of the application, chances are that's a useful and valuable page. They can tell you how you're doing against key performance indicators and that's what we're going to be talking about for the majority of this presentation, these KPI that drive your business forward. They can tell you the outcome of an experiment. If you can actually really isolate two environments and test them against each other in a scientific way, your data can actually tell you whether one was good or bad against the KPI that you care about. This is one kind of magical. Data can tell you whether or not a feature sucks. Ordinarily, I would ask this as a question, but I'll give it to you right here. Something that happens a lot in applications you'll find is people complain about search and I'm sure you're now recounting every app that you use where search isn't very good. The fascinating thing about data is it can actually tell you whether or not search is any good. The way it can do that is basically whether searching leads to higher engagement on those other KPI that you care about. So if you spend a week and a half improving your search, you ship out a search experiment and it seems like everybody's searching more but people are coming back to use the app less, then search probably isn't very good. Now if you improve search and improving search thus leads people to coming back to the app more, well then your search is pretty good. So it's whether or not a feature leads to more repeat usage, not necessarily just of that feature, but again going back to those KPI. Of the things that you care about does using that thing lead to more of that? And if so, then it's probably a good thing and it's worth more investigation. And there's a lot of things data can't tell us related to what we just talked about. Can't tell you what to do next. Data is not necessarily 100% predictive. It's not inventive. It doesn't your databases, you know, pop up and start going in research and competitors and all the different products in the space and figuring out what they need to do in order to be useful. So that is still product manager intuition. They can't 100% prove calls or relationships between past events as we just discussed. And they can't tell you when to make larger bets. Data can't hint at this. That's kind of a subtle science. But it can't tell you that the thing that you're working on is a waste of time. And it can't necessarily tell you whether you've hit relative maximum, whether you really worked this area to the point where it's really completely optimized and you need to rethink the entire thing. You just have to take that on gut. You can see hints of this in the data, but you can't necessarily know for a fact that it's time to go build a brand new app or make a totally new bet on what your product needs to be doing. So given that, what should we be measuring with that knowledge? How should we be thinking about what we're going to measure and the kinds of KPI that we're going to use to drive the business forward? First off is growth. I think that's something you'll hear from pretty much every startup at this point is everybody has some kind of growth team or some kind of growth initiative where they're trying to do this, boost users and customers. When you have those users and customers, you need to know, of course, how many are coming in, but also where do they come from? So you can continue to expand those channels. Everybody cares about this from every B2B app to every consumer app out there. Needs more eyeballs in order to be successful. Next, after that, once you have those people is actually retention. Are they coming back in 48 hours? Are they coming back in a week and they're coming back in a month? Different apps are going to care about different retention intervals. So if you're going to buy a luxury car, they don't necessarily care that you come back to the luxury automobile shop every week. But they do want to know that when you do come back, that you're going to come back to your Mercedes or to your Lamborghini dealer. And that's, you know, very different than your coffee shop, which wants you back every morning. And then finally, invites. The way that you measure invites is a number called a K factor. K is basically the viral coefficient. So every user in your app creates X number or K number of additional users. So if one user comes in, they invite two of their friends and both of those people sign up, then that person has a K factor of two or that invite mechanism has a K factor of two. So every one user turns into two converted users. That's an extremely viral app. What you really need is just basically K factor could be greater than one to know that your app is viral. That invites are actually creating more users that are converting into more users. So your app will continue to spread and grow virally. So that's growth. Everybody cares about growth. After that, I tend to break apps down into two simple categories. One is a transactional app. This is an app that has a specific goal that it's marching you towards. And then an engaging app. An engaging app wants you around. They want you to come back. They want to use the app more. They want you to take more actions within the app. And we'll see why in a second. So transactional apps are really about conversion. They're really about looking at this funnel. So let's assume two million users hit this site. And after a couple of steps, 250,000 make it to the purchase step and actually purchase an item. The total conversion rate across this funnel is about 12.5%. So transactional app is really looking at the gaps. So whether it's this one million opportunity between step one and step two, or even out and beyond to the left here, the entire internet or the entire market and capability of your site. So that's your whole TAN, what you're pitching your VCs on, all the way down to the purchase. And when you get to this purchase, potentially understanding loopbacks. So coming back from this purchase to coming back to the site or coming back from this purchase and making you, again, viral where you're pushing out referral codes and things like that to your friends. So transactional apps are really looking about funnel optimization. How do we get as many people from point A all the way down to that actual purchase at the very end? Engaging apps are really intended to keep you there. So engaging apps care about the sign in, page views. Time on site isn't a great metric, but what they really care about is the user coming back and the user coming back has inherent value. So once you understand the number of times people coming back, that retention or engagement curve, then it comes down to the specific interactions. Which interactions drive more engagement and which interactions actually will go out and engage more other users. So what are the two most important interactions on Instagram? Things to think about here are posting, the original person who posts. You could think about most social apps, follow a trend that's called the 99 one rule, which says that 1% of people are going to create the majority of the content, about 9% are going to respond and interact with the people who create that content and 90% are gonna be pure lurkers. And an engaging app, that's okay because your pure lurkers are adding value. They're adding value and signal back to the 1% and the 9% who are actually creating and generating content and getting more eyeballs back into the app. As we're at a transactional app, that 90% would be basically useless for you to drive the actual bottom line of the final purchase. Successful apps ultimately have to prioritize one. You have to prioritize either the funnel or the engagement as your core loop. That's not to say that transactional apps don't have loops in them. They do want you to come back. They do want you to refer other people. Similarly, engaging apps have tons of funnels. Just because you're coming back and using a social game every day, they still want you to go purchase coins and purchase the currency and spend more money on the app. But they believe ultimately they have to believe that you coming back more and more is gonna continue generating more and more revenue. So the big trade-off, the way to think about the difference between the two is actually in the case of let's say Amazon. So Amazon has you coming back. If you're a regular Amazon user that comes back and purchases a pair of socks every single week, this would be a very weird user, but you come back and purchase a pair of socks every single week versus a user that comes in and buys a big screen TV once. Now, Amazon might make the trade saying that you are never gonna purchase as many socks or the profit margin against those socks is never gonna be as good as the one that we're getting against that big screen TV. So Amazon might detect that I'm this user and might go down one of these two paths and actually force me into the TV path, whereas an engaging app is going to keep me around. It's gonna want me to continue taking more actions. And that the reason ultimately is that you can determine effectively the lifetime value of a user. And you can say which is going to lead to the higher lifetime value for a transactional app. In a transactional app, they have that very well-quantified and know exactly how much this user is gonna be to me. And if I can get them more efficiently to that lifetime value, it might not be worth having them come back every day. That actually might not be a valuable transaction for me. So it's really about how you think about and prioritize between the funnels and the loop itself. Where, and to think about the counter example here, clash of clans, if I'm coming back every day, I keep playing the game, chances are they're going to get me with the coins or one of the other psychological tricks that they have. So they might run an experiment where it puts a big coin purchasing thing right up in the front of the app. But if that actually causes me to go away and not come to the app as much, even if I spend less money over the phone, so it's just really that trade-off between metrics. It's pretty rare that you actually find a trade-off between your own metrics. Once you set your core KPI and once you decide whether you're transactional or engaging, but that's how they would prioritize. That's what the trade-offs would look like from one to the other. So we'll go through a handful of apps here. Go ahead and answer these mentally and then I'll come to the answers. So what are these apps? Would you consider them transactional or engaging? Obviously Amazon just gave that one away. So hopefully you'll get them pretty quick. That one, next one, that one can be kind of key. Last one, sports betting. That's a really a lot. Australia is a big hotbed for sports betting. So Amazon you probably guessed is a transactional app. They do care about referrals. They do care about me coming back but only in as much as it adds to that customer lifetime with that lifetime value. A subscription fee and engagement probably shows that I care and will continue to renew. And then sports betting, sports betting I would put some more in the transactional. They probably do want me to come back and bet more but addiction probably handles some of that. The interesting kind of corollary to sports betting would actually be more like games of chance like online poker, which I would say actually might be a little bit more engaging than transactional. What we've been driving at here is understanding between core metrics, the activities I believe to lead to long-term success and those are your growth metrics. Those are your kind of core funnel and engagement metrics versus feature metrics. Feature metrics are metrics around a given capability of the site or mobile app. So search, number of times that people search. That is probably not a metric that ultimately adds to the bottom line. It can indicate that other people might be doing more stuff that might lead to more value but if you see that let's say you run an experiment and fewer people come back after searching as we discussed before but they're searching more, you can look at it and say, well, my search, number of searches have gone up. That's a feature metric but my overall engagement has gone down. That's a core metric. Feature metrics always lose the core metrics and that's what you have to continually think about as you're building new capabilities. What not to measure? Clicks, clicks aren't always positive. Sometimes clicking around can signal user distress or confusion. There are a lot of A-B testing solutions and general analytics type solutions that will actually give you clicks as a top level metric. So just a note of caution there. Page views, I think anytime you go to one of those sites that has 15 different slides and you have to keep clicking every single time to get to each new page, that's what they're trying to do. They're trying to drive impressions of ads on their pages. So when you think about that awful experience of clicking through 15 different pages full of ads, that's really the kind of, that's what page views can lead you to do. Time on site, it's not terrible data but I think similarly to clicks it's one that people prioritize a lot within analytics services because it's easy to measure and to show kind of immediate value. It's not objectively positive similarly. It's also positive that, or it's also possible that time on site will actually be because the person walked away from their computer and left it sitting idle or something like that. The last thing to think about as you're stepping into this data-driven world, chances are your data is wrong. As we mentioned at the outset you have to form an opinion about data and that opinion itself can be wrong. The level of precision, you have JavaScript plugins and things like that collecting data. Meanwhile, those are sitting in somebody's internet explorer browser where they can counter bugs and miscounts and things like that. This is one of those things that people use to argue against using data entirely is saying that the data is incorrect or imprecise. That's irrelevant and it's also true that the data is always gonna be imprecise in some way or some form. The most important thing then to look for is actually trend data. So it's not critical that you understand exactly what this value of P is here but more that you understand where that curve is going. Because the trend will tell you whether your app is growing or your app is shrinking, whether you're adding more users significantly or you have a bunch of churn on the back end. So do your best to get as precise data as possible but when you have it, don't stress over a thousand users here or there. Really look at which way your data is trending and use that to form most of your decisions. So for further reading on this, this did come from a blog post that actually came from a Quora question. You could find that full blog post here at the bit.ly link, bit.ly forward slash DD product, data-driven product. Feel free to connect with me on LinkedIn, ask me follow-up questions on Twitter, medium. I also do Quora, that's where I, you know, the original post came from and where I do answer semi-regularly. So please stay in touch and thank you very much.