 Hello everyone and welcome to this week's Product School webinar. Thanks for joining us today. Just in case you didn't know, Product School teaches product management, product leadership, data analytics, digital marketing, UX design, and data analytics courses online and at our 16 campuses worldwide. Every week we offer some amazing local product management events and host online webinars, live streams, and ask me anything sessions. Head over to productschool.com after this webinar to check them out. Today we have an awesome guest presenting. I'd like to introduce you to Sohave Theob. Sohave is a season product management executive with more than eight years working as a PM in different companies. Currently, Sohave is working at booking.com, the world's largest online travel company. Previous to that, Sohave was heading the product team at 7AWI, one of the largest tech and digital publishers in the Middle East region. Sohave is a scrum and agile expert and a certified scrum trainer. He is also an entrepreneur that established and sold websites to Yahoo and D1G. He was also selected as an endeavor entrepreneur in 2011 for his initiative in co-funding, co-founding the gaming company Wizards Productions. Feel free to leave any questions for Sohave in the comments of Facebook and I'll be sure to ask them at the end of the presentation. And without further ado, let's welcome Sohave. Thanks for joining us today. Hi. Hi. Pleasure to be joining you guys. Yes. So let me see. Let me share my screen here real quick. Can you guys see it? Yep. Looks good. Okay. Perfect. So yeah, today I'll be talking to you guys about data-driven product management. I'm very excited to be here today talking to the product school community. I would like to think of myself as part of that community as well. I've been following lots of the presentations here and I'm very excited to be part to be presenting to you guys today about this very exciting topic. So I think to get it started, I think data-driven product management is a topic that is relevant for PMs. It's relevant, of course, if you're either starting your career as a PM today or you're currently a PM or you work with PMs at your current company and also think it's a relevant topic for any size of a company, whether it's a your own startup or you're working for a startup or you're working at a big organization. So today I'll be taking you through a few points and they're coming next 30 minutes. First off, I'll be talking about why is data-driven product management important and why is it important for any product manager that wants to have a successful career, how to become data-driven, what's the state of mind that product managers should have to be data-driven and I'll quickly go through some tactics or methods that you can apply and test to become a data-driven product manager hopefully. But before we get into all of that, let me give you guys a quick introduction about myself. So I'm currently working as a product manager in Booking.com, which is the largest travel tech company in the world and their headquarters in Amsterdam. I've joined them two years ago previously to joining Booking. I co-founded a number of startups, some failed and some succeeded and I learned a lot about using data in the products that I've built along these years. So I started the first Arabic social networking website which was acquired by MacTube, which was later on acquired by Yahoo. Also after that co-founded a gaming company went through starting a startup from scratch and they're also currently working at a big organization with hundreds of product teams and a company that runs hundreds of experiments per day. So before I get into the why and why is data-driven product management important I would like you to think for a couple of seconds of what is the prime job of a product manager? Why does a company hire a product manager? What do they expect a product manager to deliver? And just think of that for a couple of seconds. And to be honest, when I started my career I really didn't take it of it in that way but if you think about it I think the biggest or the prime focus of a product manager is to make sure that they are ROI positive. This can apply for your product that you're managing or the section of a product you're managing or even if you're working at your own startup or you're part of a small startup you need to make sure that ultimately that product or startup or initiative is ROI positive. And I think that's where product management is an area where you need to balance between multiple sides or multiple factors to make sure that you are at the end of the day ROI positive and I think that is where data becomes extremely important because it helps you make sure or increase the chances that you are ROI positive at the end of the day. So why is it important? And I think for a number of reasons obviously and I think one of the biggest or most important reasons is that data removes opinion-based decisions. This is from my personal experience and I've seen also from colleagues and friends that have worked or started companies. I think opinion-based decisions is one of the biggest pitfalls that companies and startups or even sometimes in many cases even bigger companies fall into. And opinions are very easy to form and to come up to come by and going down following like developing a product or a feature solely based on an opinion usually ends up with a failed outcome or a failed experiment or a failed feature. And in all cases regardless of whether it's a small company or a big company that's wasted resources that might have been put into a more important thing that develops that returns value for the company. Being data-driven also eliminates bias. It eliminates in some cases hidden agendas and it helps PMs focus on the customer. It helps PMs bring the voice of the customer to the table through the data that customers generate while they're using the product that you're managing. So it's very easy to remove the bias that usually are part of any discussion and by focusing on the data that helps PMs and people that work with the PMs and the company to focus on what's really important which is the customer. And of course that helps PMs moving forward and deciding what do they build and what do they focus on. And so yeah so I think the biggest contribution of data is that a big part of what products or product managers do on a daily basis is to prioritize usually a big backlog of stories or features or ideas that they get or the people around them get. And basing these ideas or these stories in your backlog on data in the form of either a hypothesis which we'll cover later on in a bit more detail but it helps you determine what is a must have, what is it nice to have and you'll be able to justify why is that so, why is a specific feature a must have versus something that is nice to have. And it also I think allows or gives the product manager the ability to say no which is something that many product managers find hard to do especially at the beginning of their career and even at later stages. It's usually easier to try to accommodate all of the ideas that come along or are suggested to you and it's much much harder to say no to an idea especially if it comes from your team or from somebody influential in your organization. So basing ideas and prioritizing features on data based on their outcome and their ROI makes I think the job of prioritizing your backlog and prioritizing what you want to build that much easier. And I also think the next benefit of being data-driven is it helps you avoid another pitfall of many products that end up trying to solve a customer problem by building new features. And I think basing decisions of what you build on data helps you avoid feature overloads that I'm building features that nobody really needs. In many cases developing a feature trying to fix a problem of customer retention or loyalty or monetization. In many cases it's more about listening to customers and finding what customers really need rather than building out new features based on opinions or based on what we think the customer needs. And data also I think helps product managers balance between all of the different stakeholders. So if you think about it a product manager's job is quite complicated. Product managers sit in between multiple teams and multiple stakeholders and they need to manage the expectations of these stakeholders and prioritize all of the features that might be required by these different stakeholders. So you have tech teams, you have the leadership team, you have other product managers and other teams that might be working on something that you own or related to your area. And also sales teams and marketing teams that also ask for new features to be built. And product managers sit between all of those different stakeholders and between what they actually believe is the priority for their own product that helps them achieve what they're after or what their targets. And having data to back up all of these discussions makes it a much easier task. Because again basing decisions on what gets built on data may help you prioritize all of these different in many cases competing requirements or competing features. And I think the last point on why data driven product management is important is it helps deal with one of the trickiest situations that I have come across in the past and I believe many product managers face which is dealing with hippos. And the hippos if you don't know what it stands for it's basically the highest paid person's opinion. It's usually an opinion of the CEO or the head of product or the head of the sales team or whoever is usually the highest paid person. And they usually have their way regardless of what the data says if data exists in that case. And they manage to prioritize whatever they think should be built next. It might be based on their own opinion or based on what they've seen at a competitor or what their friend or wife or husband thinks that you should be dropping everything that your your team currently works on and you should be developing next. And the way this the way data helps I think dealing with such a tricky situation is that it really depersonalizes the situation. So when it's all about data and it's no longer my opinion versus somebody else's opinion it's easier to explain to that person to the hippo in this case that their idea might not be the best idea. It also helps refocus the discussion about the customer. And yeah it just I think really helps deal with the entire situation because in many cases it very quickly becomes a power play or a personal situation and making decisions based on data and the outcomes that we expect out of that out of that improvement or feature really I think makes it very hard for anybody to compete against or argue against the customer's voice in this case. So if everything is really formed in the company or in the organization as part of an experiment or a hypothesis which I'll also cover slightly in the next slides it makes these types of discussions way way easier. And I think this is no longer the something that we need or the future of decision making. I think in many cases companies or bigger companies are already adopting this type of product development. So for example in booking.com where I currently work all decisions are based on experiments and all of the all of the experiments are formed through hypothesis that is always backed up by data and the bigger the volume of the data that is being used the bigger the sample size or the analysis that or the data that the analysis is based on the more the chances that the experiment or the idea has a like has a more chance to be a successful one. So I think that is more about the so I think that sort of covers why data driven is important and why you should focus on either like at all first thinking in this way and then why you should encourage your organization or your company if you're not part of a company that already thinks in this way to be to adopt such a way. So the next topic is or the next point is how to become a data driven PM and I think first of all it really requires a mindset that impresses me. You and your team and your organization needs to be embracing failure and thinking that failures or they need you need to look at failures as an opportunity to learn and every failure comes with a with something new. So you've either you validate an idea which if it works out or it doesn't work out and you know that that's not the the right way of growing your product or fixing the customer's pain point. This is not something that is easy to accomplish if again if you're working on a company that has this culture and this mindset that is really good that will help you a lot and accomplishing it but if you don't I think it's always easy to start with yourself and with your team build a successful case there and then try to convince others to adopt a similar approach. So for example of a company that has such a mentality where currently I work which is in booking there is a we basically have a yearly budget for for field experiments and this is a and actually like a monetary budget where we know that if an experiment fails it usually it might mean a customers that we've lost or convergence that we've lost or in some cases it might have led to an increase in customer tickets for example which is another monetary cost for us but that that's how far some companies take this this mindset to encourage their teams to experiment and try and fail. So the culture looks like more that it's hypothesis driven it has to be any hypothesis or idea must be backed up by data and you have to define a success metric beforehand before you start your experiment and it's never based on an opinion it can be initiated based on an opinion but then you need to validate your opinion or your gut feeling with data so we will dive a bit deeper into this. So first question is where does data come from and I think this is true to any size of the company so whether it's a startup or a large organization I think you'll always have some sort of way to start to start exploring this data so it can come from qualitative data sources or quantitative data sources so qualitative is for example a really easy one as a conference you might gain an insight from a conference that you attend where you can see an insight from a competitor or an industry trend it can come from also from customer surveys which are relatively cheap to implement and get it can come from your sales team it can come from a customer forum where customers are currently discussing something and that gives you an idea or a hypothesis and then usually that qualitative data insight gives you the signal to explore further and spend slightly more effort into digging deeper which is usually going into quantitative data sources so these usually come from your own database it can come from simply some tools such as Google analytics or other analytics tools that you might have implemented it can come also from if you have experiments or have an A-B testing tool it can come from historical or previous experiments to try and validate that signal or that hypothesis or that idea that you've picked up from a qualitative data source and then you sort of if you find the signal or if you find the validation for that idea in one or two or both of these data sources then you start basically forming your hypothesis and you're expert so this brings me back to the sorry to the topic of hypothesis testing so what is exactly a hypothesis testing so in simple terms it's basically an A-B test that is driven by data which is formulated with the formulation of an expected outcome based on this data so you basically think of a feature or a test where in the B version of that test you want to implement a feature or an improvement and you base that on the data that you've already gathered and you track that as you start that experiment and you monitor does this implementation of or is this implementation of this specific feature moving your metrics or moving your KPIs so what makes a good hypothesis usually a good hypothesis is something that you believe is true but you don't know for sure yet again this can come from your own observations your gut feeling research internal external data can come from any of these these sources or a bunch of them and you have a good feeling that this is something that is uh you believe or your team believes that it's true it's a prediction that you expect to arrive it can be easily tested and it it can be true and it can be false so that's why many of experiments actually fail it can it it needs to include a target group so usually you don't experiment with your entire user set and it needs to be clear and measurable and measuring uh uh measuring your your experiment and measuring the success of the experiment it's super critical and it's and it's also extremely critical to set up your success criteria before you actually start the experiment so before you do that you need to specify exactly which metrics you want to track and in this case you actually need to make sure that you have metrics too uh that you can correctly track and you should never develop an idea let alone start start working on it uh without establishing these success metrics and success strategies because if you don't uh and you're you're really not like thinking of that and that's just just an afterthought uh you might end up spending a lot of precious resources building something that you can't really measure you can't really tell after you've finished building it whether this was worth it was it successful or not so for example ask yourself does my success criteria for this idea lend itself to calculation is it a number compare easily across different experiments and over time it doesn't give me results that help me make a decision moving forward uh doesn't give me enough details to execute and does it reflect our strategic goals so whether that's acquisition of new customers retaining engaging users or growing your product and as I mentioned like setting up your success success criteria ahead of time is super critical it sounds easy and it sounds basic but uh it's so it's very it's very easy to uh basically if you don't set up your success criteria over time ahead of time it's easy to start interpreting the experiment the way the results are actually coming in so if you if you don't set that up beforehand um it's usually what ends up usually happening is that uh you or your team start trying to interpret the data that you're seeing in the experiment uh to justify accepting it so for example you need to ask yourself does my success criteria for this idea um does it give me the uh result that helped me make a decision and in all of these things you need to really be aware of vanity metrics so vanity metrics are very risky and vanity metrics are simply things like page views unique users facebook lights the really metrics that don't give you detailed view of what's happening in your product they're usually high-level trends in your business and they really give you very little on their very little insights on how you can improve improve your your product so a simple way of how you can start experimenting with this or start implementing it is to ask a question again the question can be can come from previous data customer complaint customers asking for a specific thing which ignites or which starts this process and then once you have that question once you start asking that question you start researching data to gain a signal or find a validation that um uh this is a good idea basically and then you form a hypothesis uh and when you form that hypothesis uh you define your metrics you set up the success criteria uh before you start that experiment um and uh you either and then you basically start that you start the ab test and uh you should always define uh think of it the runtime of that experiment um think of uh how much traffic this area of the product gets and how many users do you need to uh to see the change to be able to have a basically a conclusive result most ab testing tools have this built in so but they're all they're all and they provide these like tools that you can use to make this calculation so it's nothing super complicated but it's super is very important to to do that before uh starting the experiment and you basically let the experiment run out um it's runtime and then you look at the uh results and the conclusions and you you draw conclusions based based on the results and you either based on these results you either accept that hypothesis or you reject it and in many cases I would say 80 percent or more of these cases uh hypothesis fail experiments fail which is uh very normal and but it's again that's why a culture that accepts failure or accepts failed experiments is very important because because that's that's usually how it goes most experiments fail and then yeah you try again but then you know that it's either the feature itself wasn't implemented correctly or it was based on the wrong data or it's something that you can scratch completely off your backlog your backlog and you can focus on something else so to to recap just aware of the time we don't have much left so so takeaways key takeaways in these 30 minutes as mindset mindset is the first step without the mindset starting with yourself and with the people around you it's extremely hard to uh to make it work so mindset is the first step you need to optimize for the right metric you need to think of uh which metric is the right metric for this experiment or this feature and then you need to test fast fail fast and adjust fast you can't basically count on or spend a lot of time validating this hypothesis and it's usually a good idea to build an MVP um simplify the idea as much as possible base it on data and run an experiment validate that idea as as fast as possible and then iterate on it or scratch it off and and move on to the next to the next idea and basically always optimize for customer experience and not the effort effort or or the revenue and that's I think basically the the end of of this presentation I'm very happy to take any questions or feel free to reach out to me if you have any yeah follow up questions or or things that you would like to to discuss awesome thank you so much so hey we do have uh quite a few questions here for you if you have some extra time yeah yeah sure sure go on so uh Jen asked what data collection tools do you use to get the most value um so usually um uh in my in the current company that I work in we usually have a bunch of internal tools that we've developed and we also use a bunch of external tools so um so for example hot jar or uh survey monkey uh these are some third party tools that you can use to gather insights so survey monkey you can allows you to reach out to your customers with a survey try to validate whatever idea that you currently have with this set of customers usually is a good idea and then take it further if you get that validation and there's like hot jar you can use that to also pop up surveys or polls within the product as the as the users are using the product and then there's a bunch of internal tools so but it's very easy to um find such tools uh on google awesome okay um Yechin said what kind of degree is required for becoming a product manager computer science statistics operation research and MBA so I've seen a mix of all of these uh and the and in my experience so I personally come from a computer science background uh um I've seen product managers that come from a data science background others that come from a user experience background I think each one of these backgrounds brings in value for the product manager uh and uh it complements uh uh the experience of the product manager I think for this topic being data driven I think it adds a lot of value if at least you can rely on yourself uh to get some data in most cases small orbit companies it's uh like digging deeper and trying to find data uh uh it's not that easy and it requires backend or development resources so if you can rely on yourself at least to find these early signals uh it's usually a very good idea but I've seen I've seen uh product managers come from different backgrounds and MBA is usually not required there's a small number of companies that require an MBA to become a product manager but I've seen uh many product managers without an MBA cool so Chris asked how do you use data to balance resources between addressing technical debt versus new features that's a very good question actually um that's a it's something that I think in big products it's usually something that product managers struggle with for sure uh because usually technical debt uh gets deep prioritized from a product or a business impact perspective uh but I think when it and it's usually uh uh fine I think if you base things on data technical debt usually gets a lower priority usually which is I think fine as long as that is not impacting the performance of the product so if it comes to a point where the technical debt is costing us money or you can prove that the technical debt is slowing us down or causing downtime or slowing development I think that's also data that you can reference to make a case for improving the technical debt situation so I think in both cases data can play your role and supporting either either cases but I think technical debt needs to to be at a stage where it impacts the product in a big way to to to to get to get a priority that's I think just how how realistic it is usually it doesn't get a priority unless it starts affecting the product okay awesome so um Jenna asked again what metrics do you find generate the most ROI so that's a to be honest that's a broad question it really depends on the product itself the the the industry that the product works with so the the the sort of the broad metrics that covers most products are things like daily active users it's conversion a customer lifetime value net promoter score these are the sort of I think the broad metrics that cover most products but it really differs in a big way between products and the industries and the products operating awesome so um Christos said is your team usually diverse in terms of roles scientists designers engineers etc how can you achieve an effective communication with such diversity so yeah so so currently I work with the diverse team of back-end developers front-end developers designers and data scientists and so this is this is the the direct team that I work with and then there's of course the broader team that my team operates in and I think like the diversity is brings in it adds a lot of value into the discussion discussions that we usually have and I think but having a a joint agreement that decisions will be based on data helps a lot to move things forward instead of spending time and effort debating opinions we focus on prioritizing what we come back up with data and if we if somebody really thinks that they their idea has value they need to prove it with data so they need to spend the effort to to try and validate it or convince others to spend that effort to to try and validate it so I think it the diversity is fine as long as you have this joint agreement the decisions will be based on data great so our last question here is from chief how can you define if a metric is good enough so so that's that's a good question I think first of all you need to think of the exact feature that you're developing is and what metric does it affect and there are immediate metrics that features effect so for example if you're changing a landing page and you're changing the entire landing page for what with the hope that you're into increased conversion so one metric could be the increase in conversion but maybe ultimately that increase in conversion should increase and increase in revenue so what what do you track exactly in the experiment you track the end goal or do you track the shorter goal so usually if you experiments have a short runtime and and I would usually go with the immediate impact of the experiment so if you're improving a landing page we should expect an increase in conversion for example or increase in signups and then you try and validate whether that increase in signups increase something else later on down the road and then there's the technical aspects of our metrics currently tracked correctly are they do we trust our data sources that is super critical to be aware of and to be to make sure that you're actually tracking storing the right data and tracking the right and basically reading it correctly from the correct source because again in big organizations usually there is lots of data sources and lots of tables to where data can be stored and you need to just to make sure that you're you have the right metric awesome really interesting thank you so much for answering these questions thanks for our community to asking them to so have you and yeah I think that wraps it up so before we leave I wanted to give everyone some more information on product school and our upcoming courses and events um product school teaches product management product leadership coding data analytics ux design and digital marketing courses and they're taught by industry experts working at companies like google and facebook and in addition to that we offer weekly online and onsite events at our fifth 16 campuses across the us uk and canada we're now in washington dc so if you're in washington dc make sure you stop by and uh yeah if you're located near a campus make sure you stop by one of our weekly events every wednesday and thursday you can also find us on social media at product school and be sure to um check out the product blog at productschool.com to keep up with the latest product management content thank you all for enjoying thank you all for joining enjoy the rest of your day and um i hope to see you next week thank you a bunch so hey great presence bye bye thank you see you bye bye