 Hai, terima kasih kerana menunjukkan webinar saya hari ini. Saya sangat gembira dan menyebabkan untuk membawa kamu hari ini. Ini adalah bagaimana untuk mempunyai teman kamu untuk menjadi jenayah hypothesis. Sebelum kita mulai, saya ingin beritahu sedikit tentang diri saya. Saya adalah Walter Nal. Saya currently a payment senior product manager at AirWallX and I work on the payments product. So we're really trying to simplify global finance and enable businesses to do business globally without having to think much about finance. So a bit of my history, I was always a PM so I'm not one of those that managed to skip and exchange careers midway. So I was lucky enough to start at DataSpark and hook where I worked on recommendation and platforms before moving to booking.com where I did some mobile marketing optimisation and then I moved to Meta where I worked on the enterprise products space. Cool, so that's enough about me. Just want to quickly jump in. So with this agenda, I want to share what is in hypothesis first of all because it can't be hypothesis driven without knowing what it is. Why we should be hypothesis driven in our thoughts and the way that we build product. And lastly, how do we empower the team to be hypothesis driven? So what's a hypothesis? I suspect many of you might be already familiar with this but this was really hypothesis started from the scientific method which is a proposed explanation or a supposition made on the basis of evidence as a starting point for further investigation. So in a lot of the work that we do daily as PMs we always make hypothesis without actually knowing potentially that we are doing them. For example, we say, oh if we change the button to a red colour we believe that the merchants or the shoppers will click more hence increasing conversion rate. So something very simple like that is hypothesis because obviously you have seen something or you have read something or you have seen the data and you believe doing these changes will have that internet effect. So just as with all things we like frameworks and templates so I'm going to have a proposed format for hypothesis here and the proposed format is to achieve our outcome which will implement this idea as this will improve this particular pain point or opportunity and we believe this is a problem because of evidence. And the second part is really important because we want to know whether our hypothesis is true so that statement will be true we will know if we have achieved our outcome when we see a certain metric change or certain feedback from customers where we cannot potentially measure some cases. So this is what hypothesis is and it's really very common in science and I think very common at some companies like booking.com So we learn best true example you an example from booking.com which may or may not be true hypothesis and this evidence so I may have edited to sanitize it a bit but basically this story stems from the time where booking was trying to expand in Japan or any other country which I have sanitized so the outcome we are trying to drive is to improve booking.com's customer experience and business in Japan we should change the hero image on our homepage to one with Asian users as this should solve the perception that booking.com is only for Europeans so I'm going to stop here and talk a bit about the background of what gave this original proposal change so obviously we saw business not being strong Japan in this case and why conversion rate is lower compared to other regions so we did some research and try to understand why is this the case because the data shows a similar picture people come to booking.com and they bounce off so this is the evidence that supports this proposal change so we believe this is a problem due to a higher than average bounce rates on our homepage on the Japan side from qualitative research done in Japan so this second part I want to maybe speak more about the research done in Japan when we spoke to users the users would say oh I come to booking.com and I see European shown on the homepage I think this website is not for Asian people to make bookings or for bookings to be made in Asia so I left so you see that we have a proposal change and we look up why we feel like this proposal change is the right way to go and then lastly and very important we will know that we have achieved this outcome which is to improve customer experience and business in Japan when we see the bounce rates decrease and conversion rates increase in Japan in line with our other regions so this is like a complete hypothesis and it doesn't look like much but behind the scenes to validate and to justify that we should test this hypothesis so I think this sort of brings me very nicely to the next part which is like why we should be hypothesis driven so I think as you saw in the first part hypothesis always starts with the outcome that we are trying to drive so it forces the team to focus on impact most of all we want to do this and this is the outcome we are trying to drive so every feature, every idea is centered around the outcome it is trying to bring so there are no nice to have things it is always trying to do something and then the second part we talked about the evidence based proposal change so everyone's idea has to be evidence guided or has a reason so this part is really important so sometimes people say we are hypothesis driven that means we always have to check the data check quantitative numbers to back up our ideas so as seen in the example we did have data to show it is a problem but then we also have qualitative research that provided the evidence for the change and it is not just always we see this in a number so it always has to be number driven it is definitely not true for example, you may have an opinion which formed because of your previous learnings in another company or your previous learnings from another product and this can be the evidence as well I propose this change because when I was at this firm this was what worked and I believe this will transfer well so that is the reason as well so evidence does not need to be number driven it can be opinions but opinions based on certain experiences so I just want to like highlight that point because I feel when people see evidence guided they fixate on the numbers so that is just something that I want to call out so everyone's idea has to be evidence guided or has a reason and is to be critically tested with real users so which is the last part where we say we need to validate whether this hypothesis is true when we started pushing out it means we believe this was true so always validating back your hypothesis forces you to be very critical about ideas and what you put in production cool and I think the last part people don't really talk much about when teams are hypothesis driven is that being hypothesis driven empowers everyone in a team to form a hypothesis and why is this the case like a guess or like it's not a sure thing you're not putting yourself out there and say oh this is definitely going to be true when we change the button to rate it's going to increase conversion rate it's framed as something that you believe is going to work because of XYZ and it's something that you wish to try and test so it provides that safe space for everyone in the team to contribute to form hypothesis to the roadmap with senior management, with stakeholders when you show that your ideas and experiments are validated you don't just put stuff in the production based on your feeling or based on your opinion purely so I think that's an important point which people don't talk much about and I really like to see the teams that I work with have this aspect cool so on to the next part how do we empower teams so I think there are 2 key elements to empowering your team to be hypothesis driven and I will share 3 today because I believe this is usually the function of great teams but the first 2 are more important for the team to be hypothesis driven so the first one is running in the same direction so what does this mean so we don't want the team to be misguided and have different goals so that's what I mean running in the same direction the team is empowered by vision strategy and goals to define roadmap items and projects as a team so together when we define the roadmap items we are all very aligned because we are all running in the same direction I think this is very important to empower the team to be hypothesis driven because as I mentioned before the first starting point is the outcome and if we cannot align on outcomes there is no discussion on the changes we are going to make so this is like the first key point we need to have in the team to be hypothesis driven and the second one is running in the right direction so we want to make sure that we can test hypothesis easily, we can validate them easily so that we don't just put things out there and we say oh we want to change this, we saw this evidence so we are going to change it and then we don't check back and validate and we constantly want to make sure that we are running in the right direction and in the same direction so how do we do that so the team has to have a culture of measuring impact either via experiments diff in diff methods or other methods just by maybe just monitoring data using a timescale it depends on what your company has capabilities to do and what is suitable for your team and all rollouts have very clear impact and more importantly the team has to have easy access to matrix and this can be user feedback this can be clicks, conversion rates to understand how a product is doing today to make sure we are running in the same direction so the last aspect of a great team is usually that we run very effectively so this is great to have but I'm not going to discuss it today because we want to focus on a hypothesis part so I'm going to deep dive a bit into these two items first point running in the same direction so I really love this story when we talk about running in the same direction and this story illustrated by Alice in Wonderland so I'm just going to read it out for everyone one day Alice came to a fork in a road and saw a cat in a tree which road do I take she asked where do you want to go was his response I don't know then the cat said sorry I don't know Alice answer then said the cat it doesn't matter every single product team because if we don't know where we want to take the product we don't know what outcomes we're trying to drive in the long term or in the short term then it really doesn't matter what's the roadmap to be honest because we just want to ship stuff and we don't really care what we ship it's based on evidence or what so running in the same direction is key before we can even start building a great product so what I'm not going to deep dive into any of this but like because there are definitely experts out there that talk about like how to craft a vision how to build a strategy, how to form goals there are a lot of webinars for that but I'll just briefly touch on each of the components of running in the same direction so the first one is vision and then the second one is strategy based on that vision and then having goals to achieve the vision with that strategy in mind so I'm just going to share roughly the approach that usually works based on my experience so for vision typically for me this is the I would say the most important piece because it involves the stakeholders it involves your team so typically what I do is to have a visioning workshop with the stakeholders and the team all coming together and we come up with a vision so then that forms the basis of writing the strategy and defining the goals which then makes it much easier for us to align on that so strategy typically is something that you do more internally within the team and something that you have to work on a lot yourself which sort of lays out the challenges and approach that we're going to take to solve the challenges and we believe these challenges are stopping us from achieving our vision so defining the product strategy very widely across different teams so that every time you present whatever you have done or your engineers go out and say we are going to do this for the next spring or the next quarter people have that context and that approach in the back of their heads and lastly of course goals are really important because as you mentioned we want to be outcome driven so define yearly goals, map to strategy with clearly defined matrix so this will form the basis for that quarter or for that half so running in the right direction so after you know okay we're going to run here we're going to run towards this particular destination so we have to keep checking back to make sure we didn't make a wrong turn or we didn't run suddenly backwards and we are making progress to our destination so this is like running in the right direction these are the key items usually that are associated with that so the first one is goals and matrix the second one is to provide easy access and then obviously lastly we want to talk about how to define hypothesis sorry hypothesis define tickets and features right so the first item goals and matrix which is sort of like carry on from what we mentioned before having these goals at a quarterly level and then having the matrix that we use to measure them and this can be for example on booking.com how do we measure that we have improved the customer experience so what are the matrix associated with that this can be conversion rates this can be MPS scores this can be things that we measure within our product so define the goals and measure matrix this can be long term and definitely short term as well so to build the culture of the team wanting to run in the right direction we need to provide easy access to these matrix so this is where we need to have the partnership of data engineers data scientist and the different stakeholders where we would potentially have dashboards, matrix and framework define for the team to measure impact so defining the long term dashboards so that we can keep checking in on them to make sure we're running in the right direction to track performance we also want to make sure that the matrix that we care about are instrumented well so the team can easily insert that into any framework just to measure impact it can be an experimentation platform or simply just using a time-based method to check the impact cool, so lastly once we have all these in place we have the goals, we have the matrix we have these easy access to these dashboards we have easy ways to check whether we're running in the right direction we then want to make sure that we are framing tickets and features so that they are tuned towards the right direction to always frame features and tickets as hypothesis to test and any feature development is actually a hypothesis, even bugs because I think this is something that most teams don't do, which is like when as a bug, we fix it but do we actually know where it's going to fix the bug and why do we feel like it's going to fix the bug so framing bugs even as hypothesis sort of like builds the culture where we don't know whether this will work but we definitely something that we want to try and fix after the fact alright cool, so just the last word because we talked about what's the hypothesis why be hypothesis driven we talked about how to empower the team which is if you cannot remember anything just remember you need to run in the same direction and you need to keep checking that you're running in the right direction because in a marathon you don't want to run extra so I just want to provide some last words and tips before I close off the first one I want to say is start small don't try to aim to force everyone to start writing tickets in hypothesis I think start small to build culture and set the example for yourself features should always be framed as hypothesis and try to build the mindset that every hypothesis is a chance to learn and it may not always have the internet effect that you want at the start and that is no one else's fault that's not like oh look, all your hypothesis doesn't work there are learnings to be cleaned from failed hypothesis there are learnings to be cleaned from failed experiments so we need to have that mindset that not all of them are going to work so that will build the culture where the team feels empowered to craft their own and to contribute with each other ideas so my next point encourage the team to contribute to request create a process and a safe space for the team to contribute hypothesis this will make your product development process and your team much more vested in the success of your product coach the team to think and write ideas in hypothesis I think this is really important because this may not be a common concept which the engineers or the designers are familiar with so don't be afraid to frame big ideas in hypothesis as well so I'm going to end webinar with one last story which is you would imagine this question should we outsource our customer service or highest in-house agents like a huge question something that the management that the head of product and head of ops has to decide and typically you will not frame it as hypothesis to test because how will you test whether customer service or highest in-house agents but I believe that is not true you can definitely frame it as hypothesis first of all we have to think of the outcome are we trying to drive customer satisfaction are we trying to drive down cost what is the main outcome we are trying to drive and why we believe that in-house agents or customer service are better suited to drive that particular outcome and then we test it we can definitely test it in experiment we split our traffic half sense to in-house agents half sense to outsource agents and we see that outcome that we are trying to drive has been driven successfully or not or better serves us better in terms of achieving outcome so you can see something even as big as should we outsource our customer service or not can also potentially be tested and frame as hypothesis so don't be afraid to say oh look we don't do strategy we don't write hypothesis for strategy items because it's just not done this way we don't need to do that for tickets and features I believe framing hypothesis critically about what you're trying to do what outcome you're trying to drive why you want to do it and then like how do you know you have done it so I believe it's useful on all different levels so on that note I want to thank everyone again for listening listening to my spiel on hypothesis and I hope that it's useful for everyone and here's to everyone becoming more hypothesis driven cheers