 Hello, everybody. Good morning. Good afternoon. Good evening, depending on which part of the world you're joining from It's great to have all of you here today. My name is Ateesh. I'm a product lead at Wish and Today, I'm going to talk about launching minimum viable products So let's get started Before launching into MVPs and talking about a few examples from my days at Wish a little bit about myself And my product management journey. I Started as a product manager almost 10 years back at Samsung working on their galaxy line of devices Focusing on the content and apps and services we had on the phones Moved on to eBay where I was part of the payments and checkout team tasked with creating great checkout experiences for our millions of users and For the last four and a half years have been with Wish Working as a product manager on the payments and post purchase experience in the beginning and then Moving on to focus on the upper funnel or the core app experience Including the search and browse and navigational experience that all of our users experience on the Wish app and website So that's a little bit about my product management journey super excited to be talking about MVPs today and Glad that you could attend today's webinar All right, so what is an MVP? It's one of the definitions I found on the web, which I think captures the essence of it MVP or a minimum viable product is a version of a product with just enough features to be usable by early customers Who can then provide feedback for future product development? There's a lot to unpack here and this graphic does a pretty well pretty good job of illustrating so let's say your Objective is to build a car or build something which takes people from point A to point B What an MVP is not is is building the wheel first then adding the axle and adding the body and Then ultimately adding a roof which gives you the car because a wheel isn't really useful in Helping people get from point A to point B neither is a couple wheels connected with an axle So, you know a way to violate one of the key principles of a minimum viable product, which is that it should be useful And you get the gist what instead could work is To get from point A to point B if you develop a skateboard it still does its job and then you get Feedback from the customers, which could be on the lines of hey I want something which is faster or I want something Which takes less effort and that's how you get to a scooter or a bicycle or a motorbike And then the feedback might be on the lines of hey, what if it rains? It's not really comfortable It's too hot or if it snows I want to take my friends or family on a ride and there isn't enough space That's how you get to a car, but the core problem is still Solved by the skateboard which is take someone an individual user from point A to point B So that's the essence of a minimum viable product It's useful it lets you get something out quickly with not a lot of resources and Allows you to get feedback from your early users who then help you iterate and improve the product and take it to a final Version which hopefully aligns with what you had in mind. So so that's what an MVP is With that let me dive into a couple of examples from my days at wish Which I hope would really illustrate and drive home the point The first example I'm going to talk about is our customer support chatbot, which we call the wish assistant This was something which we launched the MVP launched in late 2018 It's been a couple of years since then The problem that we set out to solve before this existed was that So let me take a step back wish for those if you don't know is an e-commerce app It's a marketplace where we connect buyers to sellers And like any marketplace when you buy something or even without it you may or may not have issues With that order and when you do have issues, what do you do you contact customer support? So pre wish assistant days all of our customer support Contacts would result in a in a ticket in a customer support ticket, which would then be solved by human customer support agents so This was obviously not a very cost effective solution because every ticket would cost us some money What we also realized over time was that for a lot of these issues that our users faced Our agents were giving pre-formed canned responses when it's not to diss on the agents It was the nature of the response like it was a very open and shut case where they needed to Understand where their item was or delivery status Maybe they didn't take a look at the app where we showed it to them They would contact customer support agents would literally point them to that page or copy pasted in their emails So we realized that even though we were paying humans to do customer support a lot of these were canned responses Which the hypothesis was that we can automate and give a better user experience So our minimum viable product the first version that we launched was a very simple decision-free based automated solution which would handle the some of the top issues that our users faced and As the screenshot here so shows You know, it's a very simple Conversational chat kind of an experience where based on a particular order The chatbot might give you some responses You pick one of those responses It might then ask for more questions or more answers from you and you go down the floor and hopefully it ends in a resolution So let me walk you through five elements of the MVP that we used to build this product Number one was focusing on a subset of problems so Customer support and compasses a big area any issue that our users have with anything related to the way shaft goes through customer support Which includes issues with the item issues with the shipping or delivery even issues with their account Or maybe, you know, somebody hacked into their account and they need customer support to solve it So what we did for the MVP was not to take everything in one box But we told ourselves that we need to focus on a subset of problems Which we believed and our data told us would give us the most bang for the buck So in our case it was Issues specifically related to your item hasn't reached you even though it's past the due date and we had a lot of Support tickets created due to this reason and as you can imagine Oftentimes it either resulted in a refund or it would result in the customer support agent Showing them the tracking information saying, you know, the due date is actually not it's not yet past the due date And there was an error on the user's part to maybe look at it and things like that So we focused on a subset of problems and didn't take the entire customer support domain to begin with Number two, focusing on basics. So the first version of our chatbot didn't have any fancy natural language processing or machine learning It pretty much had preformed responses and followed a very deterministic path Where users had to select responses and the chatbot would reply back and you go back and forth hopefully resulting in either a resolution or in a lot of cases we would just route it back to customer support where the old process or the flow kicked in Number three Minimal resources. So we built this entire chatbot experience on web as opposed to building it on native iOS or Android this despite the fact That which is a primarily mobile platform where 80 to 90 percent of our users are Using our app. What we did was we embedded this web-based experience into our apps So it gave us two benefits one was it allowed us to release on all different platforms I us Android as well as web simultaneously and It allowed us to use far fewer resources than we would have if we developed individually for each client natively so this was actually There wasn't any negatives to doing that Maybe a little bit on the performance or are the design side of it, but definitely for an MVP It was a decision which paid us rich dividends The fourth element we utilized was getting rapid feedback So not only did we embed click analytics into the chatbot Which told us how users navigated through different paths where they dropped off Which ones were the ones which took took them maybe longer to go through and they had to answer a bunch of questions We also had a qualitative thumbs up thumbs down score at the end of the of the flow Which gave us another qualitative metric to add on to just very hard quantitative click metrics and conversion metrics And we use that to improve You know our our feature set or our experience going forward and that gets me to the last one rapid iterations All of this we knew we started with with a subset of problems and we got data From our users as to how they would would navigate through this chatbot and we used it to Solve new issues or other issues that the users might be having outside of the item not delivered Which was our biggest one to start with We use that data to automate more flows where it felt like a human customer support agent wasn't really adding a lot of value and we could Give the user a resolution based on what the system already knew of that particular order So so that was our wish chatbot experience and I described the five Elements of the minimum viable product that we use to to launch this net let me now Talk about another example from my payments days at wish Which again will hopefully illustrate what an MVP is and how to go about launching one So the product in question is a buy now pay later product You might be familiar with it Firms like clarna a firm even PayPal in its PayPal credit product offer similar features In our case the problem we wanted to solve was how do we reduce dropout before the purchase or how do we Solve the problem of people not having enough funds in their credit card or debit card Maybe because they're waiting for their for the paycheck to come or they don't want to you know Go over the limit of their credit cards, but yet want to make a purchase because maybe there's a deal or they have a discount Which expires sooner than later? The crux of it being like any e-commerce company. We wanted to make Our platform more affordable and more frictionless for our users So adding some kind of a buy now pay later flow made a lot of business sense our minimum viable product was a very simple version of buy now pay later which Let users pay Within 30 days of making the transaction. So that was essentially it you we offered it to users They would choose this option and they would pick a date within the next 30 days on which we would automatically charge their credit or debit card and That's about it Going through the five elements number one The bigger problem here is the bigger opportunity here is affordability How can we make our platform more affordable for users by offering financial solutions? In the spirit of taking or tackling a subset of problems We tackled a small set here, which our data told us would really give us The feedback we needed from our users and help us iterate So the number so the couple of things that we did which we chose to focus on was this was valid only for purchases up to $50 so we didn't we kind of limited our risk by using that threshold and As I explained in this previous slide, you could only make one payment So you don't pay anything upfront you make one payment within 30 days, which is for the full amount and that's it So it helped us narrow down the focus The second point in this case is kind of related to the first one We focused on on basics of this by now pay later proud program. So again, there were no installments You couldn't slice the payment by three or four tranches. There was a very basic eligibility criteria Behind the offering this payment Option to users. So in our case just to give you an example, we didn't have like a fancy risk engine built to really understand Eligibility what we did was something very simple but effective like are you an existing user on wish? Who has some history of successful payments and has spent maybe let's say three to six months being a wish user? That by itself increased the trust level by a significant amount as opposed to somebody who just signed up for a wish account And maybe they are false fraudster and trying to capitalize on this by now pay later feature So again, nothing very complex but simple and effective which helped us get out of the door much faster The third thing in our case was we actually built an in-house solution Very product and engineering driven and this was one of the key reasons behind doing that was a it was faster And be it took us much fewer resources, especially on the non-engineering side Had we decided to partner with a clarna or a PayPal or an affirm the amount of Paperwork or contractual agreements we would need to go through Would have taken a toll on us given our Staffing or resources at that point. So you see so so going for a more engineering driven in-house solution Was the the better decision for us back then in terms of resourcing the last couple of points obviously this being a Very this being part of the core payments flow We could track how many people are using this What's the adoption and adoption defined as if you offer this to 100 users? What percentage of them are even taking this up which gives us a sense of is this even something useful? because if it's not no there's not a lot of Even if it does really well for that small percentage if the adoption is not big enough It may not be the best opportunity for us to go after Second very important metric was default rate. So like I said, the users had the ability to pick 30 days within which They would make the payment if they didn't we would consider that a default So what was that default rate and is this is that low enough for us to continue to iterate and scale this program? If not, what can we do to reduce the default rate? Last but not the least Does this really help users come back to wish more and make more purchases? So what's the repeat rate? so we would we would analyze all of this and As opposed to the the chatbot this had a slightly longer cycle because of the 30 month lag between Buying something and paying for it or up to the 30 month lag 30 day lag So we would run this experiment or we had to analyze the data for a couple of months before we really got a handle on these Metrics, which told us it brings me to the last point how to iterate and some of the Iteration examples or some of the features that we started adding based on what we saw the MVP do was number one to at a late add a late payment feature So people who who would miss their first miss the payment schedule on that particular date We realized once we add a Feature to to let them know that that payment had failed maybe because The card being expired or now get not having enough funds and and making it super easy for them to make another payment In lieu of the original one significantly reduced our default rates another example is We started moving away from this only one payment to slice it into very pay half now and at the time of purchase and pay half later And then obviously the next natural progression was towards installments the third one Once we launched this feature and we made sure that the hypothesis was right adoption was decent and we were able to get the default rate at a Acceptable level we started adding more Smarter risk rules since it of a very basic Are you wish users have you used wish for six months and have you bought something before we started looking at more data? About your browse behavior and purchase behavior, which we could use To determine on a risk spectrum. Where do you fall so that we can we could hopefully be doubt in the more risky users? so Like you see both with This by now pay later as well as the wish assistant by focusing on a subset of problem kind of reducing the complexity being Very conservative with resources so that we use the fewest amount of resources as possible We were able to launch something which helped us get out get Out of the door very quickly and then constantly getting feedback from our users To see how it's performing and what's the next natural evolution of the product we were then able to rapidly iterate on all of these and Just to kind of bring this full circle both of these products were launched in that in the 2018-2019 timeline and a Testimony to this approach is that these are now full-blown products with with more product managers and designers and engineers working on them and we obviously continue to improve them but From from 2018 2019. They've already evolved Come a long way. We have added much more smarts to both of these products they continue to serve our users and It's it's just a just a really good example to see the value of starting small but making something useful and then Taking it to to its ultimate vision So key takeaways from What I talked about through these examples number one tackle a subset of problems and focus on the basics This is really really important because if you don't define the problem Well, if you if you kind of blow it out too much, it's it's going to be really hard to follow those MVP elements So break down the user problem. See where the biggest opportunities are You don't need to solve everything at once going back to the car analogy or the skateboard to car analogy If your biggest problem you're that you're trying to solve is to get someone from point K to point B Forget about it being convenient or if they are sweaty when they use it or if it's fast enough like those things can come In the in the in the future iterations. So that goes to my second point reduce complexity in functionality usability and design So just just keep asking yourself. Is this really essential in getting that one job done or that one problem solved? Number two be frugal with resources This is true in general in a lot in a lot of product driven companies You're always start with resources and as product managers, you need to prioritize but Particularly so when you're testing a hypothesis and building a minimum viable product So in the case of minimum viable products hacking a solution could be okay Which means, you know cutting corners Maybe when you're designing it or cutting corners from an engineering point of view not doing not coding It's really right because it helps you get out of the door faster or significantly reduces there But those are okay. I'm not saying you should aim to do it But if that compromise needs to be make made, I think that's okay As long as you remember to fix it and do it the right way once you decide to scale and move out of the MPV And this is really important There is a risk of failing to do it and then once you ramp it up to all of your users instead of the early 5% or 10% Things start breaking because you didn't architect it properly So it's it's very important to to make sure if you're if you're hacking it fix it before you scale up I talked about the second point earlier, which one is faster building in-house in-house versus integrating with your third party It will depend on a case-by-case basis So It's it's this this really not a one-size-fits-all solution But keeping that in the back of your mind and making sure that some of the decisions for launching an MVP include this particular aspect is very critical Third one. I talked about native app versus web development In our case as I showed in the example web development made more sense for the first case in the second one Actually, we decided to build it natively I think it was just not possible to build it at once for all three platforms But then we decided only to launch it on Android and iterate only on Android Before scaling it up for for other platforms Getting rapid feedback from your users another key element of launching MVPs strive to get rapid feedback If possible real-time or pleased within a day so that you're not waiting for a few days or a week before you get data Because that will just slow down your process Build analytics into the product in our case. We had a relatively mature overall product So we already had things like clicks and impressions and the system or a framework to log those and analyze those In some other case it might make more sense for you if you're really starting from scratch To embed something like a Google analytics or some third-party analytics, which would be much faster give you much better visualizations Rather than building it in-house. So again, there's always that in-house versus Integrating with a third-party conundrum Which also flies in the data analytics part that the second point I wanted to highlight with in a lot of cases like the special especially the examples I gave I Was building a product within the bigger app or within the bigger website So it's important to see how your users react overall and not just focus on that particular domain It's important because at the end of the day as product managers Your success not only depends on the success of your product or feature But the overall success of the company in our case the overall success of that So if the if the chat assistant makes people browse less because they're spending too much time for whatever reasons Or they're getting frustrated That's not good for the overall health of the product Even though it might not show up in your core metrics of how people are using the the chatbot So it's important to look at the whole picture I think that's true for any product experience, but also for an MVP not to get sidetracked by being Just focused on on your particular product or feature and not looking at the bigger picture Last point Quickly iterating based on all these steps. We start with a small subset. We get ourselves We launch a product quickly and we start getting feedback. What do we do with that? so it's very important that you take the feedback and Keep identifying what's the next problem to solve but also make sure you're understanding the the hypothesis behind You're validating the hypothesis behind the initial MVP our users for example using it before you decide to really scale it to all users and all platforms Don't go for small incremental optimizations on an MVP stage I don't think it's valuable to let's say tweak the design a little bit because it will make it pretty and maybe it'll convert users 1% more Instead test the limits So in the case of that buy now pay later if if I had to Make changes I would increase the limit from 50 to maybe 100 or 200 dollars to see okay What does that do to the default rate in the user experience? I wouldn't be testing Increasing it from 50 to 55 or reducing it from 50 to 45 to see any Incremental changes in the default rate for examples and that that doesn't go well with the spirit of doing MVP Yeah, so so that was mostly it some of my closing thoughts We talked about launching MVPs. I remember you're still solving a problem. You're still building products So anything that you've already learned about product development hypothesis building design still hold true. I Think this discussion was focused mostly on condensing all of that into something you can launch faster into something which will give You the chance to get feedback. So reducing complexity focusing on basics is is what's important But you still have to solve the problem still needs to be worth Solving for and you still need to apply all the basic product building principles So pick the most critical job to be done get the job done even if it's not very pretty skateboard Not a great comparison to a car. It can still get you from point A to point B And obviously once you launch this Measure measure everything monitor everything and then learn from it and keep reading that back So that's it for today. Thank you again for taking the time