 Alright, thanks guys. Alright, so, thanks Mira for the introduction. So, just again, sorry, I designed growth product at a better forest and she gave a pretty cool introduction of a better forest already, so I just like skipped the bearing history. But a better forest has been an intense journey so far, so it fueled by rapid growth. By the way, I want to check, can you guys hear me at the back? Because there's a microphone here, I can just blast. Alright, cool. So, a better forest has been driven by rapid growth for the past like two years and I'm not really here to talk about our history and volume guys of it, but I'd love to share a little on some like how we think about growth, right? In particular, a few mental tools that help us like think how to approach, how to run product design growth together at the same time. Yeah, and hopefully we can all take away something useful from this. So, if you backtrack a little bit, as a designer or product person, we've all faced something similar before, right? So, we do our research, we learn something pretty interesting about users, identify a problem, right? And then we ideate a brilliant solution and then we bring it up and get blocked, right? Why? Because we only have one small engineering team, so why work on this solution? We have a thousand other public solutions like mine, right? So, why would this drive revenue? Why would this drive user growth or anything metric that we care about, right? And this is a perfectly valid question because as a product person or designer, we create value, not just for the user's success, but also to help businesses grow healthily. And this is why it's super important for us to be able to think about growth in a structured way for us to make design useful in this regard, right? And that's what I'm here to talk about today. So, whether you're a startup or a corporate, the most fundamental question we'll ask ourselves all the time, right? Is what is the highest impact area we can focus on now, given limited resources? And we can break this down into three really simple questions, right? Which is just with anything in life where we want to strive for something, we want to clarify our goal, right? And then the next step, we want to reverse engineer this goal into the key levels we can pull to influence that, right? And then decide what to focus on. These three really simple steps. And let's walk through that, right? So, the most important thing at the start is to define really clearly what people like to call a north star metric. Who has heard of this term north star metric, right? So, basically, north star metric is it came from people who are selling, right? When they don't have a compass, they point to that one star and use it to guide themselves. That's like the north star, right? And that's true north. But in the case of companies and growth, this is the one key metric that you align your team on to drive, right? And this is super important because when you have one key metric to reference to, it helps you compare ideas later, apples to apples, and it sets the foundation for how you reverse engineer this later on, right? So, for example, therefore it's very, very simply. It's just revenue. So, we are driving revenue, but it doesn't have to be money-related, right? Especially if you're growing something free, right? For MailChimp, it's an email marketing platform and success product, right? So, MailChimp drives weekly active users. So, what does this mean, right? So, MailChimp's focus is a free product. So, it's driving adoption and because it knows its natural frequency of usage is at least weekly. So, if you're not using it weekly and you drop off, right, you're not considered an active user and you're not contributing to this business, right? So, what the track is weekly active users to make sure that they are growing the right numbers, right? So, Airbnb Marketplace, you track liquidity like nights booked at first. For Medium, it's a reader-writer platform. Who has saw Medium before? A chance to borrow Medium articles on the internet, right? A lot of people have. So, it's a place for people to write content, consume content, and it focuses on quality content and engagement with that. So, you can measure what the engagement is, which is the amount of time spent reading on the platform, right? So, this can form the key North Star metrics for your kinds of business, right? And to think about it, it's just a few principles need to be kept in mind, right? It has to be tied directly to the value you're creating for your user and also directly to revenue, right? Simple as that. So, now we have our number revenue. So, you can ask yourselves, how do we grow this? How do you grow revenue? How do you UX the crap-off revenue, right? But it's not so easy to answer because it's too broad. So, what we want to do is to narrow it down and figure out what influences this number at a more detailed level, right? So, take a better forest, what we drive is revenue, yeah? So, for an e-commerce platform, what we do, just with any other company, right, is we go stage by stage, taking this output metric and reverse engineer this into its key inputs, right? And at the very first high level, the easiest way to think about it is just what part of our revenue comes from new customers versus repeat customers, right? And this is interesting because our organizers are talking about how many of our users came from a repeat business new because it's an interesting thing to think about because the strategies that drive new customers are very different for the strategies to drive repeat customers, right? So, you want to account for that, right? The next level, what we often do is to segment new customers into various things like which target persona you're driving, in this case, we segment according to customer acquisition channels, right? And this makes sense especially if the people going through its channel experiences a really different part of your product, they come in different intents and they behave differently, right? And you want to account for that. For example, we track people coming from Google Organic Search, people coming from a referral program, people coming from our paid ads, right? And then this is not enough, so we go deeper. And let's take referral program, for example. So, the next step we want to reverse engineer the key inputs to this program, right? And the easiest way to do it is to walk through the customer journey or someone going through this. In any one week, there's a certain number of people exposed to the referral program. In this case, this is our checkout success page, right? And out of all these people, some percentage participate in the program, right? They convert it and by participate, it means they actually share, right? And they could go through a few steps to actually share and they could drop off any step, so you could account for that, right? And out of everybody who shares, right? Some people share one, some people share three times on multiple social channels, right? So, there's another number to pay attention to, this throughput. And out of every share, you get a certain amount of visitors that come through, right? For example, if someone shares through SMS, if you got one visitor, someone shares through Facebook, you might get 50 visitors, right? So, people call this the branching factor. How much does one share get you? And out of all these people who come to the website, some percentage of them become customers. That's our conversion rate. And these customers spend a certain amount of money per order based on what they buy, based on how many they buy, right? And if you multiply all these numbers together, you get referral revenue, right? And you can keep breaking down this now further over and over again, right? Any of these numbers. But then we are now at a place where we can choose to go deep into detail and ask targeted questions like, how do we increase conversion rates of not just users but invited users, right? Maybe you can personalize the website based on the person who invited you, right? And just use the, just leverage that peer pressure, just keep it on your face to make you buy, right? So, we can also zoom out. What's useful is it keeps us seeing the big picture that just, that's just one out of at least six levels you can pull to influence this one number, right? Which is, then again, it's part of a bigger picture, right? That drives your ultimate number, right? So, to do a quick summary, just going through the process of breaking down your North Star metric into these key levers gives us two important things. So, it tells us what the key inputs that drive our growth, not just random things, right? And it forms a foundation for coming up ideas, right? It's going to realize when you go through this process, you're already ideating multiple things like how to do this, how to do this, how to do that, right? Without even thinking that you're ideating or brainstorming, right? And also, you can, you can choose to focus on this detailed level. You can choose to go, oh, maybe you should add a new channel, maybe you should add a new persona, right? So, it gives you this multilayered approach to brainstorming. So, to show this on a completely different product, right? So, medium, let's use medium as an example. I don't work at medium, so this is the outsider's point of view, right? So, medium, community for reader and writers, for the benefit of people who don't know what medium is again. So, medium is a reader and writer platform. So, you can go on medium, you can write articles and publish them, and they look really beautiful, right? And medium itself is a really strong algorithm that surfaces the best content based on your interest. So, people like to go there and read and learn, right? So, medium wants to grow engagement, and you can measure engagement based on how much reading is being done in the medium community, right? So, you can start to reverse engineer this number. And the simplest vector is, again, you can influence two numbers, right? Grow the community and grow the amount of reading time per user in the community. So, that's the first step of breaking it down. And then, you can again go through the same process, right? So, you can break down, sign user base, how do you grow that? New users who sign in to the community by finding a medium article on Google, right? And people sharing medium articles on their social media channels, that's how you can find medium as well, right? And then, break it down again. So, for people finding medium on Google, it depends a lot on how many articles are on medium, right? Because every single article has a chance to be featured in the search, right? And this times how many articles gives you how many times medium is shown to you in Google search at any one point of time, right? And based on where it is in the search results, right? There's a percentage chance that people will click through and see medium. And once you read the article and you see this little button there to subscribe and get updates from this publication, you really like, you can click on it and then you can just sign up for medium, right? So, if you just multiply this, you actually get this number and you can actually influence, you can talk through design and see which numbers you should influence, right? So, just to open this question to the floor, right? Looking at this, let's say total articles published, right? Right now you are, you went from aggregate reading time all the way to, let's just grow the total number of articles published per week, right? How would you approach this number? Okay? Any ideas? Okay, fine. Okay, so let me just give my, we just go through the little same process, right? And the easiest way to break it down, you can break it out again, right? So total articles published, you can see it's in terms of how many writers are active at one point of time, multiplied by how frequently they're writing, right? So, then again, this is where we can start brainstorming as well. So, how do you grow total writers, right? You can work on the onboarding for writers, right? Some people have been reading non-stop, right? They may have blogs outside, their personal blogs, how do you get them to bring their blog onto medium, right? So you can design campaigns to entice them on the value of publishing on medium to build the audience, right? And create email campaigns, right? For people whom you can identify as bloggers, right? While they are on medium, you could personalize pop-ups, things like that, to get them to think about starting, right? How do you do writing frequency? Writing frequency is all about once you start writing, what's the value of writing more and more on medium, right? Versus some other platform or whatever it is, right? So you could just help them see the value through, say, if you know their job, the reason why they write is they want to build an audience. Then you could reverse engineer what features could help them build an audience better, right? It could be better analytics to understand engagement of your content, so it becomes a more useful platform, right? It could be, if you know that people like to write because they like the attention, then you could just look at how to design the feedback group when people comment on articles, when people view articles, and then you design the notifications that get people feeling really proud of themselves, right? And this can get people to write more, but you realize that we just went from not knowing what the heck medium is to ideating features, really, that kind of makes sense, right? So this is a really powerful tool for doing anything at any company, right? So, but now we have a map, but it's not enough, it's like one lens, right? And it sort of brings back to a better phrase for a while, and here's a scenario, a very common scenario actually, right? So after you went through something like that with your team members, they have started to talk business value when ideating, right? So team member one says, we should work on improving our conversion rate on checkout, right? People, 56% of people, after they add something to their shopping cart online, they leave. This is unnatural, who does that at a supermarket, right? So team member two says we should focus on increasing how much people spend for order, right? Because when they study the customer service tickets, people have been asking for various fasts and suites, right? And we can easily just source that and sell that. So what should we work on, right? Because both requires us to change the website. We have one engineer, what should we work on? It might take a month, right? If it becomes very complicated, right? So the fundamental way to approach a question like this is to be able to compare these ideas apples to apples, not just think about it as individuals, right? And what this means specifically is can you compare, can you determine the estimated impact on your north star metric of each of these things? And then reverse actually the cost to create that and then decide which one is more profitable to do now, given your limited resources, right? And what we do at a better forest, we build a predictive model, right? To be able to do exactly that, right? And here's what it looks like. People hate Excel. Many people hate Excel. I love Excel. So what we do is we took those inputs, reverse engineer, all the ones that matter, right? And we plug them into the model, right? So what this looks like is how many people visited us on, say, organic? How many people decided to stay on the site? How many people viewed product, added product to cut, went through all the checkout steps, right? And then converted. And we do this for the various parts of the model. And then we plug in our baseline data through analytics and connected this to our revenue number, right? So what you can do right now is you can just tweak one number here and then you can see the impact of revenue on revenue over time, right? So, for example, if you believe that you can improve a metric by, you can improve our checkout completion rate by 20% because you realize that a certain percentage of customers have this problem and you know how to solve the problem, right? And then you can just tweak this and you'll see the effect. You can compare between ideas, right? And what we often find as a result is that something that has a, say, 2x increase on an input actually has a smaller effect on revenue overall than something else that only has like a 50% increase, right? And this is very interesting because it removes bias. It keeps us, it keeps our mind on a model of reality that's often more accurate than what we have in our bias minds, right? And the reason why that often happens is because the effect on revenue is not just the effect on the metric, it's the effect on that metric times the number of people that's involved in the metric, right? And some metrics, some inputs like retention and virality over time, they compound, right? If you improve retention, people stay longer, they do more things, they buy more products, right? They tend to refer more customers, they tend to do more things that fits into other inputs, right? Versus say you work on one campaign that affects one week, right? So, make sense? Go. So, I wouldn't go too deep into this. If you have questions, you can ask me later. But what this does for us is not to be super precise, accurate, but to give us a really high level direction for how to make data informed decisions, right? And speaking of direction, very often we set goals without a clear path to hit those goals, right? And that's often problematic. It keeps us, it keeps our confidence really shattered when we just start and then, oh, crap, it's too big a goal, right? So, you realize that with those tools that you went through, it gives you what you need to map a realistic path to that goal, right? Here's what I can show you. So, let's say we double, double revenue by January 2018, right? We set a specific goal that we can measure that's time bound, right? The question to ask ourselves now is what are some possible scenarios we can take to get here from where we are right now, right? So, what we do is to go back to our map, right? And as one tool out of many, go into our user research to validate parts of the map, use the model to see the impact of inputs, right? And then we come out with these scenarios, right? So, if we grow revenue by 100 percent, which we know because we have a revenue program that a lot of people see, but we haven't touched in ages, right? And we believe we can, we can probably do a lot here. If we know that we have an advertising channel that's at least as big as the ones that we have already made work, that we are not touching, right? We can at least double our pit acquisition revenue. If we know that retention is pretty shit right now so we can work on that, if all these things come true and everything else stays the same, we can realistically get this number, right? So, what we do is to map a predictable path, right? With your inputs at the first step. So, now that we're closer to reality, the next step then is to figure out what strategies you want to take to grow this specific number, right? And this really depends on your company or your product, right? And speaking of strategies, it often just starts with research, right? Deeper research. And in this part, I just like to go through a little bit on a few simple experiments that a better for us ran, right? Just to show you what kinds of strategies, tactics we can run to grow. So, who has seen graphs like this before? People know what a graph like this means, right? It's for the benefit of those who don't really do analytics that much. This is a funnel diagram, right? So, what this tells me is that out of all the people who see this step, right? So, these are basically steps through the product, right? People do action one, action two, action three, right? Out of all the people who did action one, only this amount of people did action two, right? And then this amount of people did action three, right? So, this is basically the drop-off of people. This is the inefficiency in your funnel, right? And people call this a leaky bucket because basically imagine if you get a bucket of water and then you walk home, right? It's full. By the time you reach home, only one quarter of the water is left. So, you just wasted all your effort getting the water, right? So, the same thing. You take all this effort to acquire customers and only a certain percentage of them become actual customers, right? So, pretty simple as that. But the thing about this is we noticed an unnatural drop-off from here to here because we didn't expect people to really drop off at this page at all. So, these are actually pages on our website. And that page happens to be the page where people write a message, a short message for the person they are sending flowers to, right? And if they didn't want to write a message, they can just skip it, right? Press next. So, I didn't think this was a problem. I thought it was curious. And until the day when we had a couple of people reach our office, right? And this guy used my laptop to make a purchase. And he actually got stuck on this page for 10 minutes. He was like wondering, how do I write a romantic message for my wife? I don't know. He was asking me for ideas. He was like, then I come out with some cheesy ideas. I mean, that's kind of damn bad. I don't want to. So, he got stuck for 10 minutes, right? And then at the end of 10 minutes, he got distracted. He got up and started talking to someone else. So, this is actually quite reflective of what could happen in real life. Not real life, but like, if he was in his house or at his own office, right? The longer someone gets distracted, the more likely someone is to stop buying. And the more likely the person is to decide later on maybe he doesn't need to buy flowers, or maybe he can buy for someone else, right? So, we didn't want to just take this data point, this single data point as gospel. So, we went back and did some qualitative research, which is really important for finding causality, right? So, what the goal here is is to uncover what's causing the problem with that target input, that metric, that two final steps. Why are people dropping off, right? And we are very focused on this, not just the entire thing, right? So, what we want to do is to study actual users who recently exhibited that behavior, not just random people, we go to Starbucks to use a test, because the context those people at Starbucks will be really different from the people who actually did drop off. You won't know the difference, right? So, through our data, because we collect emails beforehand, we can reach into our database and pull out all those people who dropped off on this set in the past three days. And then, I just send them an email asking them basically, why you drop off, right? And then, they will reply and we synthesize our data. So, a really useful way to synthesize this, there are many different ways to do it, right? It's to do it by frequency, right? So, we notice there are many different reasons, not just that, but in the, we kind of validated our assumption that the biggest reason is not sure what to write. And the good thing about frequency is that now, you know that if you manage to completely remove the reason not sure what to write, you can save maybe up to half the people who will have dropped off. And that's just a way to estimate the impact of that. Should I work on this problem? Should I work on that problem? Why are there some people actually add, if they do deeper research, they can add a level of intensity, like this person said, I'm really pissed with this thing, right? So, you have two different levels, but in this case, you can keep it simple, just look at frequency, right? So, we decided to do a really simple experiment that we built in the day, right? So, we added a button that when people click, right? We put in cheesy lines, like funny lines, romantic, like weird, retarded, whatever it is, right? So, we put in like 30 lines, and people can click until they see the one they want, right? So, the impact of this on our conversion rates, we grew it by 11% throughout the funnel. Just simple as that, just one day of going through research, and the email didn't even take that much time, right? So, what I really want to show you is that it's not that that type of thing will work for your company, but the process of identifying, like, oh, here are the things I can look at in terms of the map, right? And then, oh, here's the opportunity, and how to determine causality for why the opportunity exists, and from there, if you've got a solution. So, I'd like to end off with another experiment. So, if you remember our referral program from the Bell Forest, this kind of referral program says, get your Nix Bokeh free, right? And all you have to do is to share your $15 off gift card with your friends. And when two friends buy, you get your Nix Bokeh free, right? Completely free Bokeh. And all you have to do is share on Facebook, or just share this kind of link, yeah? It's like, okay, okay. So, our analytics showed that our referral program was complete crap. It's like, ridiculously crap. All of all these people who reach our landing page, less than 10% of them decided to actually refer. So, people actually see, like, okay, maybe I'm interested in this, and they don't follow true. They're like, what the heck is going on, yeah? So, we did the same thing. We looked into people who saw the referral landing page and then didn't refer, right? So, these are what we call the marginal users that we are trying to get, but we lost, right? So, we did the same thing. We send the email to those people. But in this case, we also added something else, which is a live chat. Real simple. This is Zendesk chat. And we trigger this pop-up with an automated message that asks, that tries to engage the person when it's on the referral program. And just to see if, are there any usability issues? Do people have questions that they want to ask but don't get answered on the referral program? They're stopping them from doing it, right? So, that's really helpful. So, we get a whole new range of problems that stop people from referring. And it could be, I actually really shared true other means. I don't need your, like, I forgot, whatever. So, I don't want to be seen as a discount hunter. I don't know who to share with, but the biggest reason is not worth it. It's too much effort. And it kind of makes sense because when you think about it, how many people in your life right now want to buy flowers? It's very hard to think, right? It's not Valentine's Day. It's not Mother's Day. You don't know when your friends' anniversaries with their girlfriends are. So, it's very hard to just go, hey, buy, buy, help me buy flowers so I can get free bouquet. It's a hard thing to do, right? So, we decided to change it. And so, we formed a hypothesis that we could vastly reduce what you call this activation energy to refer by not rewarding people when they succeed in getting some to buy, but we drop the reward by a lot and then reward them when they do the act of sharing, right? So, this is the principle of instant gratification. You just need to do it again straight away, right? So, now all you need to do is share it on Facebook. You get $20 credit straight away, right? And then you get $10 for each friend that buys, right? So, so this was, it's still quite shit right now, but then we managed to double our referral rate and our referral revenue as a result, right? So, just to show you the process going through it again, how adaptable it is to many, many different problems that we see, okay? Did you have to confirm your margins and make sure like that $20 is not going to affect that? Yes, yes. So, we have to measure that it's accretive on that, right? So, if you just go, if you just look at things like this. So, we know what we call unique economics, we know the cost of a bouquet. So, we know how many people come through referrals, we know how many people get their additional $20 credit. So, we can map out how much money we are giving out, how much people are coming in, right? As long as we stay kind of like steady, we don't, we don't kill ourselves and we are growing as a result. So, that's a win, right? All right. So, to conclude, this is what we went through, right? So, the first thing, I didn't find enough time metric, right? Many companies actually have many, different metrics for different departments, but it's usually quite useful to be able to have everyone aligned to something important, right? And they reverse engineer your key levels to drive it, like map out those inputs. And you want to quantify your inputs in order to be able to get your baseline for how to estimate impact here. And then, use that to map a realistic path to success and then focus your experiments, right? What's that? Thank you so much. Yeah, awesome. So, we'll open the floor for Q&A. Anyone has any questions to ask? Any predictions to make? Flowers to buy? So, the numbers we get is primarily from Google and Android. All those tags, right? Yeah. Yeah. So, they are from Google Analytics. We also use another software called but not as much as we use Google Analytics because you have far more built-in capabilities with GA for a slight like hours. Yeah. But then when it comes to tracking things like retention, whether someone does certain events, other apps are more useful than GA. People tend to install something like an event tracking tool. Events are actions, like segments, and then you just try all the various analytics software and you see one that fits. Yeah. I'm interested to know after you guys filled to deliver some orders on Valentine's Day, what did you have to do as chief of product and grow to win back your customers? Good question. All right. So, some of you might have seen that happening and it was a square experience. So, after Valentine's Day, if any of you have actually bought from us, for context, we took too many orders on Valentine's Day, actually 20x what we thought we would do. And that was a very big mistake because we actually overreached and thought we could fulfill that. We couldn't. So, we are late for many orders. And flowers, sometimes they got squished because our careers were also overclocked, per se. Normally it's not like that but Valentine's Day, we actually overreached. So, if any of you actually experienced that and if somehow we actually went through a really intense recovery process and if you didn't experience that, just let me know. I'm happy to do something for you. And so, what we did after that day is everyone stayed in office for the entire week doing nothing but customer service. So, we caught all customers personally. We apologized and we found make-up bouquet. Even if it was okay, we sent you a new bouquet. So, second chance to surprise your Valentine. So, on Valentine's Day, you sent bouquet. It's not really a surprise. You sent it a week later, it's kind of a surprise. So, we just try to do our best to make up for it. And it's not easy because Valentine's Day is a special day and it's not just, yeah, I give you free stuff, it goes on make-up for it because maybe the moment was ruined. But then, we just do our best to show that as authentically as possible to all our customers who trusted us for that that we are really sorry and want to do waver it takes to make it right for you. And based on each customer, we did our best to do it right. And so, I realize that's not very chief product of it. So, it is not data-driven or waver it is. So, it's really just, at times like this, you just go down into the business and just sit down there and it's called the whole team was calling. Did you guys get back to finish to get back at least? Yeah, in fact, there were a lot of people who were really nice about it, even though we scrub and nice there. They were like, hey, I understand. And then they told us, actually, a bunch of other file companies also scrub. You see, I saw the news and they're like, okay, cool. They're not the only ones. It's not the right thing to think about, right? So, yeah, and people will say they understand and all they really wanted to know that is that we are trying to make up for it. Sorry? Stop it happening again. Yeah, so there are multiple processes where they put in place, right? So, the reason why that happened, if we reverse engine or what like that, is because we had a certain amount of resources in terms of forest, couriers, things like that, right? They are stable and really strong. So, those could fulfill, say, 500 orders in a week, right? Confidently, all the time, everything amazing, right? And on Valentine's Day, in one day itself, we did 2,000 plus orders, right? So, basically what we did is because we had a flexible process to hire new couriers, basically we were just hiring new couriers on the spot and we were also running an algorithm to optimize routes, right? So, we thought that we could stretch it. We thought that, okay, cool, it's 2,000, so all we need to do is hire 3 more couriers to hit 2,000, but it wasn't as simple as that, right? So, given the lesson, for future hike volume days, we're in advance, we will be monitoring what's our max already and we're just going to stop it. Yeah, we're just going to stop it because the customer experience is more important just like that, a bit more money. Yeah? Yeah, let's put a cap on it. Yeah. Oh, you could funnel them to another drink. Yeah, what do you suggest? Well, like what Amazon now is managing, right? They're putting slots on there, then they're kind of like letting you select a different time and date instead. Yeah, definitely. Yeah, so we have roughly, we have like four time slots in the day, so we have other caps for each time slot, right? So previously, we didn't have it, so that's one of the fixes, so that makes a lot of sense, right? We have like 9 to 12, 12 to 3, 3 to 6, and we didn't have a cap, but now we do. So once night, we know that we have, say, 3 couriers for morning time slot, the moment we grew past that limit, the time slot disappears, so we can't book that anymore. So that's how we funnel. How can you get retention rate for the Valentine's Day Order versus the Longmore Beauty Order? Very nice day orders? Yeah, to see if you guys made up for it. Made up for it? Yeah. Oh, okay. So recovery. So the average, you definitely not, okay, so because we don't actually have a control group, to know what's the impact of making up this, right? So what we could do is just look at Valentine's Day retention versus like normal retention is someone's first order was on, it's May, right? And it would obviously be worse than the typical order and someone who didn't experience Valentine's Day, right? And so on average, people or customers, they buy like three to five times, like people who are really into flowers, they buy three to five times a year, right? So some people buy one to three, one to two times, it depends on the kind of persona and also sometimes how you acquire them, right? But that's like correlation. So we realize that there are people who buy, like people who buy for things like birthdays, get well soon, things like, so not like really events, really fixed event driven things like anniversary of Valentine's Day. So those people somehow they buy much, much more often because they have more triggers over a year to buy flowers and they tend to prefer flowers over other gifts, right? So these people, their retention is much higher, right? For people who buy on set dates, very often, like their max is like three times a year, right? So when we look at this and we can compare, like if we acquire more of a certain kind of user, we'll get higher retention versus another and then based on certain events, like we did really well for a certain week of something or we run a really nice campaign, we're giving out flowers and then people have a extra strong impression of a brand that retention tends to go accordingly. It's not easy to attribute cause that like this costs retention, but we can attribute like this cost, I love your brand, and then that costs retention. Okay, so what do you think about metrics? I understand it's hard to watch and it's safe for example, let's say it was about the record program. So what we showed us is something that is shared through social media, so you can actually track the metrics that it's true and honest, but what about word of mouth, referral I say for example, if I buy the bouquet from you guys, and I have another friend that say, you know, anywhere that you buy good flowers, so I would recommend word of mouth. So how do you capture metrics like this as a measure? So this is one example, and another one is that because you know some metrics is revenue, right? But what about people who say for example, you know, you buy like three, two months, come back, how do you capture the effect on the revenue that's happening like one month later or two months later? Because for me, I don't buy flowers for every week, so how do you actually capture these kind of metrics that it's hard to capture? Okay, so from what I understand, you have two main questions, right? The first question is there's some things happening in the world that is not really captured digitally, as a result you can't actually measure it, like how people normally measure. Then the second question is how do you track retention for single users and its impact on revenue? Is that right? Okay, and I'm guessing you're asking in relation to that excel sheet. Is that right? Or just another context? Or just tracking in general? In general, it's safe for example you have a campaign for any measure revenue, but some rabbits, a loss, a big part of source of revenue may be something that is hard to measure through like a single user journey. Safe for example, if you say for example, you can push recommendation, and then for example, if you push like a very expensive loss, and then I just buy all the costs, and when I receive it, I regretted it, and I'm like, okay, I will never buy it again, because I make this very bad by decision. So this is actually, you know, it's a loss revenue, but you guys will be very hard to discover if you didn't really ask at the same time, why you didn't come back after. So how do you capture things like that? Yes, okay. So I see the pattern here, which is that a lot of these like quality, some part of your qualitative feedback, like actual things happening that's not that we don't actually track on Google Analytics for instance, or Mixpanel, whatever it is you use. So for word of mouth, unfortunately it's quite hard to track. So we have nothing set, really set up to track word of mouth. Then the closest metric to track that is actually direct and organic branded traffic. So people actually coming to our site, new customers that come to our site, even though they have never actually interacted with any other paid channels before. So and we can track this by separating new customers from old customers, because we kind of like cookie the browser, so we know if someone has been to our site before and has bought. So based on how many visitors and then customers that has not bought before, that came through direct channels, that means they type in a bare forest into the browser and search a bare forest on Google, we can track, are we growing in terms of word of mouth or are there other ways or just like sing and add and then later converting. So it's really hard to just segment the word of mouth. And the other way to come with a leading indicator word of mouth is what we call a net promoter score survey. So they call it NPS survey. NPS survey basically we send it to customers and we ask them the question, how likely are you all to recommend a friend? And then people choose one to 10. So this is two parts. One is how much they love our service and when it comes to word of mouth, how likely are they to do it? Is there anything blocking? So we measure that and once they click the number that actually gives us some feedback, why do you give us number 10? Why do you give us 7? Why not 10? And then we get a lot of qualitative results based on that and we focus on growing our number and our ops team is actually focused on growing NPS. Keep growing NPS over time, which means that everyone focuses on customer success. And that's basically our ops, sub-Northern metric NPS. So when it comes to retention, so how do you measure retention? So I think there are a few couple parts to this question like how do you visualize retention and how do you measure it? So how do you think about retention when you look at it? Actually quite typically, so many companies they use like cohort analysis. So a cohort analysis basically just takes cohorts of users and cohorts really depend on how you choose to group them. So the most typical way to do it is to group them based on when they first became customers. So let's say week one, someone becomes a customer, then you can track over say a year, two years, and you can see over the next one week, two week, three week, a month or quarter, how many of them actually came back to buy? And because you track by cohort, you're not mixing this up with people who bought today. You're keeping the core of people who bought last year as an isolated segment so you can see how they're retaining over time. And then we will compare cohort number one, which is last year, all the way to cohort number 20 and see if we are improving over time. And then we segment this based on like short term, mid term, long term. Can we get someone to buy again in three months? If we notice that our three months number keep going down, then maybe people are not happy. If we notice that they are stagnant, so how do we make it better? Can we improve our marketing to create more use cases for forecasting, for example, or just be there when they want it? And then that's how we track it. Did I answer your question? I think the other way to see it is, is it necessary to track that metric? How much impact does it make to your bottom line? Is it worth the effort? Yes. Okay, so to make sure I understand your question, it's actually interesting. So you are actually proposing an idea which is, right, if you could create a solution that helps us connect, helps us figure out the relationships between people or actually help people build relationships between when the relationships didn't exist, but they had maybe second degree connections. And we can connect people together, right? Firstly, we'll get a better idea of someone's social circle. Secondly, when we know relationships, we can start telling people, hey, this guy's birthday is coming. So maybe we can get something for this guy. So there's a question I've never considered. So it's really interesting. So I'll say it's actually useful, but it'll be useful to the level of how deep that data goes and what kinds of relationships. So in this case, the way you think about this, there's one part of it, which is what exactly is the solution. And then once you know, is that you know how the solution will impact various key inputs on your model. For example, I can see impacting a few inputs based on how we design it. The first way is through acquisition. So someone is not a customer, but maybe someone gave you an email. And through this email, you found this LinkedIn profile and you realized that these are all colleagues in the company. And then your LinkedIn profile links to Facebook and you can determine, oh, your colleague who hides in that corner actually his birthday is coming. So get him something. So actually it's a very powerful way to get that first purchase. And this exact mechanism can get you repeat purchases, exact same thing. Just telling people, just telling you that someone you care about event is coming, or maybe someone tweeted something sad on Twitter. So maybe you want to do something to share the person out. So right now, I don't really know how to make that happen, to make the connections. But then if you think about how to think about the opportunity, you just think about, so if that happens, you could do like a, there's no real way to be really precise about that. But if you just think that previously someone has customers usually only have, say one or two people, they buy four throughout their entire lifetime. But if you could expand this to five or six and at the same conversion rates, so you could 3x purchase this if it works out at a max. Then obviously not everyone will participate and not everyone will be as influenced by that. So it's not going to be 3x to be at some level less. So if you go, maybe I could 1.5x revenue, 1.3x revenue. And then so is it worth it to build? And then that comes to complexity. So like, do we even know how to build it in the first place? And then how long it takes to build? And then we could take that and kind of like map out how much development effort it takes. And you compare this to the other ideas that you have and see which one makes sense to build first. Interesting idea. Yes. I'm just very curious like how do you really manage your inventory in real time? Like how do you keep track of stock more to start, let's say for the situation of the one time they were actually, how do you keep track of it in real time? Do you use an Excel or do you use an in-house tool or maybe another technology? Yeah. Have you heard of Mergento? Yeah. Yeah, it's an e-commerce platform. So that's what we use. And Mergento comes built in with some features like that already. So you can set an invent, you can set that you have 50 of that bokeh in stock. And then once it hits 50, it takes it off the side. So that's how we track inventory. And we could set, we could plan our inventory based on, based on how we understand our suppliers are going to change over time. If we know that our suppliers have run out of tillips, two of our suppliers have run out of tillips. For instance, we will limit our tillip bouquets by a lot just to make sure that people don't hesitate. The first number we look at is how much we have and that's ready right now. Not even just like open the box. How many, how much is ready? And we'll use that as the first number. And then if our suppliers cannot get us, and the reason why we do this is because not all flowers when they come from a supplier, they open it, not all flowers look as nice as you think. Even though it's completely fresh, some buds might be smashed because they're tied together, for instance. So we don't use flowers that look like that. So at any one point we might use say 80% of all the flowers we get. And sometimes if the suppliers don't take care, they might give us a box full of crap. So we have to take into account this risk. So we only look at how much ready do we have in our fridge at any one point of time. And use that to track. Does that answer your question? Let me know if I answer a completely different question. So you asked me how do we track our inventory, correct? Or how close to real time? How close to real time we track our inventory? So this is not an Excel sheet. It's all in Magento. So if someone buys Bokeh A, then it automatically reduces the number of, let's say if we say that we have inventory stock 99, instantly it becomes like it. It's just based on Magento. And with the moment it hits zero, it disappears off the outside. Just want to check that I answer the correct question. One last question. You can take a conversation for a later with the German. I'm sure our second speaker. Based on your slides, right? Most of the time you have to make a quantitative data. So you're blasting emails etc. So what is the waiting period? Then how do you validate the result? So if they go by, you think you do some changes, you wait for a few months, they compare, and they say six months, six months before then. The differences will be different. Just want to make sure I understand your question. So the data we get from blasting emails to ask, do you have any problems with this part of the product? Those kinds of emails. So you're asking me, what's the waiting time? That means how long do people take to reply? Is that your question? So there are multiple waiting times. It's a process. The first waiting time comes after we send the email. So people, it could take maybe three to four days before enough people reply that I have enough. For that one, I blasted off to 600 people, and like 65 of them replied, roughly. And there was actually enough data for me to get like a good idea of how many people are feeling a certain thing. And it took about three, four days for them to reply. And then after that, after the synthesize of turning into a pie chart, that's almost nothing. It's just a little bit like Excel. And after that, so deciding what to do, creating the experiments. So whenever we do experiments, what we do is we plan it out. So our hypothesis, for example, for the referral program is that people, we map our hypothesis very clearly. People are not doing it because the friction is too high because the reward is not worth the effort. So if we make it, the reward more worth the effort by using instant gratification because we believe that if someone only gets a reward, and God knows how long, because who will want to buy? If we change this around, we get a lot more. And let's say we can put an estimate just based on a guess. It's not going to 10x it. It's going to be maybe 2x it or 50% increment. And then based on this, and then we map out how we're going to run it. So what's the minimum variable test to do? So in this case, we didn't have to redesign entire UX. If you redesign entire UX, it will make no difference probably because no one said anything about I don't understand what's going on. It's hard to navigate. So all we did was change the text on our back end. So we used referral candy. It's a referral program software. And we just need to tweak a few inputs and we launched it in less than a day. And then so when it comes to collecting data and validating, so there's a little bit of like stats comes into play. And so there's a few ways to think about it. So we don't accept results until it hits at least 90% statistical significance. And to hit that, we need a certain amount of people who actually went through the program at the time. If we can't do an AB test just due to the nature of the situation, we look at pre and post. And this also depends on, and when we look at pre and post experiments, there's no, we won't see success if we make very, very small changes like change the button color because there's too much variation. But if we base it on say like on a psychological input that we believe makes sense, that we believe is a big change, we tend to see really big increments. And basically just goes like this and it goes like this. And if you stay steady, we accept the test. Once enough people has gone through, once you have enough people in the post-stage sample and the pre-stage sample where we map out. And usually this could take about two weeks, one to two weeks, depending on how much the impact is. And this is a little bit of like statistical math. So if you make a really small change, and the change is like a 1% to 2% change, you need like crap ton of people to actually validate your, to tell you that your results is statistically confident. But if you create a change that could 2x your metric, you'll need far less people because the difference is far bigger, it's far more obvious that there's a difference. So in our case, we ignore all the change button color, change a little bit of text thing. We focus on things like that. And as a result, we could shrink test dates. And that's a very long-winded answer. So about two weeks, two weeks plus to validate. Yeah. Yeah, I hope that helped. Sweet. Thanks guys. Thanks for being here. Hope that was helpful.