 Hi, Ryo. Thank you so much for joining us in this product school mock interview. So very quickly, I'll introduce myself. I'm Shareen. I have a total of 11 years of work experience and I'm currently working as a product manager for Zalando. Zalando is a fashion e-commerce platform and I'm working as a product manager in their marketing department. So this is the department which essentially offers brands and you know as the sellers on the platform opportunities to advertise on Zalando. Nice to meet you, Shareen. Nice to meet you too. My name is Ryo Aparah. I'm an aspiring product manager. I have over nine years of experience consulting and contracting as a business analyst slash project manager for clients in the education, insurance and banking space. After I got my MBA in strategic management and finance from Morgan State University, I started my career as a business analyst and I gradually progressed into managing data migration, integration and enterprise-wide projects for clients. My interest and transition into product management, it came as a result of my curiosity and my aptitude for continuous learning and growth. I was a contributor and signatory to the recently launched product management manifesto and I want to take all of these strong experiences to a new organization where I can help them build products, digital products that makes people's lives easier. I'm currently in between projects and my most recent project has been managing data and fund migration project for a Swiss firm. Sounds good. Thanks so much, Ryo, for the introduction. So very quickly, let us just walk through the format for the video. I'm going to be walking through two mock interview questions with you and the first one is going to be more of a problem-solving question. The second one is going to be more of a behavioral question. So I'm going to be taking notes while you answer and then we can take, say, 15 minutes for you to answer the first question and then 15 minutes to answer the second question and then the last 10 minutes can be for feedback. Sounds good. All right, then let's dive right into it. So let's start with the first question. Assume that you are a product manager for a food delivery app and over the last week you have seen that the number of orders which are coming in have dropped by 10%. So as a product manager for this company, what is it that you would do? So I will just walk you through the steps I will take coming in. I would start by communicating in an email memo to my stakeholders and those would be the customer service teams, the operations and the engineering team and senior leaders letting them know that our app deliveries down by 10%. I would schedule a quick call to inform them of the steps I will be taking and these steps include understanding the issue thoroughly in conducting an investigation into why this issue happened and recommending and implementing a solution. The next step I will take would be to provide a rough estimate of when we can expect the result of my analysis to be completed. I would also identify the route courses using a technique called course and effect analysis and this would help us to bring storm around all the possible courses of the problem and dig deeper into the true underlying course and I would also carry out an internal and external analysis of the application's operations to understand costs. The next step will then be defining the hypothesis statement. What is the real issue here? We will come up with solutions. I would divide solutions for fixing the issue. It may be a short-term solution and also a long-term solution and then from these responses I will be able to create a detailed plan with an estimate of when the process should be complete and communicate this to all the stakeholders. Communicating my recommendations to stakeholders would also be another key highlight of the steps I will take and then I would get management approval to move forward with the solutions. I would implement the recommended solutions working closely with operations and engineering and the app developers as well as the marketing team and then the next steps will then be launching the solution in the next, if it's a marketing campaign, if it's an app, the next release and thank you everyone. We will continue to assess and monitor the impact of my recommendation using the defined product health metrics and to see if the changes have helped our product performance and then we'll repeat the steps all over again. If the issue still persists we'll continue to repeat these steps once we're able to identify the root cause and give updates of the outcomes of the analysis to senior leaders in our weekly status reports. Sounds good. Let's start with the first step which you mentioned to understand the root cause of the issue. How would you go about diagnosing that? I have some questions I would like to ask to help us to bring storm on the reason but for the cause and effect analysis I'm going to break down these questions into manpower, methods, machines, materials, the environment and the measurements. So for the manpower, my first question to you would be were there any unexpected delivery delays from our partners and suppliers and if so, what happened? How did we handle it? Sure. So for now the app just sees a drop in the orders. We don't see any change in the delivery times or any complaints about deliveries being late. Interesting. And then in terms of the method, what metrics was affected? We just see that the orders have dropped by 10%. So week on week. And what was the source of the data that there was a dropping orders? Where did we get this data from? So we have a daily metrics dashboard. These are standard numbers which are available across the company. Were there any malfunctions or errors in the database software that may have resulted in incorrect data? So all other metrics seem on track so that there were no tracking issues as such that we can see in our database. So data seems to be accurate. And then for the machines that we use, is this a mobile online desktop application? So we have both the mobile and the web version but the drop we see is across both platforms. Okay. So the drop was in all the platforms. Were there any major future release or updates to the app? No releases in the past two weeks. Any system crashes or outages that may have caused service to be slow? No. Stability metrics look about the same as they usually are. Interesting. Now do we offer the service delivery service on all types of mobile applications? Or is it specific to a particular mobile Android iPhones? So just to get that correct, you're asking if we offer the app, the food delivery app on any specific on either just Android or iOS? Yes. Yes. Or if it's all of them? It's all of them. So we have an app for Android and for iOS 4. Okay. And do we know if there was any of our payment channels reflected this downtime? No. We don't see any increase in the failure rates for our payments. So things are working as is. And in terms of the materials, do we know if there were any supply chain issues with our suppliers that week this week? What do you mean by suppliers exactly? Suppliers are the people that make our orders available. So when we have customers placing orders, they're the ones that we go and pick up the orders from the restaurants, the grocery stores, supermarkets. All right. So from the restaurant's perspective, we don't see any change in any issues with the restaurants either. Then in terms of our environment, did this drop affect only the US markets? Or did we see it across both the US and international markets? So the app is only available in the US right now. And we see this drop across our US markets. Okay. And what is, do we have an idea of what demand is typically like at this time of the year? The reason why I ask is because the decline in orders could be as a result of a seasonal demand for our products. Sure. So if we compare the same time last year, we don't see a difference or we don't see essentially a decline in the number of orders. So this trend is definitely new. It's not something that we can account for. Okay. And who are our competitors? Who are we that provide similar services? When we look at our competitors, I'm trying to understand who are the people that we've been competing with closely on this. Is there anything specific about the competitors that we want to look at? Yeah. That's going to be the next in the next stage of the analysis. That's something we're going to analyze if there's something different that they're doing that we are not doing on our end. So maybe we can talk about them later on down the line. Now in terms of measurements, how often, okay, you mentioned that we check these, these results came from our daily stats. So it appears that we check this, the metrics on a daily basis because that was to know if this was a sudden drop or a gradual daily drop. Yeah. So this was, I mean, this has been happening since last week because, and it's been a gradual drop since then. And do we know around exactly what time the drop occurred? We can check the time that it dropped. But right now I don't have those statistics in terms of exactly what time the drop happened. Okay. And did we have any other related performance indicators that also dropped, aside from orders on app delivery, any other performance indicators that we tracked that we may also have noticed it drop in? You can ask me any specific numbers that you're looking at and I can let you know if we see a drop in those as well. Okay. Because the other metrics is, look at that down the line, other metrics that may have been affected. Then the next step I will take right now is doing an internal analysis of our operations. We just completed the course and effect analysis. For the internal analysis, we know I'm just going to be examining our products, our internal processes and our technology systems and databases. In terms of the products, I would say, were there any recent developments without us food supply stores and the food segments that we catered to? Any recent developments in the news, in the media or just in their operations? No change reported. Okay. And then with our own internal processes as a company, were our food orders late? Were there any glitches in the steps and the actions that we take involved in us taking orders and delivering to our customers? No. So we don't see any increase in the delivery times. Neither do we see any reports of late deliveries more than normal. So it's remained the same. Processes, okay. And for our technology systems, were there any issues with our computers or our servers? Did we discover any new bugs? Nothing which could have led to a drop in orders. With our database, we are also looking at that as well. I ask this because it would help us to analyze our KPIs and to make better decisions. Are we sure that the data in our database is correct and up to date? That's something that we need to look at as well because this could result in incorrect data. So we check the backend systems and the data seems to be correct. So there's no flaw in or no bug in our tracking or any concern with the accuracy of our numbers. Okay. I would also be in touch with the database administrators as well to clarify, to confirm this information. And then this next step I'm going to take with you is the external analysis where we will be examining our competitors that we had identified and then barriers to entry, our suppliers and our customers. Now our customers, these are the users who have been the most affected. We're trying to understand who are the users who have been most affected. The end users ordering food or our suppliers. Do we know which of these categories? I did not understand the question in terms of it's going to be a double-sided impact. So if our customers are ordering less, then it's the restaurants which are going to be supplying. So the impact is on both if that answers your question. Yes, it's double noted. And do we have an idea of who our most loyal customers are? Yes, we do. But from the customer perspective, there is from the perspective of most loyal customers, they are the least impacted by this drop. Now, one of the things I would want to do is analyze most loyal customers and talking with them directly through one-on-one interviews and also send customer surveys to know if they placed orders or did not place orders during that week. So that's why I'm asking that preliminary question. And then in terms of new entries into the delivery app market, do we know if there were any new delivery apps in the market? I've been looking it up as well. I didn't notice any new entrants because there are very low entry barriers to creating a food delivery app these days. Anybody can come up with an app, but do we have been tracking those? So no new competitors have entered the market. We have one big existing competitor and that still remains. There are no new players. Same person. Now, in terms of our suppliers, our suppliers, these are the restaurants, our food retailers, the grocery stores. We haven't seen any new developments in their operations. Are they operating according to the service level agreements that we have in place? Yes, they are. Now, for our competitors, going back to our competitors again, what substitute services are they providing now? Are there any new initiatives or things that they are offering or that they started to offer that we are not offering? This is an area that... Sure. So our biggest competitor, like I mentioned, has started a major marketing campaign and they're offering really aggressive discounts on orders. So that's something I'm noting here because from what we've discussed so far, that seems to be a recurring item. So they have an aggressive marketing campaign noted. Okay. So my next question would then be, in terms of the environment, was there like any natural calamity that may have caused things to happen? Any power outages that may have affected our services? Nothing of that sort. Okay. And then in terms of public relations and the media, because I would want to review our most recent social media comments, where there are complaints. Have we been looking at those? Anything that came to your attention about negative comments or notes on social media or complaints? No, nothing out of the ordinary, nothing that stands out. Okay, noted. So I'm just, as we've been having this discussion, I've been brainstorming and putting all these notes together and what I'm coming up with is that I protest the statement of what the issue is. As a company, we have been providing digital food delivery services to customers in the US markets 2015. And this app is available on the mobile iOS and Android devices across platforms. But last week, we discovered a 10% drop in online food delivery orders on our platforms in the US markets only. It seems like the reason for this drop is our competitors' actions. We have a major competitor. They have started to offer some services. They have been on an aggressive marketing campaign, and they are offering something to customers that we are not yet offering. It sounds like that's what is going on here. That sounds good. So yes, that is correct. That is the reason for the drop. So once you figure this out, how would you manage the situation in this case? Like you mentioned some of the next steps, what would be your recommendations? So I have a few solutions that I put in place. The first one would be looking at our, first of all, target our most loyal customers. Because these are the customers that we know that they would always make a repeat purchase. We are going to come up with our own rewards, our own promotional offers as well. And then I am looking at the metrics. There are certain key metrics that I want us to be tracking, aside from food delivery orders. In terms of engagement and retention metrics, I want us to keep an eye on the daily active users and the unique visitors per day. In terms of the order volume, I want us to also keep an eye on the average number of deliveries and orders per hour. Because when we are doing that per hour, we can instantly, every hour, we can instantly track if there has been a change compared to having to wait for a week or until the end of the day before we note that, oh, we have gone past or beyond the normal threshold of deliveries and orders per hour. And then for our performance metrics, I would keep an eye on the number of support tickets that were opened and closed. These would tell us if customers, we are having an unusual amount of customers calling in, having complaints or things that need to be fixed. These are some of the key metrics I would look at. And then even the Chon rate as well, the rate at which customers stop doing business or renewing or canceling over a period of time. Because that would tell that maybe they are going over, they are moving closer to the competitor side. Then from the DAU metric, that's the daily active users, I would be able to identify who are most loyal customers. I would offer them an incentive. And I'm looking at an incentive that says if you place five orders, and now in this incentive, I'm trying to match or possibly even beat what our competitor is doing. So after doing my research, I'm going to say, okay, let's place, if you place five orders within a month, you get 20% off the next order. This attracts them to patronize our app more. Because we're matching up with what the competitor is doing in this area. Then for our unique visitors per day, I would, based on this metric, pull the data out, identify who are the unique visitors, offer them an incentive that says something like place one order and get 15% off your next order. This would add them to become a repeat customer. So this is essentially basically different marketing campaigns that you're going to be offering to different types of customers? Yes. Yes. Different marketing campaign for different categories of customers, segmenting our customers and then offering a campaign that would be suitable for them. And then from even having repeat customers, they can become more loyal customers. Got it. Yes. Makes sense. All right. So thanks so much, Ryo, for this. I had a few other steps I wanted to just go over with you, just to update you on how we're going to move forward with this. We will, of course, launch this campaign implemented across our different platforms and everything. I'm giving a rough estimate of about four weeks to collaborate with our stakeholders and just put this out there on social media and have a press really public statement. Yes, I would do this and then communicating this recommendations to stakeholders. That's the next step I will take. And then we'll get management approval to move forward with the solution. Our developers are going to be the one to implement this into the different areas of the application. And the marketing team would also, I'll be working closely with the marketing team on this. Once we launch it, which will be out there in our next marketing campaign and the next future release of the app. And we'll continue to assess and monitor the impact of this recommendation using the key product health metrics that I mentioned before. And to see if the changes have helped our product performance. And we'll repeat the steps continuously. If anything, if we discover that it's not even making much of an impact, we will start the step all over, identifying the root costs if the issue still persists. And we'll continue to give you updates, updates you, management, the senior leaders in our weekly status reports so that we can stay on top of things in this area. So that's it for now. All right, sounds good. Thanks. Thanks so much for that, Ryo. All right. It's been a pleasure. All right. So thanks, Ryo, for answering the first question. Now we can move on to the next one. So the next one is a behavioral question. And the question is that was there a time where you had a disagreement with the engineering team? What was the disagreement about? And how did you work through that situation? Sure. So looking back through my experience, I haven't necessarily been in a situation where I was, I had a disagreement with the engineering team per se. But I found myself in situations where I had to convince the engineering team to do something and helping to facilitate a solution to an issue. And that's how I've helped in this kind of situation. And I will talk about my experience while I was with the AIG, Consumer Insurance Benefit Solutions. We were just giving you our address by putting this into the context, what happened, the background information, how I identify the users, what the issue was, the impact to the customers, the tasks, the action, the steps that I took, and what was the resolved outcome. Now, the context of this issue was that we had just acquired a new software product, WinShore, for the group benefits business. And the software needed to be customized to meet the needs of the business. The customization involved identifying functional gaps in the futures and functionalities that came with the software out of the box and translating them into requirements. And that would be incorporated into future releases of the platform. The users for this platform are the underwriters, actuaries, benefits, administrators, claim adjusters, and analysts. And the issue that I observed here was that whenever a user enters text, words or phrases in the tool, and there's an error, maybe like a typographical error in the spelling, the application only underlines the text in red. It does not provide a suggestion to automatically correct the user. I also identified the need to display the definition of words in like a pop-up bubble. So ideally, a dictionary function would be the best, would be needed to be added to the tool. The impact customer for this situation is that documents and information that is either stored or printed out from the tool may include errors and misspellings. And this would require extra time and effort to either audit, reward the text and sentences, and all this is bad for the image and the reputation of the company. It's time consuming. And the task that I undertook in this role was, as a business analyst on this team, my role was to identify gaps in the software, functionalities, things that were missing, and write requirements that were customized and describe the needs of the business and share it with the stakeholders from the technology team. So after I identified this issue with a dictionary function that would be needed, the steps that I took to address this was, I put it as an item on the agenda to be discussed with the team during our work session. Just very quickly, Dayo, who all are the technology stakeholders that you were working with on this? So I had application developers on the vendor, on the, from the vendor side, because we had a top party vendor who was implementing all the solutions and suggestions that we required. And we also had some technical team members in health, the AIG side of the business, who were also, you know, going through all these recommendations, as we're communicating it with the vendor. And then when they implemented, they also go over, they work together to go right to make sure that it's implemented correctly before it goes into testing, which I do, I take as a, I test the platform as well. Okay. Yes. Okay. So I created a case, you know, stating the urgent need for a dictionary tool. And I said, I decided to brainstorm on the possible solutions for this. Sure. One quick question. Since you said that this was an urgent need and you said that you worked on the impact of this, apart from the usability impact, which you mentioned, were you, did you quantify this impact in some way to really understand the impact of revenue? Because I want to understand how we decided on the urgency of this specific. Yes. So I, that's in the, that's in the approach, the action that I took to evaluate the, the, the evaluate, the already quantified the impact. There was an impact analysis, and that's how I came up with the idea that it's going to impact our bottom line. It's going to require a lot of rework when we have documents and outputs from the system that have errors. So that means we'll have to start making corrections or having our developers go into the system and write code to correct the, the templates that were coming, that were coming out of the platform. So this takes time, and it's, you take time and resources from other things that we should be doing anyway, in fact, even our delivery timelines. So, so, so when, when, after we came to this point where we agreed that, you know, we needed a dictionary tool and then brainstorming on the possible solutions. So I had this, I had my a work session with these stakeholders, especially from the engineering team and the business sponsor, and the engineering team from the vendor side, they stated that they had a dictionary function that they had used for their clients on prior projects, and they would want to reuse it again and update it as needed for this application. Well, the engineering team from in-house AIG, they expressed that they had an in-house AIG branded dictionary function that they had used for other consumer applications, and they would like to plug it into the platform to be used again. Well, as a business analyst from the AIG side, I felt that these options would not be efficient and cost effective, and external provider like either Google or Yahoo, they have dictionary functions that can be plugged into the tool. So here are three different schools of thoughts, and the approach and the action that I took was, I carried out an evaluation of the three options, using possible benefit analysis, and also analyzing the pros and cons of each solution. And then I organized another session with the stakeholders concerned to present my findings to the group. The option one, which is using an existing dictionary tool from the vendor, it is not the most efficient, and the reason is because it may not always be up-to-date. It would require regular updating whenever there's a new release of the application on the upgrade, and this is time consuming. It consumes time and resources, and even with collaborating between our company, between the client, my company, and the vendor, they would have to continually have these communications and probably even reminders here and there about the need to update the dictionary. So I wasn't in support of this first option, and I explained the reason why. The second option was to have an in-house dictionary from AIG. This would be customized because it's already branded with AIG. We've used it before. It's a process that we can reuse again, but it will also need to be maintained, and it will also require regular updates whenever a new release is done, and this will cost the business time and money. Sure, but isn't this a separate functionality already within AIG? So was it not already being updated and maintained by the existing teams? Yeah, it is, but it may not be the industry standard. It's something that has been used before, but I'm not even sure that they updated it since the last time it was used for another consumer application. I just felt like we needed something that would provide an industry standard definition of text, and I thought we needed better quality. An external dictionary provider like Google or Yahoo, they have optimized processes for handling product upgrades. They have good customer service. Even when we have new releases, they are better able to manage. This is what they do. It's a technology company. They have systems for maintenance of all their applications, and they have a more up-to-date list of options, because Google updates regularly, and it's worldwide. It's a global standard compared to AIG standards. The one that we have in-house was developed by developers in-house, and some of them have even left the company. I just felt that I preferred an external provider, and I stated my preference for Google Dictionary over Yahoo. They offer more services. It's more widely known, and it's a technology company, and they have stood the test of time, even with their other products. I made this presentation to the engineering team, and I was able to convince them to adopt Google Dictionary. The vendors development team, they agreed with this, and we worked on implementing the solution, plugging it into our application. That was how we customized our application to be able to automatically correct errors and typos. The result of this was that it helped to eliminate the mistakes that came from templates, documents, outputs from the platform. As a result, errors and rework was reduced drastically by over 40%. There was a 40% reduction, where at that time we were able to spend it on other more important things, and that helped us to save time and resources on the screen. I would like to just dive a little deeper on these measures that you said. In specific, you said errors and rework reduced by 40%. So what metrics did you exactly measure after this release to understand the impact? So how we had been before then, we had QA team, we had testers who were, I was working closely with the UAT testing team. So anytime we're testing, auditing the scripts, the documents, we see some of these issues, and then we have a spreadsheet where we track all of this and then take note, inform the development team, and then they make changes into the application. So this time that we spent doing this was drastically reduced because now we don't have all these errors coming up and then the QA team can focus on testing other issues in the application. Because before then, the errors that came from the mistakes and the issues that came from the mistakes and errors in the template, those constituted like a major percentage of the issues, the bugs that were coming out of the system. So that was a major source of bugs. But once this had been implemented, this helped to eliminate a lot of those issues arising from errors in templates and documents. So that's how we were able to track it using our testing, working closely with the testing team and then Excel data. All right, all right. Anything else you want to add, Ryo? I believe that would be all, answering all the going into details of all the steps I took. And yeah, it's a successful launch of the application. All right, sounds good. All right, so Ryo, we have about 10 minutes. I guess we can go into the feedback. So taking a step back, then let's start with the feedback for your first question, which was you're the product manager for a food delivery app and you see the orders have gone down by 10%. So what do you do as a product manager? So the good thing which I liked about your answer was the fact that you covered a lot of ground and you laid out the structure for me in the beginning in terms of how you want to do that. You covered a lot of ground. There were a number of different aspects that you looked at to really diagnose the problem. So that was great. And I also like the fact that towards the end, you summarized your findings and you clearly stated that this was the reason why we saw a drop. Now going back, going into what can be improved, I feel that some of the questions that you gave were repeated again. So the first bit of questions which we answered, no two, we went into it again. So for example, from the supplier point of view, or even the point of view, I think there was something else. I think that took a lot of time, but that has already established that that is not the root cause. The second is I felt that even while going through the hypothesis, I think that structure could have been improved a little better. So when you're looking at different aspects, you split them into too many different categorizations. I think that can be improved more from a product perspective. I think there were some key things which could have been spoken about much earlier in the conversation because as a PM, you would have looked at them immediately. So for example, the performance metrics which you spoke about towards the end, I think that should have been one of the first things that you spoke about. And talk more about those metrics, what other metrics you would have looked at. And even the structure could have been maybe simplified a little bit. The content was correct, but maybe the structure could have been simplified. The other thing which we could have done better was about once we've really nailed down the hypothesis or the reason for the fall, I think then going into different reasons, just looking into different aspects would not make that much sense because we're trying to diagnose what the problem is. Once you found out the problem, then it doesn't make too much sense to probably go and figure out what are the other aspects which you could have looked into. Because at the end of the day, if you've identified one reason, then it makes sense to go deeper into that reason. And the last feedback is from a recommendation perspective. I think when you were talking about loyal customers, whereas in the conversation, I'd mentioned that the drop in loyal customers is not as much. So maybe the focus should have been not on the loyal customer base because the drop has not been on that. So the solution focus should have been on the other types of customers and maybe loyal customers could have been. We didn't need to really go into the detail there. So that's broadly the feedback there. I think just the content was all there, just the structuring and maybe making it a little more concise could help improve your answer. All right. Then moving on into the next question, I think what I really liked about your answer was you gave me, you spoke about the different options very, very clearly when we're looking at you tried to build a good structure around giving me some context, making me understand the users and then going into the specifics of each of the options, and why you selected one particular option. And what I feel could have been improved was I think it towards the initial part, the product was not 100% clear in terms of the usage. I understood that better when you explained how the impact was measured. So I think that's something that can be improved. Overall, I feel that this example was not such a great representation of conflict management because that conflict wasn't coming out that much in the answer. And the third thing which I think would have really helped was around, you spoke about why option, say the third option which you suggested was better. But I think one thing that we missed out on was the costing bit of it because the third option when you're using external dictionary, there is a cost involved in that. So when we did the cost-benefit analysis, we spoke about the benefit, but we didn't really speak about the cost aspect of it there from the company perspective. So I think that is also something that could have been added. We did speak about the impact of it, but I think maybe just at the onset, if you would have given specific metrics that, I mean, we spoke about it, but if it came in the structure of the answer, that would have just made it, that would have just ended the conversation or ended the answer at very, very smoothly and like a logical end to the answer. I didn't get the last part of what you said, given specific metrics. Can you clarify what you meant? The last point you made about given a specific that would close out the response. So what I said that we spoke about the metric, when I dwell deeper into when I asked you further questions, where you explained the exact metric that you looked at, but what I said was what I meant was that if that was automatically, that detail was automatically provided when you were talking about the impact of it, then it would have been a good logical source to the answer. Okay, so it seems like you would have preferred me to talk about that metric in detail at the end of the response. Yes, that's correct, yeah. All right, your feedback is well noted. I'll continue to work on these areas as well, and thank you for being available to walk with me on this journey. Thank you so much, Ryo. It's been great for me as well, and best of luck for your transition into product management. Thank you as well. All the best. All the best. Thank you. Bye-bye.