 Hi, everyone. Welcome to this week's Product School Talk. I'm Cassandra with Product School. As you guys know, we teach product management, coding, and data, and now blockchain at our 14 campuses. We have a very special guest with us today. His name is Joel Healey, and he's the product manager at Patient Pop. How are you doing, Joel? Good. How are you? Doing well. Thank you, and thanks for being here with us today. I know you have a presentation set up for everybody, but before we begin, can you talk a little bit about your background and how you got into product management? Yeah, so I actually got a degree in what all product managers get a degree in, is genetics and plant biology, and decided I wanted to get into technology. So I started working in tech support at Google, and over time I really gained a lot of product knowledge and understanding, in particular in local search. And so when I was brought on at Patient Pop, I was asked to lead our efforts in local search from a product management perspective. So I kind of had that. I think lots of product managers have kind of curly Q ways into product management. That was mine. Yeah, no, that's interesting. Awesome. Well, thank you. And I know I'll give you a couple of seconds to screen share so that we can get your talk started. And guys, whenever he's finished with the presentation, we'll be taking questions right from underneath the comments on the Facebook live video. So here it looks like you're all set to go. Can you see the presentation? It's black right now. Yeah, that's what I saw. There we go. Is that better? Yes. Okay, great. So I just want to quickly take I know there's not a lot of time we gave this presentation at the product school campus in Santa Monica a couple months ago. There's a little bit shorter time. So I'll try to condense it. But we want to give you five tips for what is data informed product design. A little bit about patient pop where I work today, we build performance driven websites, we care about how they do in search, we integrate with a medical office, their calendaring system to make online appointments and appointment reminders. And then we gather feedback from patients about their visit. So often when we think about being data formed, we want to know everything, right? We want to store all the information that comes in. We want to be able to read that information easily and answer any question about that that kind of information. But the reality is particularly when you're at a scrappy startup is that you may not be able to do that. So you really need to focus on the question you have, choose your metrics carefully and figure out how to grab those metrics. So let me take you through five things that if you don't have an ability to have access to everything and read all that information easily, we can kind of use it. So here's an example. One is that if you don't have all the data, maybe you can find out via survey. So at survey is can validate the assumptions that you have. And this is a simple way to do very light market research. So first think about we were looking at how how do people actually find doctors online. So we simply ask these questions in the survey. How did you first learn about your child's pediatrician and what research did you perform prior to selecting your pediatrician? And we found that more than 70% of those people that were looking for a pediatrician for their child was through a personal recommendation. Well for an online business that doesn't sound great. However, when you found out even when people got those personal recommendations, they still did online research to decide to finally make that decision. Then that really validated the assumptions that we had that regardless of where recommendations come from, people still want to find out what's happening with this doctor online. Another example is testing anywhere. So again at Patient Pop we had we have a communication through our platform to our provider's patients. But we realized that we wanted to make quick adjustments and figure out what was really going to improve our feedback recommendation after getting an email. Well instead of making changes to our platform, we were able to use SendGrid to simply test out different types of emails so that we could encourage more feedback coming back to our providers. And ultimately we were able to test things like changing the subject line, personalizing the signature or not changing anything at all. And by doing minor changes, not directly through our platform but through SendGrid, we were able to see an increased conversion rate by 600%. So testing ideas outside of our product first really allowed us to come in with a firm idea of the vision inside of our product. Another thing is feature flags. So sometimes you want to release to a subset and then let that prove out the theory that something is going to help your product grow and improve. For example, we wanted to test out changes to our structured data. So structured data on websites does two things. It provides relevance about the location and category of a business and it also can drive search features. An example here is the stars on a web page you might see on google.com or even Bing if you happen to use that. We were able to do a very small scale test with our changes in structured data. We saw an increase in our click-through rate, click-through rate. We also seemed to see ranking improved. Then we were able to launch this to 200 sites and it wasn't as impressive but we still saw increases in click-through rate with rich snippets. Those are the little stars that I was showing you earlier and then ultimately we were able to release this to thousands of sites and we're very happy with the results. So start small then scale. And finally or the second to last thing is that sometimes your basic analytics aren't going to cut it and you need to be scrappy about the data sources that you need to look at. So for example, we had an issue with how analytics was set up between our properties and we allowed booking on Yelp to our practices directly. So we couldn't figure out what was doing better in terms of mobile or analytics so we had to look into our Apache logs to figure this out. Instead of just relying on Google Analytics, these Apache logs really helped us test out a data set and understand that yes, in fact, patients were more likely to book an appointment to finish booking an appointment on desktop more than mobile. So we decided to adjust our design and think about how we need to improve our design on mobile. And the last thing is occasionally you can't get at data sources that really scale. You can't drop things even into a spreadsheet and just add things up because you have all the data there. Sometimes you have to go out in the wild and count. And this is particularly true. This is my favorite counter from Sesame Street. This can be particularly true when you're dealing with search. So we wanted to look at what websites, what publishers, and you'll see a list of publishers returned in a Google search here on the left, what publishers were providing those stars, those rich snippets, and also were ranking in search for any given provider that we had. So we looked at a large set of providers and then we're able to say which profiles were most important given the fact that they're ranking well and then also provide rich snippets that we had proven earlier in increases click-through rate. And so we were able to have, we put a contractor on this, we had them do hundreds of searches, count up all the numbers, give us a list, and then ultimately we were able to list a group of citation sites that seems pretty important to what we were targeting as a group. So in summary, here are your five tips for data-driven design. One, take the time to do surveys. Light market research can help validate your exceptions. Two, test outside of your product. So if you can't do something quickly in your product, find another source that's going to allow you to simulate what you want to do in your product and see if you can make changes there and prove out the value of your assumptions. Validate prior to release with feature flags. Pick small test groups to start first and then go big when you know it's the right thing to do. Find alternative sources of analytics data. So if you can't answer your questions through the standard data, find what else is available to answer those questions. And finally, if you need to take time to count. And that's it. Cassandra. Thank you so much, Joel. And letting everybody know that they can start typing in their questions in the comments section now and we'll get to as many as possible. So we already had a question come through. And here we go ahead and get through it. This is from David. What have been the best tactics for getting customers to provide feedback? I think just on the specifically on the email, it was useful to make sure that that email had the sound the feel from a provider. So signing the email with the username, the sorry, signing the email with the name of the person that they had just seen in the office was helpful. Also being very clear with big buttons that saying, do you want to provide feedback? Here's an easy way to do it. Yes or no, right? So just give them very clear mechanism to provide that feedback, really increase conversions. At one point, we had the assumption that maybe we should take out the big buttons and the big clear HTML that we had in the email to make it really simple. Conversions actually went down. So having clear buttons in the email were very useful. Awesome. Really good tips. And here's our next question. So this is from as a software engineer, what do you think would be the skills to learn on the job to successfully transition to a product manager role? So not having that background myself, I know that product manager is essentially many with software engineering backgrounds, I think do very well, but they have to be extremely adept at communicating and wanting to be able to communicate both sides of the communication spectrum. You have to be a great listener, understand the problems that your users are experiencing, however you define users or customers, and then be able to articulate solutions so that your stakeholders can buy into those solutions that would be best for those users. And articulating includes being able to do that market research, includes being able to come up with data that supports your assumptions, and then being able to articulate that to your boss or stakeholders or even the users convincing them to use your product the way that you think it should be. Here's another question. How do you develop a deep understanding of customers when there's an enormous lack of data, so due to lack of funding or resources or know-how? Yeah, I think there's at my time, even at Google, we did something called user research and we actually brought people in and put them in front of the product and let them go. And I remember a story that was told about a user that we asked them just to search on Google and they sat in front of the computer and they stared at what you know to be the Google homepage, the logo on the search box. And after a while, we said, is something wrong? What's going through your mind? And she said, well, I'm waiting for the rest of the page to load. And that was in the early days when you had news and everything happening on people's homepage and this really sparse Google homepage was atypical. And I think just asking a person is extremely useful and informative as you start building a product. So you don't need to spend a lot of money to do that. You can set out a table on the sidewalk in front of your company and just ask people walking by, hey, can you just look at this? Imagine this is the scenario. Do this. Show me how you would do it. And just look and listen and try to understand what's happening. Our next question is from Namitha. What are your thoughts on embedding a feedback mechanism for customers within the product and how effective are they? I have seen feedback products installed on many different products. I think they can be very effective at gathering feedback. I think there can be a lot of noise in that. Sometimes things look like spam. It depends how controlled your user base is if they're always signed in, et cetera. Ultimately, I think the power in feedback is how you can respond to those people offering feedback. So if it feels like it's going into a void, they're never going to, you're not going to gain the trust from your users that feedback is actually being incorporated into a product. So one of the things that I've been a part of when we have gathered that feedback and found clear opportunities to act on that feedback, we've been able to email all those customers and say, hey, here's the feedback you gave us. We were listening. Here's how we tried to address that problem. We would love continued feedback now, and that creates this virtuous cycle of feedback iteration and launch at what you really want to get into as a product manager. I've actually had that experience with an app. When you send them a suggestion or something and they respond back, it does make you feel like they're listening. Our next question is from Andre. How do you determine the minimum number of data points you need to make decisions? For example, if you get 10 responses on a survey, is that enough? So it depends. But if you're talking about, I'm not a statistical genius, but you can look at the confidence interval of your survey. I can imagine 10 is never enough. But if you're just trying to validate that you're not doing something that is so obviously wrong, sometimes it's just enough to ask one person and hear that feedback of this doesn't look right. So when you're doing those small user studies, what you're really looking for is you're not making a huge error. That's not obvious to you. When you do large-scale user feedback, what you're looking for is refinement and really how do you boost whatever your key performance or key outcome is in your product. So when you're doing that ladder stuff, I think scale is very important and you'll want to look at statistics and get the right confidence interval and the number of pieces. But if you're starting very small, just a few people can really tell you if you're missing the boat. Okay, awesome. Our next question is from Amir. Based on your experience and the data you've collected, what are the most effective online paid or unpaid marketing channels to acquire customers for your company in the past? To acquire customers for my company. Is that what it says? Yes. Yeah, I mean, I don't necessarily deal with the lead gen side of our B2B business, but I think ultimately, it depends how you look. When I think about patient acquisition for our providers, when we're, like I said, for them, the best kind of lead acquisition they get is a referral from another doctor. So if you're looking at what's the most scaled way to get lead generation for them, it's through search and the most prominent search is Google. I mean, it's like 90% of the stuff they're getting. So that's the easiest way to get scaled lead gen, but their conversion rates are very low compared to someone who's gotten a clear personal recommendation or a patient, a provider referral. So it, yeah. So Google is great for scaled lead gen, but ultimately it's those personal relationships that can turn, that have higher conversion rates. Though I would say, and that's kind of what I alluded to in the presentation, that even if they don't start on Google, they often in somewhere in their decision-making process are using a search engine to discover and understand more about the process. So it's usually a part in the marketing funnel of learning and engaging and really finalizing their decision. Right. Thank you. And our next question is from Yasin. As a product manager, needs to make changes into the products and deliver it to the organization, how can you proceed with this strategy? How would you, how do you prepare for that? How would I prepare to make changes? Yeah, based on, I would assume based on feedback from your customer. Yeah. I mean, I think it's always good to understand your key stakeholders, right? So I would not only take the customer feedback, but I would validate that with the stakeholders. I would say we've gotten this group of feedback. Here's something that they're generally saying, does this feedback resonate? If not, why? Right. And then be able to take that, okay, let's take that into a problem statement. Here's the problem we're experiencing as a result. Let's brainstorm solutions. And sometimes usually that doesn't have to happen with the stakeholders, but if you do have a group of product managers you can talk about that with, that's a great opportunity to discuss what you can do. And then you come up with a product plan. It doesn't have to be super detailed, but it should answer that, that problem statement that everybody agreed was the issue in the beginning. And then you come back and say, here's, here's our opportunity of what we can do. And here's what we considered and don't think is the right fit, right? So that you demonstrate you're showing that you've been considerate about all the options available and think, and you can then articulate why you think this is the best option. Awesome. And it's definitely about being, being able to communicate well. So our next question is from Pruta. How often do you engage users for feedback and when is it too much versus not so much versus just right? I would, I don't have the answer for when is it too much or not enough. Because I tend, I tend to do it in every touch point. And I, and sometimes it's about engaging. Sometimes it's being explicit about feedback. I would imagine you don't want to do it like send an email that says, leave us your feedback about our product, right? That, that can get, that can wear, right? So doing it more than once a quarter depends on the relationship and the type of relationship you have with your customers. But more than even, even a tight relationship where you're frequently communicating with customers. I can't imagine more than once a quarter of explicit feedback is very useful. But if you do send regular customer communications and at the bottom there's an opportunity to leave feedback, then you're, then I think anytime you have communication, if you leave that opportunity open, you know, saying how was this, how was this communication? How are we doing? Then you get a constant stream of feedback. And I think that's reasonable. Awesome. And we have time for one, maybe two more questions. Let's see. Here, this question is from Greg. What books do you recommend to get started as a PM? I can't answer that question. I haven't read any books about being a PM. So I don't know. I love books, but I don't know. I don't know what I could recommend for a PM. Do you have any? Where do you? Well, we actually have the product book that we released pretty recently. Actually, if you guys type book in the comments, you get a free copy. So oh, there you go. Yeah, that works. And if you have any other resources that are go to blog sites or anything like that, then you're welcome to share. And I'm going to look. Get a free book today. Type book in the comments. Our final question here is from... So a lot of the books I've read about business are more business, are broader about business. So whether that's how people make decisions, how people, how businesses fail and thrive and the difference between that. And I think those are important paradigms to understand. Because ultimately, any given product or product feature can be in your mind as a product manager, a little business that you're trying to run. And so ultimately, you want to see how that business can succeed or fail and plan for that. But I haven't read like product, how to be a product manager or anything like that. Okay, awesome. And well, our last question, which would be my last question to you is what advice do you have for aspiring product managers of people that are looking to break into the space? Yeah, I mean, I think, like I said, I think communication skills are really important. Ultimately, becoming an expert, becoming the person that can, that understands your field, understands the passion of your users, and you have that passion yourself. And I think those are key qualities that can make a great product manager. But really being able to listen, really being able to turn that into an actionable passion about, here's something that's going to improve your life and improve what you do for the better. And I think that that's a great place to start for a product manager. Awesome. Well, thank you so much for your time today, again, Joel, and appreciate the presentation and then answering all those questions. Yeah, thank you for having me. Great. And thanks, everybody, for joining us today. Lots of good questions and you can still continue to typebook if you want to get a free copy of the product book. And if you want more information about us, you can go to productschool.com. We'll see everyone next week. Thank you again. Bye.