 Hello everyone and welcome to this week's Product School webinar. Thanks for joining us today. Just in case you didn't know, Product School teaches product management, coding, data analytics, and digital marketing courses online and in our 15 campuses worldwide. On top of that, every week, we offer some amazing local product management events and host online webinars, live streams, and ask me anything sessions. Head over to productschool.com after this webinar to check them out. Today, we have an awesome guest presenting. I'd like to introduce you to Lillia Gorbachev. Lillia is currently a product manager at Intermedia in St. Petersburg, Russia. Previously, she has worked as a project manager at Yendex Money and a software testing engineer at both Logic Explorers and EMX. Feel free to leave any questions for Lillia in the comments and I'll be sure to ask her them at the end. Without further ado, let's welcome Lillia. Thanks for joining us today. Hello everybody. Thank you for having me today. And I'm so excited to be here. So let's start. So today we will talk about testing with partners, how to test with partners, how to work with partners, and how to select partners for your testing. In most cases, we will be talking about beta testing. So before we start, just several words about me. I have more than 10 years in IT and I started as a QA engineer. And still, software development, software engineering is my passion. That's why I'm still here working as a product manager. So how we, we will talk about how to attract partners, how to be attractive for partners, as a beta testing provider, how to plan a beta, and what we should do and what we shouldn't. So let's start. When I talk about partners, I have in my mind several types of partners and it could be a value-added reseller, it could be an OEM, it could be an MSP, it could be a referral. But for our discussion today, in most cases, I would refer to MSP and value-added reseller because these two types of partners provide services, but they are working as independent providers. And they are the most active participants in beta in most cases and they have knowledge that we're interested in. So first of all, why do we care about partners? Why do we trust partners? And why do we want to test our software with partners? So different companies have different styles of business and if we do sell our product through partners, it means that partners are our best friends. We trust partners, we work with them. And according to analytics, up to 70% of IT purchases are made through partners. That's why it is important channel for us, for our companies and for us as a product managers. And we need to care about partners because they actually make money with us. It means they bring money to our companies. And what is important here, that actually we need to have two sales. The first one from our company to our partner and the second one from our partner to the end user, the end customer. It means that if we don't have the sale number one, we have nothing. So we need to take care of our partners. We need to be on the same page. At the same time, our partners have the knowledge about their end customers because they do care about them. They, in most cases, they sell not only your products, but other products. They do provide services like support and maintenance and a lot of other services. So they are actually treasured for us. They have a lot of information we as a product manager are interested in. So let's imagine we have a new exciting feature that we would like to test. We would like to have a beta testing with our partner. So from my experience, there are two main options here. Option number one, when we test the feature, the product with partners only. And this is a great option when we want to mitigate our risks, when we want to have more control over the testing process. But it is not suitable option for all cases because we have a limited target audience. And in cases, when we need a special target audience, beta testing with partners will not be the best option. That's why we have option number two when we test with our partners and their clients. It is better because we have more audience for our testing. At the same time, we have partner as a source of information, as our provider to end users' world. And we can reach our target audience. We can select the target audience more carefully to make sure that we test our product, test our feature with the right target audience. But at the same time, we have more risks. We cannot control, like end to end, the beta testing process. And in most cases, we have less feedback because we have additional steps in our beta testing process. It means that we will have less information. So let's talk about partners, how we attract, how we engage partners, how we need to be attractive for our partners. So I use multiple sources for partner engagement, feedback, sales team, interviews, events, and cold emails. So let's take a look at each source of our partners, our beta process. And I will share how I use different sources for my purposes and what are the best practices from my experience. So the first one, this is the feedback. It's completely free resource for you because everybody now has a feedback button on their site. And a partner end user could actually provide the feedback for you. So if your partner provided a feedback via any channel, it means they already invested their time in you, in your product. They do care about your product. It means they tend to invest even more. So if you follow up with this kind of partner who provided feedback, ask more information, ask what you think about this or that feature, ask to participate in beta. They probably, in most cases, say, yeah, yeah. I want to try. Let's do it. Another pretty interactive source of partners, this is the events. And it could be conference if your company organizes conferences for partners. It could be any kind of forum or Q&A session or a webinar, anything. In intermediate, we have tried a webinar for partners as a source of beta testers for us. And it worked pretty well. So it was a regular webinar for partner. And at the end, we said, OK, if you want to participate in beta for this product, this feature, please let us know. And we have a number of partners. So this is the great source of partners. And at the same time, if your partner participates in this kind of events, it means that they actually involved in your product, they are interested in selling your product. It means that you can count on them. So they will not only just say, OK, I will install the application and do nothing after that. No, no, no, they will install your application. They will sign up for beta. They will provide feedback. And if you ask to perform any kind of special scenarios, they will do that. So this is the main target to find not only just any partner, but engaged partner. Sales team. Yes, we all know our sales members. We need to keep in touch with them to know what's going on with partners, who is interested in what features, who is saying what, and the sales team, they actually great source, not only feedback, not only information regarding lost deals, but also information regarding your partners, who actually asked regarding this feature in last month, for example, who is super excited about the upcoming feature you're working on, who said that he's super excited and wanted to participate in any kind of activities that you will provide. You will know all of that from your sales team. Another source of your partners interviews and surveys. So probably everybody has surveys or interviews. I think this is what we do, actually. So no matter why you survey your partners, at the end of your survey, you can ask for participation. Like, are you interested in this? Let's try it. And actually, when you ask to participate in beta, it doesn't mean that you need a perfect product at that stage. So your partners are really partners for you. And you can create manual beta testing with manual steps. It could be not fully automated workflow for them. And it is critical to try something very quickly to understand is it working or oh no. And to try these raw ideas, raw features, raw products, perform some kind of testing, it is really great to have the feedback as early as possible. So when you run your feedback sessions, regarding feedback sessions with your partners, ask about beta participation. When you interview your partners regarding your new features, when you try and test your ideas, ask about participation in beta. And you can interview a partner for one project, one product, and ask to participate in the interview for another product. It will be OK for them. So don't just don't underestimate this source. And another one called emails. And I know that a lot of us don't like cold emails because of low response rate. And from my experience, I can say that the response rate for partners is pretty high comparing with the end users. So don't underestimate cold emails. They are not dead tried. And you will have more, even more engaged partners ready to participate in your activities. And at the same time, when you send cold emails, and for example, your partner replies that, I'm so sorry, I don't have clients with this new feature or with this target audience. It's OK. But at the same time, your partners will have the feeling that you, first of all, care about them. Second, you value their opinion. And probably next time, when you propose to participate in another feature test, and they will be happy, they will be happy to provide feedback to you. So we have a list of partners, right? We want to start. We are super excited about our new feature. So what about the plan? I can imagine that in your mind, you have the plan. Probably you have the plan in writing. But what's more important is to provide the plan to your partners. Because they are not only excited about your features, they are pretty busy doing business. That's why your plan will help them to plan their activities and to make sure they can participate in your beta. Then they can spend this amount of time on your beta and make sure that they actually can provide the valuable feedback. So when you provide the beta plan to your partners, make sure that you provide the start and the end date of the beta. Because they need to know how much time they need to invest in this kind of activity. So what to expect? How much time you need to invest on a daily basis, on a weekly basis, I don't know, whatever. Because when you set the expectations, it would be accepted as a positive. Because what you are doing, what you want from this process, that's why it is OK to participate, not to skip it. And it also will be great to provide some kind of guidance. What partners need to do during your beta testing process? Do they need to act like they do it on a regular basis? Do they need to actually follow special instructions or special scenarios for your testing? Do they need to pretend that they have another role? So you need to set it pretty clear, because probably your partner needs to identify the person who will participate in your beta. So another important point here, how to provide feedback? Because this is the actual target to get the feedback. And the gathering feedback mechanism should be super easy. It should be so, so easy. So because if you say this is the email address, just send me an email with your feedback, you will get less emails comparing with other automated feedback mechanisms. Because when they do use your application, they have an idea or they found a bug. They need to have an ability to send the feedback just immediately. They don't need to search for your email. And probably they don't remember. They cannot find your email. And this is how you can actually fail to get the feedback. So make sure that the feedback mechanism is super easy. And nobody can misunderstand what's expected here. So from my experience, it is also a good practice to explain how the feedback will be protested by you. Because they provide feedback to you and it's some kind of black nothing. So explain how would you process the feedback. So what are you looking for? Probably what kind of feedback you are looking for? I don't know, performance or bugs or any anything. And at the end of the day, set expectations regarding the exit interview. If you will have so. Or it could be a survey. Or you will ask to write some kind of free form email with overall impression from your feature. Something like that. The next important aspect of the beta testing with partners is calculation. You need to calculate how many partners do you need to have enough feedback and another aspect of it. So you need to have a certain amount of partners. So the feedback they will provide will be meaningful. So what do I mean by that? If you have two partners to test your feature, probably it's not enough. Because they can have pretty narrow experience. They have pretty special target audience. So you might not get this overall average feedback. So make sure you include different partners with different target audience. Or if you need with special target audience. So make sure that the feedback you will gather, it will be something for you. It will not represent some kind of corner cases or anything like that. At the same time, don't think that all partners will be 100% involved, 100% active. And everybody will provide 100-point list of feedback to you. So if you decide that, for example, you need 50 partners for your testing, add another five to have overall 50 partners feedback. Because somebody will have some emergency or they will be overloaded with their business questions. So make sure that you cover these risks. And another important point that beta testing with partners, this is the day-to-day interaction. And make sure you can handle this load. It could be your team, your product management team. You can also ask your development team, I don't know, QA engineers or developers that are interested. You can ask UX engineers to participate in this interaction. But make sure that you can actually handle the load. And several important items to remember for you. Don't ignore the feedback you're in beta. And this is the common case when you cannot handle the load. And you can do nothing during this beta but to process the feedback. So planning is everything. Always follow up. So make sure that you answer and say thank you for your feedback. Because it will help to build relationships with your partners. And at the same time, probably it will not be the last beta for you. So plan ahead. After you close your beta, ask your partners to participate in some kinds of future activities. So this is how you build your team, your trusted team of partners you can rely on and you can continue testing with. So that's all for today. Thank you for your attention. And I will be happy to answer your questions. Awesome. Thank you so much for that presentation, Lilia. That was great. One question we like to ask all of our speakers is, do you have any advice for aspiring product managers? Yeah. The best advice I can give is try it. Because at the beginning, it might be scary. You might be afraid to ruin your relationships with partners by doing something wrong. But we all human beings, we all make mistakes. So just try it. Just start doing that. And this is how you get the experience. There is no other way to get the experience. Awesome advice. I like it. One question I think is more of clarification. When you use the word partner, are you referring to stakeholders within the firm? No. No. I mean partners outside the company if you resell your product through the partner network. This is why if you resell your product through partners, this is how your partners know about your customers, about your end users. Awesome. OK, thank you. I think that wraps it up. We don't have any other questions. So thank you so much for presenting today. Before we leave, I want to give all of our viewers some more information about our upcoming courses and events so they have the resources to become a product manager. Our product management, coding, data analytics, and digital marketing courses are taught by industry experts working at companies like Google and Facebook. And in addition to that, we offer weekly online and onsite events at our 15 campuses around the US, UK, and Canada. If you're located near campus, make sure you stop by one of our weekly events every Wednesday and Thursday. Also, you can find us on social media at Product School and be sure to keep up with the latest product management content at theproductblog at productschool.com. So thank you all for enjoying. Thank you all for joining. Enjoy the rest of your day. And I hope to see you next week. Thank you, Lillia. Have a great day. Have me. Bye.