 All right, low folks. My name is Andrea Botnar, and today we'll be talking about ML product development and specifically how to build machine learning products that sell. Before we dive into the topic, a bit about myself. I'm a chief product officer at Data Orb AI, and I'm also a faculty with NYU where I teach machine learning at an undergrad level. A bit about my background. My journey in product development started back at UMass Medical School in Worcester where I was a machine learning engineer and a product lead for clinical decision support tools. We were looking at pancreatic cancer and enabling fundamentally doctors to make better prescription and infer outcomes based on the treatments they were prescribing to pancreatic cancer patients. That topic got me very interested in machine learning and specifically in building interfaces that could facilitate the communication of information within humans and machines. And I moved on to do a PhD at MIT where I focused on natural language understanding. At the time, what transpired was the fact that in order to have real intelligence, we have to really understand how humans think and how humans think over abstract concepts, which is what language is all about. So I spent a number of years building multilingual language understanding algorithms that were able to semantically understand multiple languages at the same time and then transfer a deep degree of understanding across languages, something that's referred to in the academic world as transfer learning. But while you're in academia, you get to move the needle on theoretical frameworks, but the limiting factor is data. So after my academic journey, I decided to move into the industry specifically in the startup world. And I've been with a number of startups where I was leading product and engineering before I was the founder and CEO of myself. And the recurring theme of the projects I took on was commercial AI. So different types of AI applications that were driving business outcomes through intelligence, through some form of optimization, some form of automation. Recently, I was a product lead at Google Clouds, but I built their healthcare IP2B product department. As part of that role, my team launched to the market a number of AI as a service products. And we worked very closely with a channel partners to package those products into business applications. And today I'm the chief product officer at DataWarp where we are building a customer engagement hub for the enterprise. Our customer engagement hub is powered by a number of AI engines. And our vision is to improve on the human experience, both the employee experience and the customer experience based on both symbolic AI and deep learning AI. Very cool. And today I wanted to take you on a journey where we discuss a bit more about what happens after you build the product. So there's a good number of applications and there's a good number of references out there that can coach a product manager on how to build products and how to systematically get to product market fit. But what's really interesting is that especially in the B2B setup, what happens after you get to product market fit can make or break the product as well. So that's specifically when sales, motions come into play. So today we're gonna talk about pricing models and why it's super critical to figure out how you're gonna sell a product before you even build it. We're gonna talk about how you measure performance and especially when that performance is a hybrid evaluation of both the computational system and the human performance. And I'm gonna spend some time briefly going over, go to market strategies and most of the examples, most of the references and the framework I'm gonna share with you makes more sense in the B2B setup. So when you're building products for other enterprises, for other businesses. Cool. So diving right into it, pricing models. You know, my experience in building B2B products was very much shaped by my participation in the SAP IO accelerator here in New York City. So for those of you who are not familiar, SAP runs an accelerator for startups where they help promising entrepreneurs figure out how to position their technology in an enterprise setup. And in 2018, Demetrix, a company where I was leading product engineering participated in the SAP IO program. And one of the key takeaways for us was the fact that the sales model and the pricing model for your product is something that will make or break the product once you move into production. So with that being said, lots of kudos and lots of things to SAP for getting us there. What we've learned through our time is that there's many ways you can price a product. And if you had to go back to the textbook definition of pricing models those can be sliced across two primary dimensions, the complexity of the sales process and then how valuable each individual customer account is. So if you are to look at the upper quadrant those are pricing models that are usually packaged around the contract, right? So you have either transactional pricing models or enterprise pricing models where the sales process is more complex but the account is highly valuable. On the lower two quadrants over here you have more of a self-service pricing model and the lower right quadrant is basically a pricing model that no one should try to figure out because if you are going after a complex sales process and there's low value per account it's gonna be impossible to build a business in that area. So if you had to think about some companies out there that have built themselves around such pricing models and developed the business with the different types of pricing models for a complex sales process with high value per account we have Oracle but have Salesforce. Then Amazon Web Services, AWS, it's a very good example of a high value account with the self-serve pricing model. And then Slack is an interesting example because in a sense they target both high value accounts and low value accounts so high value accounts are enterprises, low value accounts are consumers like UNI that can just either use their premium offering or upgrade for some premium features. So that being said, what we just saw is one way you can slice and dice pricing models but there's no shortage of ways to get creative with pricing your products and there's a number of strategies out there. The most popular ones are cost-based, demand-based, competition-based. And there's a number of other pricing models that currently are overlooked by the industry and they usually fall in the category. So cost-based pricing model is when you look at your cost of goods and you add a markup on it or marginal cost. Demand-based is when you go after price skimming or price penetration. Competition-based is when you're trying to match your competition or when you're trying to give a discount compared to your competition or you're going with a premium model. But what's interesting is the last category, the other category, because in there is where we have the value-based pricing models and the value-based pricing models make a lot of sense when you're building machine learning products. And why is that? Well, machine learning products usually go after established business workflows. So you will have a business workflow that's been around for decades, if not hundreds of years. And what machine learning products enable is further efficiencies or more intelligent ways of fulfilling the same business workflow. Because the business workflow is already established, there's well-defined patterns for how you budget for that business workflow, who fulfills the business workflow and what does it mean for that business workflow to be successful. So your machine learning product has to prove itself that it makes a difference on top of an established business workflow. So when you're pricing for a machine learning product that differentiates and reinvent how a business workflow gets fulfilled, you have to price for the differential. When you look at the most typical way of pricing a product cost plus, in that model, you're basically looking at the cost of goods, you're amortizing some R&D investments, and then you add a markup on top of it. With a value-based pricing model, which is what we're seeing very often in the industry for ML products, you start to talk about perceived value. So if you are enabling operational efficiencies in a call center, that means that you can price for some of those operational efficiencies that you bring to the table. And that's a well-beaten pass in the sense that perceived value is something that you can even quantify for your customers. And you can have data to back up the value that you bring to the table and to justify why you're pricing your product a certain way. But there's another way to think about value-based pricing and this optimized value-based pricing model enables you to go even further up the value chain. So when you build a product that improves on a business workflow, you can improve on operations fundamentally, you can improve on revenue, but you can also improve the brand perception. And brand perception is an intangible in a sense, but it is a key differentiator for businesses. So if you can define and articulate how your machine learning product impacts the brand image, then you're in a better position to have a strong pricing model and to drive that product adoption with a pricing model that you computed behind the scenes. So just to give you an example of a product that can be priced across these different dimensions. So let's say you build a grammar assistant, okay? So this product basically checks for typos, spelling mistakes and also helps you adopt the systematic communication style, all right? If you were to price this product with a cost plus framework, you would basically look at your cost to serve. So how much does it cost for you to process a hundred characters from a given customer? You would also look at your cost of R&D and try to amortize that and you would add a markup, right? So that's like a very simplistic approach to cost plus pricing. But the reality is that when you help your customer produce high quality content to produce clean communication, there's additional value bring to the table. It's not just a matter of grammar checking. Now we start to talk about some efficiencies, for example, time efficiencies. So if you're selling this product to an executive that normally spends 10 cycles going back and forth with their assistant, just refining internal communications for their department with your product, you probably can reduce the number of back and forth cycles by half to five cycles, right? So that's important time savings. And you can capture that kind of value, right? You can look at how much time was previously wasted, what was the quality of the output compared to the outcome you can drive with your product. So that's value-based pricing. You basically look at the time efficiencies or you look at the speed of execution. Now there's another way to package value into your pricing model. And if we look at the optimized value approach, we basically take a step further into assessing the value bring to the table. And if you look at the big picture when someone communicates with a clean style systematically, it starts to reflect on their brand, it starts to reflect on their image. So that's very important for individuals, it's very important for brands. And one might go to say that you can turn someone from being a sloppy writer, from being that sloppy colleague into a hot pepper because right now the way they present themselves externally is very clean, it's very polished and that's the value bring to the table and you can monetize for that specific value. With that being said, but I shared with you as a theoretical way to price your model. And of course you have to first discuss the pricing approach internally to figure out the justification for why you would bring to the market a certain price point. But the reality is that each price point you will introduce to the market will face some dilution. So let's assume you go to the market with an optimized value-based pricing, okay? That's the pricing that makes more sense. Basically you have data to back it up, you can articulate why it makes sense for the customer to pay the specific price points. What happens is that the customer will have a perceived value. So let's say you come to the market with $1 per processed email, the customer will say, well, actually, I think it's worth 50 cents. Like I don't think it's worth me paying $1 for you to spell check and polish my language in an email. So you're gonna have to bring down the price and you're gonna have to feel the dynamics in the market. But then there's another reality that you get faced with. So as product managers, we also have to understand who are the buyers. And there's usually a difference between the customer that you're building for and the person that's gonna pay for that technology. And when you land on the buyer, there's usually multiple buyers, especially in complex sales cycles and large enterprises. And those buyers are faced with a certain budget and they're faced with more complex purchase decisions that will move down the price because the willingness to pay for your product will be influenced by other factors outside of how fantastic your product is or how much needed it is in a given enterprise. So those are some realities that as a product manager, you have to navigate and fundamentally, especially in the beginning of a product launch, pricing will have to stay fluid and you will have to adapt to the market. And just like a good rule of thumb is that you can always bring down the price. It's basically a bad street cred to come up with a low price and bring it up just because you later figured out that, hey, there's so much value on living on the table. Cool. So when you price the differential, we talked about charging for the perceived value and we talked about charging for the brand image, but those are subjective concepts. And as a product manager, you have to be data driven. It does help to be data driven. And when you want to price the differential, you have to understand what are the economic value drivers that your product impacts for your customer. And just like a clean way to separate the value drivers is to look at growth drivers, efficiency drivers and financial drivers. And I'm decoupling growth drivers from financial drivers because sometimes a customer might be more interested than just getting market penetration regardless of the financial drivers that market penetration comes along with. Because if you are able to capture the market, you can find creative ways to monetize that market later on, okay? So those are some example, economic value drivers. And in reality, these economic value drivers end up moving the usage, they motivate usage metrics. So if a customer is incentivized to move in the market really fast and your product enables them to move in the market super fast, they will use your product more frequently. Those usage metrics basically inform the pricing metrics, you will price per usage or you will use usage information to adjust the pricing model. And similarly, the economic value drivers, they determine the value metrics. And those value metrics are what your customers will use to evaluate the usefulness of your product. And the value metrics are also what your sales force will leverage in sales conversations in order to make the case and close the contract. And pricing metrics usually track value metrics. To give you an example, if your product brings only a 5% efficiency improvement within an organization, versus if your product brings 80% efficiency improvement within an organization, you will basically look at the different pricing model. And if your value metrics go down, the pricing metrics will also go down and vice versa. So the two are correlated, right? So with that being said, let's assume you figured out, you figured out how you're going to price the product, you figured out who's buying the product and you have a version of the product, I could say the better release that goes out in the market. Your job is just beginning and it's getting very interesting because the moment you push the product into the market is the moment you can start to measure and you can start to validate some of the assumptions you made about the market, about the customer and the buying patterns. We talked a lot about the value metrics, fundamentally the value metrics are the outcomes your users are looking for. To give you an example, if you think about your user as a marketing leader, that marketing leader wants to communicate value to target segments. And the reason this marketing leader is interested to communicate value to target segments is because they want to generate the map. Similarly, a sales leader is interested to communicate value to prospects because they want to close sales. So those outcomes are fundamentally the things that you need to track as your product is enabling, as your product is argument to work done by your end user. And when you go after quantifying these outcomes, there's an art and there's a science to figuring out what are the value metrics and to tracking those value metrics. And I'm saying there's an art and there's a science because some value metrics are objective. So for example, closing sales is very objective. It's either the sales came in or it didn't. Generating demand can be latent because sometimes, especially when you do PR, for example, you're not gonna see immediately the impact of your PR campaign, but you might notice that your phone is ringing more often. So that's an indirect way of figuring out what's going on in the market. And also when you start to track outcomes and when you start to prove out that your machine learning product is moving the needle for business outcomes, you will have to understand which business outcomes and which value metrics actually matter to your customer. I won't go into too much detail on this topic, but there's a very good book I would recommend and I wanna leave you with this. It's called How to Measure Anything by Douglas Hubbard. And the whole premise behind this book is that you can find ways to measure even intangibles in business. And there are systematic approaches to building and scaling products without making too many assumptions and by always listening and taking feedback back from your customers, incorporating that into what you're building for the market. Cool. So now you have the product is developed. You know how your sales force is gonna position the product in the market. You have a way to track the performance of the product. The other thing I wanted to go over is how you're gonna scale the growth of your own product in the market. And one interesting takeaway from my experience as a product manager is that partners are critical and fundamental to scaling efficiently in a market, especially in markets where business workflows have been established for a long time, in markets where you just want to bring a novel approach to efficiency or operations. It helps to look at folks who've been in that industry for a while and find ways to partner instead of always going with a competition approach. So when you think about your go-to-market strategy is fundamentally sales and there's multiple ways to sell. You can sell two, that's basically direct sales where you would build an internal sales force and that sales force is gonna manage one or multiple accounts. And that's a limiting factor in a sense, right? Because you're gonna have to hire one sales person and it's like roughly a one-to-one ratio, right? Each sales person is gonna handle one or maybe two, four accounts. But there's a limit to how many accounts a sales person can cover. So to scale your sales in the market, especially since you're building a machine learning product that makes sense within a bigger product ecosystem, you can start to look at distribution partnerships. So over here, you can either choose to integrate your machine learning product into an existing umbrella of products that you can whitelist it with an existing partner or you can choose to offer the product as a value add feature or as a value add functionality for distribution partnerships. And these distribution partnerships basically become an external sales arm that can put your sales force on steroids. And this approach is very useful because it enables you to get more feedback from the market in a shortened period of time. Now, the third approach to your go-to-market strategy and one that is actually very productive when you're innovating on the existing business work close with ML, this approach is what I refer to as strategic partnerships. And over here, you can look to co-innovate, to co-develop with market payers that complements some of your business activities. So if you have to think about machine learning products, your specialty will always be the machine learning technology. And given how fast and how quickly this kind of technology is evolving, there's never gonna be a shortage of areas of development to keep it busy. So let's say you want to deploy your machine learning technology to life sciences research and the end outcome that you're looking for is new drugs that get released on the market. If that's the approach you want to take, then it makes a lot of sense to basically seek a strategic partner that has that subject matter expertise that has the workforce that understands the life cycle of drug development. And then you would enable the workforce with the technology you're building with the machine learning algorithms specialized for drug research and drug innovation. And then the output of that strategic partnership would be an artifact, would be a drug artifact that gets distributed on the market. And there's ways for you to basically structure the strategic partnership. So there's a win-win for both parties and we both get to benefit from the financial outcomes of the drug being released on the market. So strategic partnerships are all about amplifying each other's strengths and being more strategic about capturing new markets or just expanding your portfolio of business applications. Cool. Now, that's all I wanted to share with you today. That being said, I do think machine learning development is a very interesting area, especially when you get to package the development of this machine learning into products that solve business applications where we're at a pivotal time where a lot of deep learning algorithms are starting to enable more quick development of machine learning products. And to these like deep learning algorithms are also enabling some transformation of traditional business workflows. So I would love to keep in touch with you folks as you're trying to brainstorm creative ways to go about product development within organization or if you're stumbling into some technical limitations and you're like, hey, we're almost there but the deployment or the implementation is kind of breaking how can we come up with a more robust and rigorous way to match the product requirements for the technical requirements. And given that you're listening to this webinar, I think that that's a strong sign that you're either working on machine learning products or you're interested in working in the space. And I welcome you. I hope more full folks will join this business area. Thank you.