 Thank you all. Good evening. It's very exciting to be here today. My name, as it says in the slide, is Satyag Pai. I'm a product manager at Google. I consider myself fortunate that I've had a chance to work in product management for over ten years in two large, pretty exciting tech companies. And I'm here today to share with you what I know and what I can share from my own experiences around how to be a versatile product manager. I'll go a little bit more into my background and the specific type of products I've worked on in a few slides. But I thought I'd start off by first just, you know, making sure we are all on the same page as to what does it mean to be a versatile PM. As I was starting with a couple of you before the talk started, the term versatility can mean quite a few different things. So let's get pressed on which aspects I want to cover over the next hour or so. And to do that, I realized it's up for 6.30 on a Thursday evening. First of all, thank you for coming out. The weekend is coming up. So I thought I'd get us warmed up by making a couple of quick movie references as we start approaching the subject. So the first one is this movie. It's called American Sniper. It came out a few years back. The interesting thing is Bradley Cooper plays the role of a U.S. Navy SEAL. And the movie goes into quite a few different things, how he ends up getting selected as a Navy SEAL. When he goes on tours of duty, he's essentially parachuted into a variety of situations and ends up being very successful, becomes a war hero. The plot explores a few other things as to what happens when he comes back to the mainland and between those tours of duty. But the part that I wanted to focus on was that aspect of a Navy SEAL where you're so well trained that when you go and are dropped into a situation, regardless of what the environment is, what is the task at hand, you can be effective from the get go. And that's the aspect of versatility which I want to touch upon today, is as you guys express your interest in product management and want to go and be successful product managers, regardless of what type of product it is that you're working on, can you go ahead and hit the ground running, quickly get up to speed, ask who's the users, what is the core value prop of the product, and start working with your private customer base and your audience to go ahead and be very effective in that role. So I'm not promising that by the end of the hour, you'll all be able to start in the next clean Eastwood movie, but I'm hoping that I can share enough that you leave with a good understanding of what are the different types of things that go into different types of products and be effective in your own ways. Alright, so let's jump in. Why should we care about being worse than that? In fact, there was an interesting question from one of you in the audience which was, as you look at the roles, do you think about it being more as a generalist PM or a specialist PM? And what are some pros and cons and what choices do people make? I believe to a large extent it ends up being a very personal choice. There are PMs who end up liking to specialize in a specific area and are good at it, enjoy going deep and continue to do that for a number of years. But even if that's what you choose to do, I believe it's very important to learn to be worse than that and to be able to adapt for a few different reasons. What is if you take a look at which type of organizations typically hire product managers? You can think about either the very large companies and large organizations often like to hire generous product managers, which is folks who can be working in a specific product area for a number of years and at a later point of time in their careers, if they were interested, go to a completely different product area and still be successful and effective in that other space. So that's one aspect. The second reason is if you look at startups, now personally I've only worked in large tech companies, so I lean on somebody who's actually very well-versed in the space of startups. I don't know if many of you have heard of Steve Black. He was recognized by Forbes magazine, I'm seeing a few at Sinole, has been one of the 30 most influential people in tech, I think in the year 2030, and he's actually done quite a few startups himself. I believe at this point he's a consulting associate professor at Stanford and talks about entrepreneurship. So what he says is, in his opinion, startups are organizations that are in search of a scalable and long-standing, repeatable business model. And when you reflect on that, it actually does make sense that then it's not unusual to find that a startup goes there and starts approaching a specific product direction, starts building more of that, and then realizes that they need to pivot to a completely different type of product in order to be effective at what they are trying to do. So you hear of cases where somebody or a particular company starts off as a B2C company and then realizes a few months into their discovery process will actually be more effective being a B2B product and maybe you shift at that point to become an enterprise product. So if you think about these different types of organizations and the fact that even once you start working as a PM, these shifts can come sometimes outside of your control. It's therefore important to be able to quickly adapt to the situation. The other thing is your product strategy often evolves. So a lot of the time you think you're going to go ahead and have a hypothesis, this is the need, this is the type of product I'm going to build, this is my minimum set, I go release it and watch for feedback. And then you realize that inevitably, even like I make mistakes all the time, you realize, wait, these things were not right or we thought it would work this way, but it turns out that this particular demographic has a different purpose. And then you are having to shift after your product is on its way, after you already are a PM in that role. The last bit is technology evolves. We're here today on an October evening in 2017. Let's go back 10 years. The year 2007 was the first time that the iPhone was announced. Now imagine a world a couple of years before that and imagine the world five years later and today and just think about the number of product managers who needed to work on or wanted to work on mobile or rich smartphone apps or tablet apps. And this was a very large technological shift there were many other companies and things in play, but this happens all the time. Some of you mentioned you work in the enterprise, so if you look at what's happening in the enterprise space, I believe it was 2013 when Docker containers were first announced. Kubernetes, a really popular container orchestration service, came out only in the year 2015. In a span of five years, large parts of how you would go about building excuse me, your own enterprise product from scratch today have changed drastically. So these are just a few examples of where technology changes really quickly and therefore believe it's important to be able to adapt and be versatile. And the last reason which I didn't put on the slide is actually you all involved. So let's do something interesting for a second. Imagine it's your most amazing day, you are the product manager for whichever product for whichever company you most admire and let's say you're walking to your first year for super excited. Now raise your hands every one for a minute and then start dropping them as it no longer applies. Let's go ahead and do it. Then you enjoy working on that product six months from now. How about two years from now? Five years from now would you want to work on the same product? Hands are starting to come down. The point is your own interests, your own preferences change over time and therefore being versatile adds another dimension. So let's move forward. Why do I believe I can help with this conversation today? Some of you are mentioning you are interested in my sort of own career, my own journey, being a BM in the industry. So I head off as a computer science grad. Really enjoyed working on hard technical problems. In fact, as I was chatting with a few, as I entered the program I honestly didn't even know there was a role called as product management. Little that it would interest me so quickly. I ended up starting off as a BM intern at Microsoft. That turned out to be a pretty interesting experience, love what I was doing. The team was happy to have me and I went back and joined full-time working in the Microsoft Identity Innovation. I don't know if several of you who are in the enterprise space may have heard of a product called Active Directory. It's very widely deployed on premises in a lot of large companies. And that was the product team that I worked on for a number of years. So part of that there was some very interesting situations I found myself in. So one is, you guys by the way, I'm just sensing already are very exciting and lovely audience. Thank you. There are times when I found myself in a room full of CISOs, CIOs trying to defend our latest security offering while they are as they should, scrutinizing it very deeply. A presentation like this might not be met back with so many science smiles that I'm seeing right now but rather, wait, does this really work or how can you prove it or has somebody actually audited and can prove that this thing you're telling me is correct and a variety of situations like that. Pretty good at learning experience. I did that for a few years and then I shifted years to working on consumer apps. I ended up starting off with a desktop or a tablet type app. Then we took it to the mobile form factor and we took it to different operating systems and the idea was that we then also ended up connecting the entire experience that you had when you ended up at our site either through a web browser or through a mobile app or tablet app or any of the properties and you got a nice cohesive experience. So that was a very different set of challenges. You're trying to make UX which is very rich and compelling. You're trying to get users to, of course, continue to be able to do whatever they wanted to do while still trying to engage them and retain their interest and doing this when there's a variety of other things that they're interested in like in the whole ecosystem. So it was a very unique experience and then for my third challenge I joined Google and I worked in the Google Ads Organization and as a part of this role I worked on a CRM and sales intelligence product and this was a very, very different experience too because now the set of folks who I was chatting with who I was interested in having taken active interest in my features were very senior business executives and it's important to be able to demonstrate first identify and then demonstrate the value of the products and features to them, to that audience and I'll talk more about some strategies and things to try if you find yourself working in this space. So as a result of these three different products that I worked on I ended up working for over a decade and as our products could reach out I wanted to have this talk reflected on a few things that I learned things that I applied which helped me be effective some of the mistakes that I made and what did I learn from those and that's what I'm here to share with you. All right. Let's continue. So before we jump in and start saying here's what's different let's take a look at what's common what in my opinion does not change as we look at building our products no matter what type of product it is. So I'll go back to the second movie reference I said I'll work myself for a few minutes and then we'll start digging in deeper I wanted to show you this particular movie which is called Dangle. The actor that you're seeing on screen is Amer Khan he is a very famous Bollywood actor and the interesting thing about this movie is it actually shows the life of an athlete when he was an athlete and then many years later when he was a parent and had two teenage daughters who he was trying to motivate to go into the same sport that he was very effective at and part of that's of interest is as a result of that during the course of the movie the actor ended up playing both a young fighting athlete and the dad who after some years had put on a few pounds and looked quite different but the interesting thing is he went ahead and decided to not use any of the batting or like questioning and he actually chose to put on about 55 pounds to do the other role to lose them all the way back I'm not saying to be a great product you know we need to go and be like the weight loss masters biggest user or what not but the very interesting thing is he went through this whole experience in fact jokes in an interview that I made sure I did this particular role first because then it wouldn't just be me but the entire production unit which made sure I lost weight again and went back to my original form that's the first example there's one more let's take a look at this actor how many of you have seen House of Arts yes I've seen many hands who have I have a little bit that is when I first came across his work as Remy Dalton on the House of Arts and his name is Marishala Ali what I did not know until I was reading another interview by him is he was actually doing three different roles at the same time working very hard seven days a week and the challenge as an actor is how do you go ahead and transform excuse me go across these different roles without getting them mixed up in your head and so he had a very interesting thing that he did he says he took the time to go ahead and build out three playlists and before he went out to the set or when he was in his trailer he would actually listen to that particular playlist to get his mind into the form of the role that he was going to play and he'd then go out that was another very interesting day give me a moment the third story I shared is actually my own this goes back many years I was in high school we all like to try different things fortunately I did not try too many other crazy things but I did decide to dabble in acting a little bit so I go to our high schools annual night and I was about I think 15 or so years old played the role of a 60 year old man and we had the final dress rehearsal before the actual performance so I memorized all the lines, gone ahead and made sure I set them on stage without fumbling with too many of them and the curtains were down we were to come back the next day but I thought I was walking back backstage true story somehow in my gut I felt like I know that that wasn't as powerful as I think it could have been so I decided to walk up to a dramatic teacher she was a wonderful lady encouraged us to buy our various kinds of things was very active in environment and what not and so she tells me one thing she says you know what, you're done go home and get some rest but I'll share with you that when you come back tomorrow in your mind come back thinking that you are that 60 year old man so I went back, slept gave him the next day literally imagining, like intentionally jokingly with my friends, you know stumbling a bit with my key in and what not while I was backstage before I went on I was actually in my mind pretending to be an old man it was an interesting experience I went in, was at least for me a pretty powerful performance the school recognized me with some award and certificates and what not but the point was I was very happy for having tried out her advice and by this time I'm thinking all of these stories are great I'm not here to talk about acting so much but the thing is it's not about the acting it's about the character in fact it is about putting yourself in the shoes of your character and as you guys think about our products the character is your end user let me share why I pick up on this many of you have probably heard about the stern user empathy several times in different context but I'll share that this is the one that I chose to go into today for a couple of different reasons one is as several of you mentioned you are currently not in product management looking to transition in I'm glad I've had the opportunity to speak with a few other folks who are in that situation and work through them and try to help them in my little way and one of the things that I found was pretty interesting a lot of folks who are not in product management are obviously very effective and whatever else they're doing right now very smart, very talented take the time to read and invest and learn about the role, learn about what it takes but here's what I'd find so after a few minutes into a conversation like one which you might have one-on-one if I was chatting with you we'd get to a point where we'd say let's take a look and see how do you productize this and at that point I'd find that as they were approaching the question or they were structuring their answers they had the frameworks in a template stumped back like they knew this is the framework this is how you apply it read it and somebody choked about a coding interview or a PM interview book start talking to them about metrics oh these are the AARP metrics these are the pirate metrics I've seen this presentation I asked them one of them all right go ahead and describe for me what would be the MVP that you would propose MVP as many of you probably know it's minimum viable product looks at me and goes I'll tell you that LMLP this is the most loveable product MVP v1 is going to be amazing but then as they started sharing it came across as a distinct incoherent set of features that were coming together and not actually solving the user's problem and if you think about it take a step back these folks who are having conversations with are smart, they're intelligent they get how to apply the frameworks they probably knew more metrics and I'd go to the top of my head to be honest but I believe they could have been that much more effective if they started thinking from the lens of what is the user, what are they trying to do what problem am I solving and then they applied the metrics the frameworks and templates to their thoughts and so if you take one thing away and I do believe honestly it is common does not change based on which type of product you're working on is to think about the user think about what matters to them and then forward it might seem like I have this on the slide and I've been talking about interviews but even today when there's times when I'm trying to prioritize between or wait that's the one that's 80% done but I run into this issue how do I prioritize? well that has a dependency you take a couple extra quarters should I wait, should I go to market with a different future going ahead and leading in with this principle really helps address a lot of that so start with this and see if that helps you that's been one thing that definitely has helped me question you talked about your acting career and putting yourself in the you know the 60 year old's mind you talked about also really putting yourself giving empathy to the user you know I've always found that actually spending that time with the user putting your feet in their shoes sitting with them side by side and understanding their problems from a real time basis is often the best way to get done yeah great great comment then like that approaches this broader subject of how do you go ahead and put yourself in the shoes of your users one thing I will say just a second I'll get to you is by the way my acting career thank you for being so kind it was all of two performances long one probably in the 10th grade the other one in 12th and then I took a very different engineering oriented path but that's it like I think your meta point is valid is what are some ways that you can do it there's a variety of different strategies one that you started with is very effective for certain types of products actually in general no matter which product you probably can't go wrong going and observing your users for a certain amount of time and getting a good sense especially if the target user is not one that you're typically well versed with like if you were to design a product for somebody such as yourselves you'd have naturally a very nice and intuitive understanding of what they probably expect but if you quickly shifted the conversation to you're saving to design a product for some folks who are living in Africa or about 12 years old and will never ever actually work on the internet and say some other things that are unique to them live in a very hilly terrain like just out of few dimensions and then it might be much harder intuitively to know what they want going ahead and actually observing them is definitely a good strategy beyond that there's a few other things you can do based on the type of product and how much you need to scale what wide of a user demographic are you targeting there's things like partnering with if you're in a larger organization your UX research team can be a fantastic ally even with simple things like if you have a product that has a product forum go ahead and actually take the time to read several forum replies yourself you probably have a support partner who's going to go ahead and distill and tell you here were the six like biggest issues these are the ones that are coming up the most these are the three that the biggest customers are blocked on but after investing the time and going ahead and reading several of those yourselves will give you a very different perspective you had a question I guess it's a bit easier to kind of trying to go into the mindset of user when you are in the kind of consumer product right when you're making a product for corporate like active directory who's the user? CEO? CDO? the poor guy with computer who's clicking the buttons? it's actually a very interesting point if you wouldn't mind me tabling that since I have a slide to cover exactly that in a few moments but we'll come back to that and the comment for the whole room was it does get tricky especially when you talk about non-consumer products because there may be more than one road that's involved so I'll share some couple of examples when that actually happens in a couple of slides okay so let's keep moving forward and talk about from here's a part that was common to the very different challenges that come in if you start focusing at or thinking about different types of products to go ahead and do that just as start a software grounding us on your terminology this isn't any formal classification that you might find in a textbook it's one that's more for convenience and for structuring that's part of the conversation today there's three buckets that I would go ahead and split things in two one is the enterprise product this may for instance typically be a lot of times the infrastructure or deployment oriented consumer product which several of us or most of us are innately familiar with because they're probably using many consumer products every day typically or most often I should say dating the form of apps or websites or things like that and the third one is business products which may be certain things that actually exist and advance the function of the business so Salesforce CRM is a great example certain types of finance software like say accounting is another example of the type of business product a lot of the time you'll find that folks typically tend to bundle the first and third categories and you certainly can there are several similarities but I just broke them out as separate because I think there's a few special strategies you can try if you're in this bucket versus the first one I'll share one of those okay so let's jump in and let's start talking about enterprise products now enterprise products in my opinion and none of this is an exact science and there's obviously some simplification so if there's parts that don't resonate with you and we'll have time for Q&A at the end let's definitely come back and explore but essentially it exists to solve a problem that is key to the customer's heart and the problem could take various forms the industries could be very different depending on which customerization are you talking about but at the end of the day that is often the core of where an enterprise product originates from and they can start getting more complex they can start evolving in various ways so what are some key characteristics and I don't obviously try and cover everything I just tried to do things which I tell were relevant as you were thinking about it as a product manager and one of the key things that comes in an enterprise product is actually getting it in the hands of the users at that particular customer and there's two large obviously simplification but ways that you can think about it one is on-premises or when the customer self-deploys and self-manages their product if you go back many years and even today in a lot of organizations managing things in their own data center which they deploy their IT or their development team who supply and actually manage would be sort of the first one the second is this more recent trend and it's recent as in it's been a few years but it has been accelerating is the software as a service model or a cloud deployment model in which case the key difference is if you are the team or the company that's building the product you actually manage the software the hardware that it runs on and you make it available to your customers as a service and then they could go ahead and connect to it you might still have some storage space for them to be able to store store the right types of things various other functions around it but the key thing is you yourselves are responsible for managing and the reason that some of you might be going a little bit too quickly towards department management department model in a product-related conversation the reason that I think it's important to recognize this is this actually ends up impacting to a large extent how quickly are you able to change things how easily or quickly can your customers actually get at them and things like that and I'll share a couple of suggestions in the next slide around this next with us that service is the latest uptime guarantees are critical they are of course very important consumer products too but let's take in fact mail or Gmail or email for consumers versus enterprises if Gmail for consumers was down for a certain amount of time a lot of us would be upset but if you actually are upsetting into an organization typically it comes with a specific SLA and the org is actually relying on you to make sure that the email services up for a long amount of time so that they can bet their business bet running their business on your product connection and a level of importance as you think about enterprise products the third is the customers control over an upgrade cycle and this might be this actually I can probably best explain with a story several years back I was talking about an enterprise product in a room full of IT architects and execs and I was talking about like here's all the amazing things that we have in the latest version it's if you go ahead and just deploy it X, Y, Z here's the incremental value of the lab and what it came about to us they were like well this is great we are still on the product that's three versions behind which you released seven years ago and the reason that's an important thing to process and this goes back to your point about actually talking to customers and users of course is there may be many different constraints which come to customers upgrading to enterprise products which may have things to do with reluctance or encouragement of their own internal departments budgets especially the new product there are different pricing points supportability because oftentimes enterprise product especially when they are in this category may be managed for a certain number of years and then they tend to run out of support market a smaller level of support from the vendor and things of that nature but that is a factor of even if you get something out today if customers are the ones who are typically setting the cadence of when and how they deploy it might be a certain amount of time before those features are actually the key metrics that you typically tend to think about or focus on in enterprise products now I'll share these for each of the categories obviously many of them do overlap what I've done is distill which few that I feel are more important or ones that are worth paying a closer attention to for each type of product so in the enterprise space the thing that starts mattering is how many deployments you have and what type of deployments are those now there's some products which have just one specific customers that deploy them but these are very large deployments so the companies focus shifts or the product teams focus shifts to making sure that those large deployments are successful then there's many others but you might find that it's actually deployed by a wider set of customers but they are like very concentrated in certain geographic regions and what not so having a sense of where is the deployments happening which type of customers are actually applying them and watching them and importantly the next thing is number of users per customer or the amount of revenue per customer it is also an interesting move to track to sort of contrast between the third and the fourth one on the one aspect you'll probably be charging for your product in a lot of cases and different types of things one is you might have a fixed access fee for instance X dollars per user per month you might have a flat fee in some cases you have licenses or contracts for instance models but the key thing is the third bullet is captioning what is it that you are deriving out of it of your product the latter bullet is also important to keep in mind is what does it mean for the customer to be actually deploying your product what is it that is costing them and you ideally obviously want to be increasing your revenue while ensuring it's easy and cost effective for your customers to continue to use theirs and there's the specifics but if you think about now this setting of enterprise products and think about specific strategies you can share here's a few that come to mind one is and this is where we can tackle your point on which like who's the user right so there is a difference typically in enterprise products between who's the decision maker of whether or not your products actually going to be deployed and who's the actual end user and end users also may take different roles and keep in mind because yes you do need to make sure that the decision maker which could be an idea which could be a business vertical owner managing a certain function in the company and the end users may be everyone who's either in that part of the organization or even beyond that specific organization in that customer side second one is well rounded launchers matter and this is a key point to make in enterprise products if you just go ahead and get a lot of features which are not that visible to the customers then it's possible that either it does not end up getting deployed as quickly as you were hoping for or not to the extent that you had in mind and the best way I can explain this is if you think about let's try and make it a little bit more complete say I developed a new type of database product which had several efficiencies which handles new things which other database products couldn't and if you think about it again going back to your customer how do you think they would approach it or how would they even hear about it one is they might see some log posts on your official your company's official log they already might describe it here's all the features they probably would go to certain conferences occasionally they probably keep track of industry news via newsletters that they subscribe to and a lot of the times they're networking with peers who are in similar roles and other organizations and then they're cool oh it can do these things that's fine then they start getting a sense of okay it is actually starting to get deployed the ones who have deployed it have this to say and that's where case studies come might be and the best case studies are not terms that are very marketing oriented like of course everyone wants to put their best work forward but are actually explaining the real thing of that going like for instance hey this database is amazing it will go to deployment it increase our database size by 30% and if you have databases that are over 500 MB in size then this and this thing happen so when the actual customer who is about to deploy a database product starts to get a sense of these types of inputs from others who are actually trying it out that's when it starts becoming a more real consideration another thing is if you could now take that database and try it out either directly on your company's website or at conferences and industry events that's another place where they can go ahead and get their feedback and if you think about it if you were the person whose job was at the customer side to decide whether or not you migrate your company's database to this new one would you go ahead and bet your career on this decision without doing your fundamental research would you want to be the only one of the 500 database admins in the area there are things like this which are not technically the bits that you're shipping or the service that you're making available but are all the things that go around it end up having a large influence on how your product is actually received by customers and is a key part of their consideration decision the next thing that often happens is you actually talk about even proof of concepts that happen at customer sites so this is one where you don't just want to go ahead and make sure that your product which does require close working closely and directly with the customers before it's deployed having a successful POC happen especially with your larger customers there's often a key consideration them coming in hearing about it from sources outside of just your company's official sources by good independent veterans in the industry that they hear about and trust is also a key consideration so team all these things around the launch are actually quite helpful but it's about ongoing usage and not just making the sales decision so for instance let's say that you sold your database product in a model where they have now gone ahead and licensed it for three years great you made the sale today the terms are signed and the money's going to come in at whatever latency of nature now think about it how does the deployment decision or the reconsideration decision happen probably like when we think about how are we going to spend our next year's budget is it worth or does it make sense to still go ahead and actually license or use this particular database product or shift to a different model and that's where actual usage matters so in business models where you're sort of locked in for a certain amount of time or I shouldn't say locked in more like there's a periodic expected amount that's coming in still making sure that it's active in use remaining in close touch with your customers to help address incremental features they might need and just helping them delivering that in the product not every level feature or little enhancements needs to or should be independently monetized going ahead make sure it's successful on those levels is a key aspect how do you decide whether you should push for an upgrade before the agreement comes in or should you even try that how do you decide that is you know person making that choice so when you say upgrade you mean as the customer as the company as the company offering up to the customer that's a good question so once the product has shipped especially ones that are typically deployed for a certain number of years what are some ways you can think about when are you going to go ahead and make incremental things available to the customers and how strongly are you going to push for their adoption before whatever the national I think it depends on what type of incremental upgrade is these days unfortunately there's a lot of like security related issues that are coming to the fore in a variety of products and industries and so one strategy which a lot of companies do is something that actually will help increase the security of the product or plug which is now being disclosed it is something that's typically made available at the next cycle often monthly batches or if it was a service then you go ahead and just upgrade everyone without sort of waiting for you know more form of communications and then let them know what you did do is one type of thing that happens the second is in some cases what you might do is if solving a specific type of use case so let's say in my database I had an issue there for certain types of integers I had a performance of let's say 40% now I have a fix for it I've gone and worked with my engineering team we have a way to patch it to address it do you go ahead and actually take it and send it out to all your users or do you go ahead and only make it available to the subset that are actually impacted by and I think the decision of that would come down to if you had a way to understand how is your product being used by each of the customers you might be able to say you know only 10% of our customers have integers that are this large which are impacted by this issue so let's let them know that there's an option and they can use it but we'd wait or hold off most of them on the other hand if it actually impacts a substantially larger one of your customer base if it was a ship in the box type of product then you might make it available to them to like your entire audience and sooner another thing is the more predominant model in enterprise products especially if you go ahead and build something new in a lot of cases that's actually making it available via SaaS model or like via cloud departments where effectively either you manage or you're able to influence the deployments that your customers are using and that's a key consideration because in that case you could judiciously choose to go ahead and deploy and make the upgrades without a lot of machinery to be involved even if you manually do a lot of steps and that's a big advantage the one principle I leave you with is obviously you ought to be cognizant of what are the implications of actually making the upgrade in that case for instance if going ahead and fixing this issue impacted a key aspect of your product in a database product it might be latency it could be the size of the database and things like that then you would want to make sure you have the right type of safeguards or guidance in place before you aggressively go ahead and try you would try to fix this issue which is probably just sitting performance by a certain point of time but you ended up doing it in a manner where the whole deployment had some other benefits things like that and the last bit to focus on here is ensuring interoperability with relevant products and the reason that this is important is enterprise products especially the ones that are going out today or for the last several years are rarely fully contained isolated products but it's very unlikely that even this database example that I'm describing would be a database that you deploy and like it's probably useless if it doesn't actually power up another service that is using the database it is also possible that it might need to interoff with a variety of other types of products for instance some type of a logging product that they might have deployed so that their security teams could get all the key audits and logs in one place some type of access control product they might have deployed so that if they wanted to say anytime somebody asks for authentication it shouldn't be that your database product doesn't actually confirm or doesn't have a way to work with those types of protocols things like that so the key thing is a lot of the times you may not actually have to go ahead and invest in making code changes to support these scenarios some of these even these two examples that I took might happen at a layer that is lower like not visible to the layer at which your product is written or coded to but it's important to be aware that they can happen and try and test now sometimes standards compliance yes very good point standards and compliance especially in areas where the industry is a little bit more involved you'll find that there is certain protocols as you are noting which have been standardized whether or not they are officially standardized by a body like W3C if practically 80% of all the products that are coming out are using it and yours doesn't then that can also end up coming in the way of making your products effective so there's more considerations like this that we can chat about at this point if you dig deeper we'll probably start getting into some more engineering related conversation so I just want to bring it back a little bit to the product side yeah that's what I'm going to ask actually what does bring it up from the maybe really what's a good successful team looks like for a P&M to leverage because I think you've touched upon development support and so when you actually start a new product or you're managing an existing product you know what are your key people you actually depend on to be able to make new decisions right? that's a very good point it's like product management like many other disciplines but especially for P&M if you just have a P&M and don't actually have an engineering team that is building the product you're not going to have a product I mean the P&M could themselves choose to code if they want to do for certain smaller products but typically that's a key part obviously you need an engineering counterpart but it's not just that to your point like in which cases what type of teams are affected I think it's honestly a function of to a large extent the type of organization that you're building it if you're a small nimble group of people in a start up or a smaller team in a larger company it's quite likely that one person will have to wear more than one app that we can in cases especially if you have like four people might be that the P&M is also themselves taking on the roles which typically a UX researcher might in some cases a UX designer might at least to the level of wireframes and things like that in larger organizations or if you're in a company that does have a larger product and they do have these roles available the way that I think it typically happens is you think about the type of product that's being there to front end back end UX designers and various roles are available I think the key thing is you want to make sure what type of product is it what type of a visible surface area is it going to have for customers in many cases you may then decide that for instance if it's a UI oriented product or one that has a large UI surface you obviously will want to work closely with a UI or a UX designer I think a more practical way I've got with 10 different products in the company how and when do you make sure you work with them to get attention to your product versus like hey maybe there's another product that is better worthy of their direct time and one way to think about it is see how granular of an ask you wish to make on them and then you might decide that hey if you're actually crunching on resources here's the thing that we're trying to do here's why or where we believe a UX designer can't have time for a sustained period but if you're actually short on time then we work with them to go ahead and have a few reviews like maybe a review of wireframes get their feedback go ahead and make some tweaks come back and get their feedback in the next round and so on it's another one and around of your question the other roles that come into play since we're talking about enterprise products a lot of times the folks in your professional services organization or the ones who are customer engineers does make a lot of sense to partner with them proactively especially when you're in the earlier stages of trying to fine-tune your product and make sure it lands well and can meet needs any other rules with that answer or question? Yeah I actually I'm an IT director at BDI right so I I work with vendors and typically my contact has two types of people one has 33 skills connect is what I was trying to serve me support I love they I feel they're actually better than BDIs at times because they're connected my problem is quality you know I almost the PM call me I tell them go back to the support because they have it all you know and so so now if I go into a PM role in the software company right what's interesting for me is what's my sphere of influence you know whom should I be putting my hat on right to make with the same if I take long signals from the sales guy for example that he thinks he can get more money from the customer that's just a wrong way to go about it right and so what do you you know at Google or in general PM if they're the product steering committee right who are the core members that decide major mind work work right I think I see like two parts of that who do you partner with and second is when it comes to actually making product decisions what are good and effective strategies I think we got most of the first part earlier just to recap it's a variety of different disciplines based on that product and what states the product is at you can decide what makes most sense honesty it's often also a function of the other person's time but as a PM it's I believe the emphasis on you that if you felt for instance you did need the attention of a full-time UX designer for your product including the security and having the right conversations is something that you would champion the other part about what are effective ways to actually make decisions on product strategy I think let's take another example of like a sort of a larger organization which has few key products and it's important to make sure we're building the right things in so in these cases you might find that the decision committees effectively are much larger I believe it's useful to have just to round that off obviously someone from the product management side somebody from engineering the right voices from the field organization which depending on the architecture could be some or all of sales sales ops support customer engineers folks who are actually working with customers to deploy them sometimes called solutions or field engineers in some companies and some representation of these the one way that you obviously don't want to get tied into a decision by committee because you obviously have the right type of reviews so you go ahead and take your five largest things that you're going to go to or thinking of doing and get that feedback early one thing you can also do which especially since we're on enterprise products is consulting your largest customers or going ahead and actually see if they're willing to try out a couple of things that you're thinking of in their specific department and see what were the results of those learning in other cases what you can find and happens even in larger companies that happen a lot in smaller organizations with fewer PMs is you might find that for certain areas of the product the core team which is the PM and their engineering counterpart or internally depending on how well things are structured themselves go ahead and start making a lot of these decisions especially when you're in experiment building so it's not a type of product where you could go ahead and make a change could have an experiment see the AB test results of that and then make a decision assuming the change wasn't that drastic you can probably go ahead and push that deeper into your organizations and let team make the right type of decisions team can decide to come up and get feedback early for like a few other things that they're thinking of for instance to help prioritize hey here's six ideas we have which of these are probably going to be the most impactful to our customers versus here's three different enhancements we're thinking of that are somewhat more creative enhancements what I'm describing typically might happen more in a consumer product where you decide for instance even small things changing the type of logo that you have that somebody might need to flip on or what not but end up having a bigger difference you might choose to experiment more and then analyze the experiment results to decide which ones you actually make your production all right so let's move from enterprise products to the next category which is consumer products these are the ones where again just to try and have a simple look at them I believe they fall in two buckets one is that a lot of them are actually there users which may be that you effectively have let's say you're making a game simple case or you go ahead and invest in a YouTube or sorry YouTube videos or some podcasts and the idea is at the end of the day you're doing this in order to delight your users to entertain them and it's a very different type of product than one where you are actually still going out there to address a specific problem and sort of one way to think about them is at the end of the day is it more like a toothbrush or is it more like a television stand and maybe elaborate on that with an example most of us hopefully brush at least once a day probably more often or my dentist says we should more often and then we use a toothbrush and so you go ahead use the toothbrush you're done now there are certain bells and whistles you can't have like I think the last one that I got had some fancy parts for like some of the toothbrush was rotating and apparently it's giving me cleaner teeth and what not they exist to solve a simple purpose which is to keep my teeth clean but if you think about the television set it solves a very different need where I'm sitting down in front of the TV set maybe it would be to watch the last season of The House of Cards seeing more episodes than I should because it's late and I should catch him on my sleep but that does happen you go there to be entertained and delighted you're playing a game and you probably want to be a mercenary for half an hour an hour or longer and here the type of product actually helping solve a problem there's probably products that start transgressing the two boundaries and it's hard to put them in one but in the other but it's very I feel like effective at least it's like trying to do an initial assessment of which type of product is it maybe take an example and that helps explain the point so let's think about messaging to go back many years there was this need that yes we have mobile phones we could pick up the phone and call each other but a lot of the time it's not practical to actually talk to or you don't want to be calling them for every little thing and then text messages were the simple product that helped me the need then if you think about it they have certain problems one you typically could send only one to one though some phones had multi-send feature it had a certain mode of cost to send each one there were problems or complications when you tried to send it internationally in the US a lot of users may have had their SMS plans turned off and you wouldn't know before you texted them and then you started looking at certain types of other applications and one more problem was they could only do text up to a certain number of characters so then somebody came up with the idea of a MMS and yes that could send media but then not all phones could support it and there were some other challenges around carry and drop and what not fast forward a little bit more and then that's how we ended up in the world of text messaging apps most of you probably use some form of either WhatsApp WeChat or one of the other popular chat applications and now if you think about it it's the world to a point where you're able to go ahead and send messages individually in groups of folks and you're able to send rich media you're also able to send videos which a lot of the other formats that I spoke about could not do but if you take a step back and think about it they are essentially trying to solve the same type of problem is how do you make messaging easier in a variety of different contexts and applications if you then think about you might want to send an elaborate example but you can quickly play out in your mind how things like videos on the internet or how gaming applications are going and at the end of the day starting to think about which type of product you are interested in in the link is the key dimension that helps make a lot of other decisions or factors similar the second thing is in the consumer space these days there's a lot of things that are vying for our attention so there was this phase for those of you who follow the sort of apps ecosystem a few years after the iPhone announcement and other OS is emerging mobile applications was something very exciting a lot of companies a lot of products coming out everyone's saying we even hear the expression there's an app for that but the problem that happens is after a certain amount of time you start losing track of which apps will be given installed on your phone when did you last use it but then there was another issue when we start where and start finding notifications great except the notifications started getting too many and so in the end of the day for consumer products it is tricky to solve to make things a daily habit unless you're solving a very poor problem with a very clear premise and if you think about it there is still a somewhat smaller set of things that you actually want to do on a day to day basis so thinking about when and how often would a user want to actually use your product once it's in steady state it's been out deployed it's been a certain amount of days since they installed it and how is it that you have to make it valuable or have the right type of triggers or motivating factors for them to come back in and use it is another key aspect that comes into play in consumer products because otherwise yes you can go ahead and get a certain milestone and you are absolute of 10,000 downloads 100,000 but at the end of the day a lot of those installs are often forgotten or even if they're there and the user knows they're not coming into it as often so the thing that becomes interesting is thinking about the demographics do you target and maybe the product that you build even if it's a consumer product is not of interest to everybody yes there's messaging matters to almost everyone there's probably only so many messaging apps that are going to end up in interaction but there might be a different type of need you solve which applies to a specific segment of the users and that could be a daily habit for that subset of folks for instance for folks even if that's only say 15% of all consumer users it's still a pretty large base and you could choose to target that so starting to think about who you're targeting your app to or your consumer product to can be a key consideration what if that base is not large you have an amazing let's say you're very passionate about this product and as you mentioned in this case I think even in this area five mile commute is a long time usage but if you live somewhere where your demographic base really really like that product would you go for volume or do you go for what are you like what are you how do you decide probably not gonna like my answer but my answer is it depends it really depends a lot on what specific thing are you trying to do if you're and what your sort of goals are for like that product are you trying to go ahead and grow your expanded user base and really trying to get when this actually good segue to start talking about metrics in a minute but are you trying or are you trying to very deeply meet the need of a certain set of users is a key consideration other aspects that might matter is it just a number of users is it also the amount of time that they're spending in the app especially if you're an entertainment product maybe you're fine if there is like a smaller set of users but they tend to spend 30 minutes, 60 minutes in your app more than once a week that might be a very good state to be in the other thing is even when you think about consumer products they may actually be tied into a smart phone a lot of apps for instance let's take an example your banking app yes you're ideally in love for your users to be able to get to their bank accounts to bother checks from their phone but if the rest of the time they're not actually coming into the mobile app but are doing their banking to the website or other mechanisms that is also probably okay so factoring in what type of product what are you trying to go after is what will dictate or inform your goals and that's why it really depends on the specifics and it can change from one type of products so let's talk a little bit about metrics next what type of things come into play and there's obviously a variety of different metrics some of those become more relevant on certain types of products or one form factor like web versus mobile app but there's a few common things so let's think about it one is this category of acquisition metrics and so one way to think about it and I have this I don't have the visual on the slide but somebody shared a very nice visual you guys are familiar with the marketing funnel concept several heads not great so that's one way to think about it is how many users am I starting to acquire or starting to pay attention to even what I'm doing so let's try and define that the moment somebody installs your app and opens it for the first time let's call that a simple acquisition so then what my track is number of users who came into the app the next is activation I came into the app but let's say I actually went ahead and completed the sign up flow so at that point I went there and I said I'm willing to be an ongoing user so you sort of activated them then if you think about it the third category is retention is after you ended up at this date at this date how many of them are coming back to your app 7 days after the installed it 30 days after the installed it could be good activation another one could be especially one that's more common on the web but definitely can apply to the app daily active users 7 day active users monthly active users and in fact these metrics are so important where if you take for instance Facebook I haven't checked their last couple of quarters but in their initial earnings calls they were actually talking about they probably still do the number of daily and monthly active users they have that's a metric that's watched closely by even like Wall Street investors so some of these are of interest outside but for you as a product manager these are great metrics to know whether or not your product has succeeded in this engagement and this is also an interesting one where you think about again it might be best explained with an example so let's say actually let's do an agnostic one and then I'll take an example so something like number of sessions and average time per session are interesting metrics to track since we're talking about Facebook for a minute in their case yes it probably follows that if a user is actually spending more time on the news feed checking on what's happening with their friends or other pages they like it's a good thing so it's generally to track if it started falling by some multiple amount there'd probably be a lot of teams inside Facebook or their product managers and generating teams whoever would pay attention to that and then think about what are the implications of that trend do they want to try and do other things to try and change it for instance changing how the day of the news feed changing how often are they sending the reminder emails I got an interesting one today saying such and such person commented or liked them such and such other person's posts and like that's an example either what the other person might want to click on straight away or maybe why am I getting this notification but not to click on the notification the idea is by using mechanisms like this you can try and influence the engagement rate that you're seeing in your credit care product there's various kinds of metrics there's of course much more literature in this that there's many good articles and stuff some of you might ask offline I can try and share some resources but the idea is being careful about which type of metrics you track and tracking a good set of metrics is especially important for consumer products because in consumer products typically it's actually harder to spend time directly and I shouldn't say that across the board but in many cases with your end users you may not be able to go as deep you might not be able to sit next to them you might be able to watch a few but Facebook can't spend like you know 100,000 can't afford to spend the time to watch 100,000 users watch their news feeds every day or like every month so this is where watching these metrics and trying to understand and work with trends or changes in the metrics is an important thing another thing though I do want to question is if you start looking at metrics you probably obviously want to do a few that you quickly start tracking but you probably want to take the time to have either yourself or somebody in your team spending energies to understand the trends that you see for instance let me take a simple example if you see monthly active users suddenly drop off by 30% something's not good but then if you try to group cause it is it so that you've slowed down the rate of acquiring new users or is it so that that's constant but for some reason these users have stopped coming in just this next level diagnosis might end up impacting what decisions you choose to make to try and react to them if it's so that we just are we typically acquire 1000 new customers users each month that rate has slowed down to 200 and typically like some percentage of them convert and become monthly active users okay and maybe the attention where we need to pay attention is to start getting the word about our app or if it is so that it's the same users that are coming in that are coming in net new but are just stopped using the app that often then you might choose to do a different sort of thing just investigate what's happening then you might go and say okay let's look at let me take a website example where it may be simpler let me look at the last visited URL so if your website has like 50 different pages what's the page where we are using the users the most that with two weeks back or a month back and try and analyze it so as soon as you start watching just a few of these metrics especially if you're trying to root cause probably need to go a few levels deep and start doing this type of analysis I don't cover them in the rest of the deck but just as an example Google analytics as one product that doesn't this type of thing there's obviously many others in the industry which are one in one use so what if you make a new product or you're testing something new you see a draft anywhere you anticipate a new interface adjust to it how do you diagnose that and say it's been too long that they haven't my results haven't gone sometimes it drops because you expect it people don't like change but sometimes you force them to do it and how do you decide where to go from there right so I think one thing is for you anything drastic especially if you have any popular website such as Google or MSN.com Wall Street Journal or what not something that really matters a lot you typically do is you go ahead and always experiment first and then you can experiment by saying there's a change that you want to make we're going to only send it to a small subset of our user base right say 0.02% depending on how many or 1% depending on how many users the site was getting and then you actually spend a lot of time analyzing the results of the experiment and if the experiment was in conclusion you can always run it a different way for instance start getting different demographic different time of day and what not but there's these types of things which you can and then make a decision does that answer the yes we will okay how do you go ahead and just some sharing some suggestions on building effective and useful consumer products very important we find the goal of your productivity like going back to that previous slide what is the problem that you're trying to solve and then because what can sometimes happen is if you're not careful for instance let me share with an example let's here start tracking page views per day pretty simple yes how many pvs am I getting the thing is I can get page views probably one of two ways one is I can go ahead and have well thought out articles and like the user takes the time to read it or I could go ahead and do one that maybe the 10 richest people in California like as everybody does next next there are some standards set by actually I have set some ones for us but there are some standards on what counts as a page view and if you flip through something like this five times it might count as five pages so for a while you could get a metric high that oh this metrics amazing everything's going great but you lost focus on what was the goal while you were trying to solve this you were trying to give good advice around help let's say to my user base and so there's different metrics you can try to get but the larger point is being mindful and keeping track of which specific problem they're trying to solve and then focusing on the right metrics will actually help you keep tabs on your product as it goes couple other things to share is we spoke some about target demographics but it's actually interesting to think about it is a lot of times when we think about demographics we might only think of age or geographies like hey it's people in a specific user base or it's people in this geographic region versus the other gender like a few other things but the interesting thing is if you start being attention to this and combine it a little bit with the last bullet point here's an interesting exercise to try is to try to understand who are your friends so if you have a consumer app or a consumer website and then you start trying to get a sense of which type of user tends to spend a lot of time what you might find is there is a specific demographic combination of these factors which is actually responsible for say 30% of all your behaviors and then you check how many of them there are and then you try and make active efforts to try and grow that type of demographic and that might be the thing that actually keeps the needle in terms of your consumer apps and option that's another interesting way to think about consumer apps how do you stay relevant to consumer apps and you know like products like Blackberry, Calm, and so on these are very large products that ended up yes at some point I'm starting to lose traction how do you stay relevant I think one is obviously watching the metrics but I think your question is in a different direction is trying to keep sense of what else is happening in the industry like for example you mentioned Blackberry there were some very large trends in the smartphone industry which for theater which again I don't follow them too closely so I don't want to conjecture here but potentially failing to adapt to some of those trends in a timely manner in other cases it might be that there is a different sort of app or a different type of website which is what users are gravitating towards which has the network effect for instance one of the reasons why personally you don't use WeChat as much is I don't have that many friends or people who I know who are on WeChat my case a lot more of them are on WhatsApp but if you talk about let's say a user in China the exact opposite maybe they might have almost every one they don't want to WeChat the last category we'll talk about is business products and so to contrast this I think with the enterprise products which we covered first the one thing what it is their key function is that they have advanced key business functions and that is an interesting factor to keep in mind is at the end of the day if for instance you are a finance or an accounting product what are some things you're trying to solve trying to make sure you have the right things in place to be able to do tax calculations at the end of the year and you have the right type of records that are available inside the company so this is the type of key aspect of business fun the other thing which you a little bit becomes acute here in different places there's different personas you have some executives you may have people in sales ops you may have people in IT you also have your end users which are actually interesting pieces to get into integration with custom processes and tools is another key aspect in business products and then over time the focus shifts from going ahead and just having a basic thing in place to being able to provide the right type of systems so let me take an example let's talk about CRM product a basic feature might be you go ahead and indicate here's all the customers or here's all the deals you haven't played here's your status and then some way for them to be able to track their activities and you probably go ahead and spend time on doing something like this CRM product is fashioning but once you have solved a lot of these core use cases or features what you might then arrive at is alright now how do I go ahead and actually start having the right things be available in their workflows before they actually user realizes that you are needed and that's where it's very impactful for instance the moment that I'm getting off a call or just going to imagine something here is I get off a call and I'm starting to make some noise and typically at this point I tend to send an email to my team sharing an update and the system was able to understand that there's something like this happening and is able to have features that start making this available and other type of smart could be you're starting to provide more analysis for the user while they're in the app so that they have a better sense of okay this is the revenue that came in here's the trend we are seeing here's how it varies or not with what we're seeing in other parts of the sales org so these types of things are pretty powerful value apps that can come key metrics to keep in mind are product reach the total amount of revenue for instance again taking our CRM example what's the total amount of sales that are happening through the app through the app what's the number of deals that we've had how are users engaging and what's their productivity that's another way key aspect in business software is at the end of the day if you're helping obviously meet the needs of the organization for which you build a product but are also able to help users be efficient and productive going back to functionality like we were to do that in a business setting where they can go and do their thing and get out and it takes them a small amount of time versus other products that can be an interesting advantage some strategies to try is understand the decision makers and functional roles we covered some of this earlier so I won't go over it again currently discuss the right metrics and expectations it's not interesting when you talk about it especially in business settings and sometimes in enterprise software settings the standard metrics are important it's good to have that conversation in cases where you have a high touch engagement model with the customers because it might be that for instance you were very focused on trying to make the whole experience be very fast whereas what they wanted was more coverage and then you focus on a thing that was different than your user base was actually interested make an ROI argument and demonstrate measure with impact and prioritize keeping your end users happy because at the end but then there's a certain set of users who are actually building the product for and as a PM you are their first and foremost advocate so go ahead and pause here these are all the key trying slides we have it's just up to you we can go and do one of two things we can probably open it up for Q&A and if you wanted to we could take a couple of product examples and actually try things out I'll be breaking to Q&A and see if there's any questions please how do you do enterprise products which are long product low level cycles that's an interesting question I spoke about experimentation consumer products but how do you go ahead do it in enterprise depending on the type of enterprise product if it is one where you have a high touch engagement with your customers and I keep saying high touch engagement just to explain that for most enterprise products you'll probably have some customers who you have a relationship with where they are happy to collaborate with them either directly at their sites or at conferences or other mechanisms and there's probably a larger set of customers who yes are using their products but are not having that people from engagement with them so in cases to experiment if there was a few if there were a few customers that were interested in trying out a specific aspect one is you can engage interest honestly a lot of the time is what you're building will help solve that customer's needs they will be happy to try it out or at least get a desk setting that's one way to experiment try and integrate it with their on their actual deployment if you were doing it via the SAS model or by the cloud deployment model which is effectively where you control you could still do an experimentation in an enterprise setting by for instance changing something about going backward database analogy for just two person of users noticing what happened before and after of course if it's a big change you're probably going to let them know if it's something minor within SLAs you could still do it and see what you get and then decide whether you make that available more broadly what kind of treats that you prefer than habit formation for new customers or old customers in which type of product in general or in general like what kind of techniques that you see or do you see any specific difference in enterprise versus consumer yeah so how do you sort of go ahead and have habit formation especially if it's a frequent need type of a product then you go ahead and see if you're actually solving or focusing on the right user need and are you continue to track and see if there's ways that you can do it better start policing if you see users are falling off and are starting to go on the other side discuss an example like try and analyze diagnose that on the positive side try and accentuate economics try out different things for instance if one of the metrics you chose to focus on was the common stone that somebody spent in your app or a website and then you can explore for instance by experimenting with a different type of content that you might make available to them or some different UI changes if that helps but at the end of the day it is and I think one part of that comes down to just innovating and experimenting and the other part comes down to doing the right type of metrics analysis that's probably a general answer because you mentioned is there a specific example you could talk just from the what you thought of this feature and how you would like sort of check your type of analysis and actually what you think is something specific I think given that that's a product from an employer that I worked on previously maybe I like take that's a good example but in general maybe offline we can chat about some general industry use cases where that happened and how customers and companies on the tools that you found very useful as a PM on daily basis tools tools can take a white variety of shapes some that I personally found useful let's see one is obviously various kinds of trackers could be Excel sheets Google Sheets whatever you do prefer going there and using some ways to structure some of the fields so you don't always have free flowing text it's a great way to organize some more interesting ones because when you go in and talk about and there's several so I'm not saying these are the best but just the ones that I happen to try for things like UI walks there's a tool called Sammi which is something you can experiment with experiment with that on time that was useful another was in PowerPoint or Google Sheets itself like if you're doing UI prototyping another way could be to go ahead and for instance draw out your let's imagine you're about to build a mobile app and then have certain templates be available below and then you make that the slide background and then you try and drag and drop things these are some ways especially when you don't have a UX designer or you don't have as much of them as you would like you can experiment with then there is other types of more formal tools that are on experimentation various frameworks that depends a lot on what type of the deployment system your company uses whereas at BM I've typically not been involved but after they are installed working with whoever is running the experiments to see the experiments being run and then analyzing the results