 Thanks so much. It's so great to be here. Let me get started. So my name is Karen Bruch. I'm a senior product manager at LinkedIn. I run LinkedIn salary, which is a tool for being able to access aggregate compensation data that we've crowdsourced from LinkedIn members. Previous to that, I was leading growth for Yahoo Mail, which is the topic of discussion tonight for the most part since I did that for a while and I've been at LinkedIn for about seven months. So how many of you are product managers? Cool. How many of you want to be product managers? Cool. Some people who are just here for general interest, for the food. Great. So bear with me. You don't have any reason not to trust me yet. So can you stand up? Can you stand up? Thanks. And we're going to do a body poll, which means we're going to, with our bodies, say how ready we think we already are to take on a product manager role. We can start all the way down here if we want or we can go all the way up here and on the count of three we're all going to do it. Ready? One, two, three. Okay. Okay. Cool. All right. So we've got like a little landscape going with some mountains and some valleys. That's really great. So what I'm here to tell you is that it's all about the perspective that you already have. You could be a baby. You could be a baby wearing glasses. It's about figuring out what is unique to you. Thank you. You can sit down if you want. And figuring out how to hone your perspective so that you can really impact your product. So tonight I'm going to tell you a little bit about my perspective. That's me. I'm using a regular phone. So anyway, I have a work in progress mission statement. It's to provide access to opportunity at scale through data and technology. This is work in progress, but this is my North Star and my way of figuring out how I'm going to make decisions about what I do, what I care about and where I'm going in my career. So this is my career, my life arc. So I was born in New Jersey. I grew up. I tap danced and I played the cello. Not at the same time though I tried. Then I went to Barnard. I fell in love with economics. I studied economics and social history. And I also, I learned how to become a leader. I was part of a leadership program called the Athena Center for Leadership Studies, which was fantastic. I interned at Grovo, which is a micro learning startup in New York City. And that was when I realized that technology was really cool. I did a combination of product management and their first kind of marketing. They were about 25 at that time. And that was kind of my first foray into technology and startups. So when I graduated college, I decided to move to California from New York and eat kale for the first time. I still like it sort of as much as I'm supposed to. And I actually started out in a marketing role. So I was a marketing associate at Yahoo. And then over time I was able to make the transition to product management. From there I became a data nerd and an experimentation nerd. That's actually how I made my transition from marketing to product was because I kind of fell in love with experimentation and latched onto it and just really learned a lot about it. So a lot of what we're going to be talking about tonight is around that transition. So now I'm solving salary transparency by running LinkedIn salary at LinkedIn. And so I'm kind of fulfilling my mission statement already, but we'll see how far that can go. So we're going to focus just on this specific piece of the journey for tonight. But I'm happy to take questions about any part of that. So imagine that you are a first time PM and you are tasked with leading growth for Yahoo Mail, which is a mature product that's been around for 16 years, probably more actually. And you have a fully stocked engineering team. You have eight engineers, you have design, you have an analytics, and you have complete autonomy. Basically, leadership has said we need a growth team and here is a team and we are going to give you complete autonomy with how to solve this problem and no direction whatsoever. What are some things that you might do first? Does anyone have an idea yet? Find out what they mean by growth. Find out what they mean by growth, which is related to figuring out what metric might matter. Anyone else what might you do in this situation? Yeah? Benchmark, where they are today. Benchmark, where they are today. So again, understand the numbers, where you are, where you want to go, what matters. Anyone else? Yeah. It's not complete, skill sets to achieve that. Skill sets to achieve that. So in terms of specialization. Specialization for now, what kind of, yeah, exactly. Great. So the first two are what number are you trying to move? There are plenty of people who have given presentations on the metrics that matter and how to figure out what your North Star metric should be. I'm not going to focus this presentation on that, but I will tell you that the metric that Yahoo Mail was trying to move was daily active usage, for Yahoo Mail, which is a huge product, is a huge number, and we were focused on existing user and new user retention. The next thing we need to do is to be able to fail quickly. Nobody had ever tried to do this before, so we had to figure out a way to move as fast as possible and fail as quickly as we could. How did we do that? So there are a few points on this slide. We worked on building experimentation infrastructure, so we moved from single layer to multi-layer experimentation. If you don't know what that means, it basically exponentially increases the number of experiments that we can run at the same time. You have to learn how to do great experiment design and avoid those, oops, we set up the experiment wrong pitfalls that many people can fall into. You start also with hacky designs and UX research, so if you have an idea, you come up with something that's really easy to build, you build it, you validate it, if the metrics look good, then you can build on top of it and work on that craftsmanship. You also have to have a decision-making framework, so when you're running a lot of experiments at the same time, you have to make sure that you know exactly when you start them, when you're going to end them, and when you're going to decide whether to move forward or not. We were very rigorous about our experimentation cycle, how long they were going to last, and very rigorous about the decisions that we made. Then the last one is keep your learnings together. We ran hundreds of experiments, and you need to make sure that you don't forget anything, keeping your experiments in one place is definitely advisable. What did we do? We actually doubled the number of experiments that we ran every single quarter for an entire year through the tactics that I just mentioned. We were learning so, so fast. The question you might ask now is, hey, Karen, you were running so many experiments, and you were doing so well at it, but what were you actually experimenting on? What were you doing? What were you trying to find out? We focused our efforts on trying to validate first what other teams were placing their bets on. There were different feature teams within Yahoo Mail that were working on different parts of the product that had these bets about what they thought was good for Yahoo Mail. What we decided to do was test whether those bets were, in fact, the right bets for Yahoo Mail. We built infrastructure to drive awareness to those different features. We measured retention. We made sure our metrics were really good, and we looked at all this stuff, and we ran hundreds of these tests. You want to know a secret? Nothing worked. Nothing worked. Just to go back, can we get new and existing users to use the product more by showing them these awesome tools around writing and organizing email? You have folders. You can make your email look stationary. You can add gifts to it. You can do all these things, and this tool is just amazing for writing and organizing and sending email, but nothing worked. Nothing was moving the needle, so we had to go back to the drawing board and figure out what was wrong. I thought about it for a very long time, and then realized I had an epiphany. I realized that the issue that we were having was that we were looking at Yahoo Mail as a tool. I'm sure most of you guys have gone to a restaurant before, so we have common ground there. This is a plate for anyone who doesn't recognize it. When you go to a restaurant, you may get a plate that's square, that's shiny, that's nice, that's not nice, and it may be very, very good, but you're not going to a restaurant for the tool of a plate. You're going to the restaurant so that you can eat, so you can have a steak. What I realized about Yahoo Mail was that we were focusing on features where we were kind of shining this tool. We were adding little things here, little things here, but we were ignoring what was central to the experience, which is the content that is in Yahoo Mail. We weren't experimenting around that. We were just assuming that Mail was going to come in and that we would be a tool to be able to organize and store it and write emails and so forth. What should we do? We should serve users with what they deserve. We should give them features that get them closer to the meat of why they come there in the first place. They're coming for the meat when they're at a restaurant, give them better meat, not a better plate. It seems intuitive now, but this is a big shift from the way that Yahoo Mail was thinking about feature development. Yahoo Mail is not a tool, it's a content experience. What happened next? I had this big epiphany and I had to prove it out. The first thing we did was this very kind of hacky test for the Rio Olympics. It's very small, but over there on the right hand side is a tiny metal counter. For the Rio Olympics, we put a metal counter into Yahoo Mail where every day you would come and you would see the updated count of the metals for the top countries. I went into a meeting with one of the senior execs at Yahoo and I said, I want to run this experiment. It's a content experience in Yahoo Mail that's not necessarily related to email, but I want to see if it's going to drive growth. He said Yahoo Mail is a tool. This is frivolous. This is not something that we want to add in Tiamo. We don't want to dilute the experience of the amazing tool for organizing and sending email that we've created, but you can do it this once, but don't annoy me again. It turned out that for the people who interacted with our metal counter, we saw a lift in days visited and compose, read, and send, which are all the metrics that we care about. It worked. What did we do next? From there, we then said, okay, we need to make it easier for people to add content to Yahoo Mail. On the mobile side, because I was running at that time, both desktop and mobile growth, we actually let Yahoo Mail users add any mailbox to their account, whether it was a Yahoo address or not. This was also a really new concept because for Yahoo, there's a lot of pride there around having a Yahoo email address. We want people to have Yahoo email addresses. We want to keep that to ourselves. We don't want to dilute that, but our hypothesis was if you let people add the content that they already care about into your experience, then you have so much more freedom to get them to stay there. We were right, in fact. When we did this, there was a material improvement to our new user activation rate on mobile, which led to higher retention for mobile. We're now going to level up. I really like this gift, but we're going to level up and take this hypothesis to the next level. I'm going to ask these questions. We're going to go through them again, but what data is already flowing into your product? What parts of that data are users already interacting with? What are the rhythms with which they do they have around that data already? Then how can we create new rhythms to get them to come back more often? I'm going to go through an example. What data is flowing into the product already? For Yahoo mail, this may be obvious, but we have billions of emails. We have billions of emails coming into our product every single day. We have a ton of data flowing through our product. What parts of that content are members already interacting with? Can I get a show of hands of anyone who participated in Black Friday? We have some people. Anyone who opened a coupon in the last month? That's a lot of people. Anyone who sent an email to their grandma or a similar relative in the past month? Okay, a few less. What are the parts of the content that members are already interacting with? Shopping. People receive a lot of shopping emails and they interact with shopping emails. Because we were looking at the data and focusing on the content, we were able to see this. I just want to do a little rewind because we were talking about building features for being able to write the best emails possible. Stationary gifts, this and that, when in fact what really, really matters here is the fact that people are receiving shopping content and opening shopping content and not very often writing those very deep composed emails. What rhythms do users already have around that data? I mentioned this already, but people love coupons. People get deals. They know when there are sales. They love these coupons. This is something that we decided to focus on as part of this content-centric experience. How can we create new rhythms to get users to come back? We know that we have billions of emails. We know that of those billions of emails, people open shopping content. We know from that shopping content people open coupons. We know that coupons have already an inherent rhythm within them, which is an expiration date. They already have a time in the future where you want to make sure you come back before that particular time. This is already a rhythm there. How do we take advantage of that and amplify it? I'm also a tap dancer. I mentioned that before. That's actually me. When you're an improvisational tap dancer, you have to hear the rhythms and the music, and then you have to create your own rhythm on top of it. Even if a beat is really slow, it's up to you to decide how fast you're going to take advantage of that rhythm. If there are any musicians, you may see that you have eighth notes and quarter notes, so you get to decide. Just like this, a product manager has the opportunity to take a rhythm that a user has naturally fallen into, and then figure out how to harness that and amplify it to get them to come back more often. The team released this just a few weeks ago. This is a coupon clipping feature where Yahoo! Mail team automatically categorizes when you have a coupon, and then they allow you to clip that coupon. When they allow you to clip that coupon, you are giving them a signal that says, I care about this piece of content, and we've made it very easy to show you that you should. Then you give us a signal to then let us remind you to come back and see it again. We've taken something where there already was a rhythm, and we've now gained more control over making that rhythm, amplifying that rhythm for users. Now we have identified a content experience. We've tagged content. We've extracted coupons. We've allowed users to clip them. We've created a system to be able to remind them about it. We're now starting to create a viral loop. How do we scale this? I'm going to talk about two different systems that I worked on while I was there, and they've probably moved on and built a lot more features in these now, but we built a content-centric experimentation platform. Unlike before where experimentation, all those experiments I ran were based on simply on UI. The content-centric experimentation platform tags the content as it comes in, so that we understand what it is and therefore can run experiments around it. The second thing is the Athena targeting system. I named it Athena after my Athena leadership program. It basically is a way to use business signals about user activity to target features. Here we have a way to classify content so we can know when there's content that's interesting that we can use on the content side. On the member side, we know how often they use it or how often they come, and we're able to map those two things together and then deliver the experience that's important to that user based on what they care about and based on the content that is in their inbox. We went from having a global feature set to being able to have a unique makeup of every single inbox from a content perspective and from a user behavior perspective that we can marry together to provide signals and pushes to get the user to come back. That's it for talking about Yahoo Mail. You may be asking, okay, so now you're at LinkedIn. You're running LinkedIn salary. Why? So raise your hand if you like getting paid. Yeah, cool. And if you want to make more money, yeah, cool. And if you want to know if you're being underpaid, yeah, okay, great, everybody. So LinkedIn salary, if we go back to this model of figuring out the data and then using that data to some ends or means by creating these rhythms for people, LinkedIn salary is an amazing next product to be working on, especially given my mission statement and given that these kinds of problems disproportionately affect women and people of color. One, so we're starting with data that people care about. You all have proven to me in this room that people care about salary data, knowing if they're being underpaid, knowing if they could be paid more, just understanding information so that they can ask for a raise effectively. Then we're going to build the best compensation data in market. So LinkedIn is a huge platform. We have a lot of avenues to collect this data from our members. So we're going to go ahead and kill this. It's going to be great. After we do that, we're going to create new rhythms for when and how early a person knows the value of their work. People don't have salary data generally about their roles and their titles. We're going to change that and we're going to make sure that every person has access to that information. And after we do that, we're going to eliminate information asymmetry from the labor market. This might take a little bit of time, but you can see from the trajectory of how we are starting to unwrap and uncover and push the market towards a particular way with the content-centric experimentation with Yahoo Mail, you can see how the tools exist to be able to do something like this for LinkedIn and for LinkedIn salary. So that's what I'm working on now. So just a reminder, this is my personal mission statement and I think that using data and using this way of massaging and creating rhythms around data, I'll be able to be more able to fulfill this mission. So we're getting towards the end now. So I would like to leave you with just a few questions to ponder. I don't have the answers to them. You may not, but just some things to think about. One is how are the products that you use every day underpinned by a flow of data, whether it's your Facebook news feed or your email or whatever it is that you use every day? How is there already data flowing through that? And then I want you to think after that, how is that data flow being curated by a person? If we're all product managers and we're all making these decisions about how data is being pushed and how we're creating these experiences, think about the person behind that data flow and what decisions they're making about what you see and also what you don't. And then the last point back to the beginning is how are you with your collection of experiences shaping the products that you're going to be creating? Because we all have a perspective and we're all bringing that perspective to work and that perspective is causing shifts and changes already even if you don't know it. So I would encourage everyone to kind of hone that, understand that because you all have what it takes to be a product manager and to make these products successful in your own way. Thanks. Q&A. You can ask any question you want. Yes? I'm going to repeat all the questions. So how much of my job is related to experimentation versus being in meetings all day? I actually wouldn't equate those two things. With Yahoo Mail, Yahoo Mail is a mature product so we had a ton of users and were able to run a ton of experiments. LinkedIn salary is a pretty new product so we run fewer experiments simply because we are still figuring out the strategy for the product. Meetings are a separate thing. If the question is about how many meetings does a product manager normally have, the answer is too many and a lot. But that doesn't mean that things aren't getting done from an experimentation standpoint. Yes? Sure. So the question was how to distinguish my role versus the data analyst's role that I was working with. To take the question to the next level, I have this idea that my team, my cross-functional team, is kind of a brain trust and we all bring some expertise to the table. So I have an analytics partner and I had an analytics partner at Yahoo as well. That person sees things from a slightly different perspective than I do. So we both work on the data together and we both work on analyzing and seeing things in that data. That person likely does more of the number crunching on a daily basis simply because I am in meetings. But we work together to look for outliers, to understand where the data is going, and to understand what it means. Yeah. So people answer questions with data all of the time. Just to repeat the question. So how do you translate the data into understanding a rhythm? So people answer questions with data all the time. But you have to be asking the right questions. You have to be asking questions about the headroom. So in the case of coupons, when we were initially sizing the experiment and figuring out what kind of impact we thought it would have, the answer was how many people open a coupon per day? How many people open a coupon once a week? How many people open a coupon once a month? And then what is the delta between them? So if you know that, for example, there is someone opens a coupon once a month, then there's like 29 more days that they could open a coupon. But if you see that people open coupons every single day, then you know that there's zero additional days that you could drive them to open a coupon. So we did calculations similar to that to understand. But a lot of it also does come from this kind of user experience research and really deep understanding of our users. So for example, we would do user research every week where we would ask users, hey, just like I asked you, when was the last time you sent an email to your grandma? Or when was the last time you opened a coupon? And what you would find is that there's often a difference between what people perceive as their behavior and what their actual behavior is. So you look out for those kinds of things as well. Yeah, such a great question. So we were talking about tagging the content. How do we decide what tags we're going to have on the content based on what we're going to experiment with in the future? So the answer to that is that we had to be really careful about how we designed our backend system to prepare for maximum flexibility. For example, a content piece could, a piece of mail could, would need to have the ability to have multiple tags. So it might be a coupon, but it might also be an offer or something else that we define. And we also would have to build in capabilities to be able to review and change that. So a lot of the early hacky days of this coupon experiment without this content experience that I was involved in involved manual extraction, a combination of manual and machine extraction for coupons. But the way there, I'm not sure exactly how they finished this content experimentation, content centric experimentation platform, but I know that building in that flexibility from the beginning is important for making it work in the long run. Other questions? Yes. In shifting from the Yahoo Mail as a tool versus Yahoo Mail as a content experience, do we do anything to validate that hypothesis? So the experiments that I showed you really did. So first of all, the fact that what we were currently working on and what we currently thought were our bets were not working. So that was one form of validation by disproving what we were doing. And then the second form of validation was things like those, like that Olympics experiment that I showed you, where we had something that was content-based and we knew that some amount of people might be interested in it. And then we looked exactly at those people and saw if there was something that they cared about that we highlighted, then would they come back? So in this case, it was people cared about the medal counter in the Olympics. But if you zoom back and say, what is this thing that a few people care about that we could put directly in their face, then you can scale that idea and test multiple things. But I think the point is that not everything is going to work. You still have to be ready for failure at any point, which is why what's actually more important than this coupon experiment or the Olympics experiment is actually how we are going to scale this idea, how we're going to create a backend where you do have the ability to be flexible and test things in the future. Yeah. How are you doing? I can talk to you. So we think about data and machine learning. I mean, did you look at the data, for example, if you look at the data of number of users engaged in open mail, junk mail, et cetera, or did you really focus on, okay, let's experiment on the Olympics for coupons and focus on test for reviewing that set for data? Or did you look at the data in total? The question is, did we look at mostly data around kind of the holistic product or just the test you wanted to run? The holistic product or the test you wanted to run? The answer is both, always both. So we always had kind of this base layer of what are the experiments that we're running right now and are they working? And then we were also doing these kind of deep, broader analyses of our content that were much more extensive and took a lot more time. Yeah. So were there any organizational challenges to getting buy into that shifting strategy from content, or sorry, just from as a tool to as a content platform? And was there anything that you did to help get that buy in? Yeah. The question is about buy in on the strategy shift from Yahoo Mail as a tool to Yahoo Mail as a content platform. This was not an easy thing. So as you saw, we ran hundreds of experiments. We basically validated or disproved many, many people's hypotheses until we could get to this point where it's like, okay, well, if that doesn't work then what will? I do have to say though that because of the stage of the product, people were very open to this idea that there could be something new and different that we needed to try. And they were very supportive in backing us from the beginning. So even giving a PM a full team with total autonomy, no direction whatsoever, is already saying, I care about this problem and I'm going to invest in it. And so when we would continually go back, when we were pitching Athena, when we were pitching content centric experimentation platform and pitching, we pitched all these things up like bottom up from bottom to upper management. If we had the theses and the data to back up our assumptions, we were often supported. Yeah. Yes. So the Yahoo example was great. How we were able to see and verify that what the examples the experiment experiment you're running didn't really back up. But the items were material and any advice on to as an approach to directly get to that material experimentation sooner, faster, to really have a more stream on approach to really So any advice on getting to the more effective stuff faster? So yes, I think first of all, me sharing this journey with you is part one of that. But what I really encourage for your product is to kind of think about these concepts and think about the extent to which you're moving the needle with the things that you're working on right now and ask yourself, are you really working on the hardest problem that is closest to the user value? In the case of Yahoo Mail, working on content is the hardest problem. It's much easier to build an organization mechanism, you know, JIF builder. It's much easier to do that than to focus on the really difficult problem of the billions of emails that are coming in. So I would just hopefully you will be able to take some lessons here and apply them. Yeah. Sharon, you obviously have a lot of experience with experimentation and especially in the Yahoo and the LinkedIn with millions of millions and millions of data points. Were there any experiments that you were tempted or you actually decided, okay, now let's interview individuals and understand that this behavior is really what we think it is. Any guidelines on, all right, when do you break, when do you actually talk to the individuals for help? The question is on using behavioral data versus more qualitative data and talking to individuals versus using this kind of these big data sets. So with the coupons piece, this actually started because we have an amazing user researcher, frankly, at Yahoo who every week when we talked to just a handful of users would ask the same questions. How often do you check your email? What kind of email do you open? What kind of emails do you send? And he had a list of questions that every week he would ask these users and it was really only four or five each time. And he is the first one who kind of rang the alarm bell of, hey, people care about coupons. Like this is important and people care about shopping, people care about coupons. But then it took us really believing and this shift and looking at the data to kind of rally around that and say, okay, like this is a thing that we need to really focus on and this is a thing that we think is going to work. Yeah. The question is, how did we get to understanding what the user value was and what the hardest problem was? So it really was a process. We started with running all these experiments. We started, we were kind of amping up our infrastructure as a team and it really came out of this limitations lead to creativity. Nothing was working the way that we wanted it to and so we really had to get creative and think about what was wrong? What was the character of the kinds of things that we were trying? What was the character of the things that other people are trying? So I don't have a scientific answer for you as to how we deduce that, but I think that's also the point. Taking a step back, like you personally have a perspective that you're going to bring to a problem that's going to cause it to move differently and you should trust that it's instinct because if you think about these things, like to all of you guys it may seem a little bit obvious, of course you'd want the stake, you don't want the plate, but it really is something that requires you to step back and trust your instinct. Yeah. The question is about user acquisition. So because Yahoo Mail is such a huge product, it was actually much more important for us to focus on existing user retention than it was to focus on new user retention. We could definitely have focused on that and had gains there, but the like amount of gains that we would have need to get were much bigger, we're sorry, we're too small for new users and much bigger for existing users, so that's why we focused on that versus on a new user acquisition. Any other questions? Yeah. The question was how did I make the transition from marketing into product? I'm so glad you asked. I know people are wondering about that one. So I was a product marketer assigned to this newfangled growth team and I didn't know what it was going to be. I started working with them. I realized that they were doing something incredible. They were trying to get at the heart of what really was the opportunity for Yahoo Mail and I started learning about experimentation and I had a mentor who was the first PM on that team who said, hey, you want to learn more about experiments? Great. Why don't you analyze this experiment? Yes, let's do it. Then why don't you report on this experiment? I was like, okay, so I made some awesome slides. Then why don't you report on all of our experiments at a weekly product management meeting where we talk about it? So I did and every week I went and I was like a thousand percent prepared because I was super excited. Then I started running my own experiments and then me and Josh Jacobson, who now leads the mobile communication products at Yahoo, we had basically split his role entirely in half. So he was working on some other big bets that the team had started on and I was running all the experimentation for the team and so when he moved over to run our mobile apps, then that's when I made the transition. The question is how is LinkedIn salary different from Glassdoor? Thinking about what we just discussed about these billions of emails coming into Yahoo Mail, LinkedIn has 500 million users. So LinkedIn has an ecosystem that has job seeking, it has content, it has networking, it has job posting, it has recruiting, it has all of these different things and so what we have is a way to tap into this ecosystem in a way that will create a lot of opportunity for LinkedIn salary. So we have a lot of ways, we're kind of a baby product, we're still pretty new, but we have a lot of ways that a lot of potential that we can tap into by being part of the LinkedIn ecosystem. Yes? The question is talking about my transition from product at Yahoo to product at LinkedIn. So the two are very different, even though the CEO of LinkedIn actually came from Yahoo, so it feels like cousin sometimes, which is kind of fun. I think the biggest difference is that when I was at Yahoo, I was working on Yahoo Mail, which is one of Yahoo's biggest products where there are 17 product managers working on this one product and then when I moved to LinkedIn, that had been around for a long time, when I moved to LinkedIn, I was working, I'm now working on a product that is in its infancy that I'm leading as the sole product manager and that is really in more of a strategic part of its lifestyle. So I think the cultural differences is not really what is making the two experiences unique from one another, it's actually the chance to lead a product that is so important from the ground up. How do your experiments change when you're moving from something that's more tactical to more strategic? More mature product. So the first thing is the volume changes. So you have less play dough to work with in some sense, so you have to practice the product management skill of being able to see things through the fog. A lot of product managers start out really deducing their way to answers. With all the experimentation we did, there was some measure of that. When you're working on a product from the ground up, there's less that you can do and you have to really kind of validate with user research and trust your instincts in a way that is a little bit more free than when you're working on a bigger product.