 Hello everyone and welcome to this week's Product School webinar. Thanks for joining us today. Just in case you didn't know, Product School teaches product management, product leadership, coding, data analytics, UX design and digital marketing courses online and at our 16 campuses worldwide. On top of that, every week we offer some amazing local product management events and host online webinars, live streams, and ask me anything sessions. Head over to productschool.com after this webinar to check them out. Today, we have an awesome guest presenting. I'd like to introduce you to Alex Engel. Alex prides himself on being a strong athlete that brings both data and user research to any project he works on. He's driven by his core values of curiosity, effectiveness, and delivery, and has a proven track record of tackling difficult problems head on. On top of this, Alex is an expert in AB testing, experimental and data-driven design, and brings a strong customer-centric perspective to everything he does. Feel free to leave any questions for Alex in the comments of Facebook and I'll ask him at the end. And without further ado, let's welcome Alex. Thanks for joining us today. Thanks for having me. Exciting to give a little bit of a talk here. The purpose of my talk today is I'm going to be talking about turning visions into OKRs and objectives. So the primary goal here is to give some insight into how at DraftKings, I was able to take some of my experience from my work in the video game industry and take it into a different field. And primarily how I was able to do that, taking the video game experience into research-driven and behavioral psychology-driven objectives, turning that into roadmaps, and then turning that back into OKRs that makes sense for the core measures of our business. So without further ado, let me get started. So like I said, my name's Alex. I'm a product manager at DraftKings. Most of my work at DraftKings is in the customer loyalty and retention space. So I'm going ahead and working on taking our customers and trying to bring them through the customer lifecycle beyond activation. So after a customer is activated and become a DraftKings customer, how do we make sure that we retain them? How do you make sure that we keep them excited and engaged in our products and ultimately build sort of a moat of loyalty around our products so they don't want to go to any of our competitor products? Prior to working at DraftKings, I worked on a large number of video games and video game studios. So I've worked on about half a dozen titles. Some of the larger ones are here. During my time on those titles, I did work related to anti-fraud and online security as well as operations of the online games. And then as a producer and product manager running games like Star Trek Timelines and Game of Thrones Ascent. You'll see a bunch of YouTube videos with me in it over the years as I worked on Destructor Beam on those products. It was a lot of fun, but now I've made the switch over to the tech industry, which has been pretty exciting. And it's been a lot of fun taking my experience and moving it over to there. So our challenge as PMs, right, is how can we take what we know and what we've learned and overcome the barrier of what would it actually do, right? So often we'll be taking a look at what our competitors have done, what research has done in the space. And we know that it would be valuable or could be valuable to our products and our companies. But we have trouble translating it into actionable things, right? And we have trouble getting buy-in sometimes on what those projects would be. My vision when I joined was to radically reimagine customer loyalty as game design. So some of the online games out there, especially the earlier ones in my career, even online, for instance, when I started working on that project in 2004, it was maybe a couple tens of thousands of people playing it. If you look at it today, it's over half a million people who log into it every month and are engaged, sustainable customers. The only difference is it's been 16 years, right? So how do you get a retention and not only retain but grow your product for 16 years? Most tech industries would kill for that kind of retention. There's a lot we could take and a lot I could take from game design and turn it into a tech product. The first system I wanted to go into, the first thing I wanted to do was bring the concept of daily rewards. This is super common in the mobile game space. It's just a way to reward a customer for making one single touch point and one single step in your product. Most of the time it's tied to a real-life calendar or it's tied to a calendar of times you want to log in. And so when I came in, I had this sort of thesis or hypothesis of daily rewards and courage players to come back day after day, expose it in the core game loops and meta game loops that drive retention and engagement. And now a lot of those terms here I found were challenging to translate across the video game to tech industry divide. So I ended up digging deep into behavioral psychology and saying, well, this is actually just an example of fixed interval reinforcement, which has been proven to help guide behavior and create feedback loops. As long as it's clear and regular what a customer needs to do. This is essentially the carrot that drives exposure of our players to contests and promotions. I am tying it to an OKR here of our users understanding what our premium currency or our loyalty currency called crowns is and how many people believe it's valuable. We ended up finding was that after we released this system it reinforced existing loyalty systems that were built before I joined the company. And those systems in turn increased the people who said, oh, I get it. I understand why I want to earn this stuff. I get how what I earn is tied back to the rewards I get in the future. And I understand and now I believe it's valuable and I want to acquire more of it. But when we first ended up pitching it, it was challenging to turn that into an OKR. I knew it was the right thing to do, right? And I knew that there was research behind it and that research showed that it was valuable for customers to do. But I had trouble translating into KPIs so the business really needed. One of the things we wanted to do was increase day after day paid active day retention. And so we ended up tying this OKR back into that. And we said, hey, crowns, the more you want them, the more we can incent paid active day play by saying earn bonus crowns or extra crowns by playing in our systems. This is the foundation that allows us to do that, right? You can't do it unless crowns are considered to be valuable or useful in the economy. In order for us to start doing that, we got to lay out this first system. So in the context of setting our OKRs, we ended up pegging it towards the usability of crowns, the value of crowns, with the understanding that that laddered up into that paid active day or retention KPI. The second system was introducing things like XP and levels. So if you've gone ahead and you've ever worked with a video game or played most video games, there's a concept of progression, right? So experience points and level are really a proxy for how far a player's progressed in a game. It's an easy way to indicate that a player's made progress and whatever a skill, their play, any system that's set up in a video game. And again, the reasoning there, this was actually a very challenging thing to pitch, is because it seems very gamey, right? It's very much like, oh, that's a video game. How would that possibly apply to a tech company or any other action a customer is doing? And I ended up landing on inertial tendency, which is the concept of partially completed goals, where customers desire to complete whatever we've started them with, right? There is an expectancy to earn new levels. There's an illusion of progress that actually ends up incentivizing people to engage as much as they can with a product. And it's not just a video game context. If you take a look at things like loyalty cards or if you go like a car wash and they say, go ahead and take 10 car washes and get a free one, there have been experiments done. Nudes and Dreas is the famous one where they give you one that has 10 that are unpunched and you punch one right away. Or they give you one where two are already punched and the one that has more punches in it, even though the number of actions are the same, so it's like eight that are unpunched or 10 and two are already punched. The one that's 10 with two already punched tends to be more successful and drive more people to the system. So I went ahead and tied that directly to retention, right? So if I can go ahead and I can build these systems that are going to encourage people to come back by giving them XP or levels, we can directly affect user retention here. By using positive reinforcement mechanics like XP and levels, we can target specific cohorts of users who are affected by that and really improve week over week retention. And the last one I'm gonna talk about here are using things like achievements. Now if you use something like Fitbit or any other system out there, many of them have achievements built into them and it's not necessarily obvious why they would work. Achievements have been around for 20 years in the video game industry. We usually say, hey, it's our long-term goal for our players. These are milestones that allows us to give out awesome rewards as long as players continue playing them. But Noon's Andreas again said, hey, this is actually not just in video games, this is everywhere. Anytime you provide someone with artificial advancement towards a goal, they're actually more persistent in reaching that goal. You frame the task as one that's been undertaken as incomplete rather than one that's not yet begun and people are more committed to completing that task. And that laddered up into the task that we wanted to really incent. Now we already had one where we're trying to get that day after day and that value proposition built up. We have one where we're trying to do week over week retention. This was a system at OCR where we laddered it back to the core thing in every business, right, revenue. How do we take people who have engaged in this kind of system, like an achievement system, and improve their user revenue by an amount through goal setting? So let's say one goal is going ahead and playing every single day of the season. Now you could put a really strong and powerful reward at the end of that, because you know the investment in time and energy and revenue is gonna be significant to reach that point. That then changes the number of people who actually play every single day of the sport for us. So we can go ahead and tie it right back and say, X number of people entered into Y number of additional contests, which increased revenue by Z amount. And that's been really beneficial for us. So where I've had a lot of success in general in sending these OKRs and tying them back to KPIs is taking a look at a lot of the core scientific background behind why people do whatever they're going to do in behavioral psychology. It's really simple, right? Like in all of our KPIs, whether it's retention, engagement, revenue, we're actually talking about is getting a greater share of the user's time and wallet. And these are all mechanics that are well proven for a customer to come back and engage time after time with a system, and you can tie those directly back to the KPIs of your core business. And in fact, you're doing yourself a disservice if you don't, right? If you are doing a highly deeply specialized system and you not tying it back to one of the core KPIs your business, whether that's paid active days, whether that's retention, whether that's revenue, then it's going to be challenging, especially the higher up you get in pitching your ideas. It's challenging for people to buy into it because they go, okay, I understand how this is going to change behavior, why in my product, but what's the actual bottom line impact? And you can model that out by saying, you know, if we improve for the following cohort, their paid active days or the retention by like 5%, then what would that actually turn into? So after you do your initial vision pitching, right? And you've pegged it to some of the company OKRs, you have to start building out roadmaps. Change your ideas into deliverables, right? And again, as PMs, that's one of the areas where we're expected to provide a lot of value. How do we actually take this vision and these goals that we've set for ourselves and internally with our teams and turn it into an actionable roadmap? Just two things that I'm going to provide some insight into here. Gantt charts are always your friend, right? I hate them in the sense that I hate putting them together and keeping them updated, but really there's little that's more valuable than setting up a Gantt chart. So this is just an exercise I ran last year for one of my projects that I was rolling out. And I just had to go ahead and say, okay, great, in a perfect world, these are all the individual deliverables we need to deliver, what week or sprint do we think we're gonna get into them? And that really forced me to start breaking larger goals into smaller pieces because I'd go, wait a minute, like this has dependencies. Those dependencies logically need to happen before this goes live. So let's start putting that out and start moving backward on the Gantt chart from whatever our estimated goal time is. And you can easily share it, it's very visual, you can allow, you know, my directors and VPs are able to take a look at it and go, oh, I totally get it, you wanted to deliver and at this point it's gonna be in test here, it's gonna be live here, it's just your friend. And then this is something I use pretty regularly too, which is laddering up or setting goal horizons for your projects, like it's super easy for you to set the vision at the end. You have to kind of lead people to that. So an example here in 2019, I used this format just last week to try to pitch some of the bigger stuff that we're doing at the end of this year. You take the small idea, which is the first step or the next step that you need people to buy into, you ladder it into the larger idea, which usually incorporates what came on the left and the larger idea, the second one here, that's what your roadmaps targeted towards, right? Your next three to six month roadmap. Then there's the big idea, right? The big idea is where should it be at the end of the year? Everything ladders up to this. And then behind it, right, is the vision. And the vision doesn't change for outside of what, three to five years, right? So everything I'm gonna be working on over the next few years is all gonna be driven by what's the core vision that we've committed to as a product team that we've committed to as a company. The last thing I wanna take a little bit of time just talking about is measuring, right? So measuring is super important and it's something that I'm deeply committed to. It's also very challenging to track when you're building a lot of these complex, okay, ours when you're building a lot of these complex visions. I don't have a lot of visuals here, but I'm gonna walk you through a couple examples just because the visuals can get difficult and require a lot of proprietary information in order for me to show you guys. But obviously the most valuable thing you can do in measuring is A-B testing. Anytime that you can go ahead and A-B test one idea versus the other, then you're gonna be able to have just a solidity of data and solidity of progress that's unable to be seen in any kind of linear or past and future explanation. There's always confounding variables, right? Unless you can isolate the independent variable that you're testing to one thing versus the other, there's always gonna be people in the room, whether it's your analysts, your fellow product managers, they're gonna question the conclusions. You always have to put caveats at the end of them or uncertainties at the end of them. Yes, we believe that this is something that's going to do X, Y and Z, but we can't prove it yet. So how do you tackle the A-B testing and when you know that A-B testing is appropriate? For me, it's when I can isolate a change down to a specific variable. So if I can change the layout of things in an app, then I can go ahead and A-B test that. If I can go ahead and I can change the text in an app, so the title or an image or creative or any of the copy, then that's easily A-B testable. But all the things that I ran through earlier in this deck can't be A-B tested. There's either two reasons, one of two reasons why that is, right? So the first one is that it's too complex. The number of systems it touches, the integrations that have to be run, just make it technically unfeasible or blow out the timeline to some ridiculous time before you could actually feature switch A-B tested. And that's a prioritization and a functionally like straightforward thing that you can tackle. You could say, look, I only have so many engineers, I only have so many resources or the tech debt's so great that A-B testing this would be really challenging. And so that's a lot easier to tackle than the second one, which is we shouldn't A-B test it. There are cases where especially in the video game industry, A-B testing something for a player might actually almost be unethical. An example would be and something I talked about in a talk recently a couple of weeks ago, which is purely hypothetical, but what if you were A-B testing price discrimination based on demographic information? So if you went ahead and said, I know that this customer is more likely to spend a lot more money on something than someone else, then you intentionally manipulate what that customer's like, what the odds are of them getting a rare item in an item pack or something like that is based on that info. Most people in the game industry, for instance, would say that's unethical because you're manipulating the odds and the odds are unfair for people and it's opaque to the customer. The customer doesn't know that that's happening. So whenever you do things like in the past when I've had to change odds of cart packs or loot boxes, you have to be very transparent with it and you can't A-B test it. So you have to do a lot of other funky directional testing. So one thing you can do is you have a really good LTV modeling. You could do predictive LTV modeling versus actual LTV. If you're within say 95% certainty that you understand what the LTV of your users are over a period of time and you roll out a change, you can then take a look at the LTV projections and trajectory of those users. And you can say like did it change is the Delta persistent? Is the Delta outside of the area of significance? Is it trending positive, negative? Even if I don't know what the exact percent increase is, can I get to that significance by saying what's at least not negative, right? We projected that an LTV of a customer was gonna be $30 over the next 60 days. I'm gonna go ahead and say that now it's $35 and that's a significant difference from the 30 that we had before. It's at least a good change that we made. And the other thing you can do is you're gonna allow your users to opt in. So you can say we have this really cool stuff like we've introduced to XP or Levels, we've introduced achievements, we wanna see how it's going. We're gonna allow you to opt in. The biggest issue there is then you bias the sample size, right? You bias the number of people that are going to join. You typically have people who are early adopters join and their results and behavior is gonna be not the same as if someone was a standard user, let's say. And so again, you kind of have to model what their past behavior was. If you have year-over-year behavioral stuff and you can control for other things that have changed like other product improvements you've made, that's another way that you can track and measure the impact of your work if you can't A-B test it. But ultimately, you're gonna have to accept with a lot of your big visions that translate into strong, powerful goals that can't be A-B tested, you're gonna have to accept some level of uncertainty. And the best I can say for the work that I do, right? In many cases is directionally it's positive. It gave somewhere between X and Y amount of increased revenue and even X, which is intensely conservative and here's how I made this conservative estimate. Even X is X number of orders of magnitude greater or times greater than what the cost of the team is, right? Or what the cost of the effort was. Still challenging, right? And still something that can be difficult to measure but those are some ways that you can say, hey, directionally at least these things are positive. Even if you can't say, I drove X millions of dollars in additional revenue. So that's most of what I have here right now. I'd love to take any questions you guys have about either the work itself or taking visions or broader objectives and breaking them down into more concrete deliverables that you can deliver into your teams. Or if you have any questions about draft Kings product process and how our product team works here, I'd be happy to take them from the audience. Thank you so much for that presentation. That was great. I'm not seeing any questions in the comments just yet so we'll let people type them out. But one question we like to ask all of our speakers is if you had one piece of advice you could give an aspiring product manager, what would that be? Yeah, the number one piece of advice I give most new PMs is understanding why, right? So it's not sufficient for you to say that a thing works for a competitor. You have to be able to explain why because you're always going to run into people and conversations and meetings that are going to say, I get that you believe that this is valuable but I need you to tell me why it's valuable. It's one of the things that I feel super strongly about going back to the fundamentals of research and user research and say it's valuable because our users have come in and told us it's valuable. It's valuable because it's one of the top CS ticket drivers. We're just saying it's valuable because we know from past research and past products that regardless of what product a customer is accessing or using this kind of mechanic or this kind of feature is valuable to them. But good example might be using a hamburger menu for your settings and your deeper level interactions with your product. People know what a hamburger menu is. People know that it opens up all the options that even if they can't explain it they understand what it is. So going in and saying, hey, it would be great if we centralize all this information in our app in this kind of hamburger menu and then that's where like cross sales going to be or anything like that. You can go back to user research and say, see, here's a UX test that we ran where customers were able to more easily access information through this kind of menu versus the other. So either going back to data or established research and having that ready when you first pitch is going to address those complaints before they come up and it's gonna make a stronger pitch in general for any of the projects that you're going to do. If you don't know why you're gonna be doing something just grab someone, right? Grab someone and say, hey, listen, you're uninvested in this project. Like start asking me questions about why we're doing this and I'm going to take notes and if I can't answer them that indicates maybe that's a place that I should spend more time researching. Awesome, that was a great answer, thank you. So we have a question here from Brian. Where do you see DraftKings going five years from now and what's the main reason I should switch from FanDuel to DraftKings as a consumer? I think that's a great question. I think that where we're going clearly if you take a look at the industry in New Jersey right now, you can see that sports betting is becoming larger and larger. You can see that the public attitudes around betting has shifted substantially. Certainly it's become a much more public topic, consumer sentiment and in fact moral and ethical sentiment towards it has been shifting as well. I think I saw some industry research showing that like 77% of people in Mississippi I think in Mississippi were okay with betting on sports. That's a huge change from just a couple of decades ago in the United States. And I think that if you take a look at the industry as a whole providing integrated apps and integrated experiences between all your products and especially between all your sports products is super important. I think that what makes a difference between DraftKings and FanDuel in particular and I think FanDuel is a fantastic product and I think they have a wonderful team over there is just that DraftKings we have a team internally that is intensely committed to sports and intensely committed to living the kind of culture of sports betting and daily fantasy sports that I think makes the business very special and I think honestly makes the product that we put in extremely special on my life turned on. But I think that that's number one for me and I think that FanDuel has that as well. I don't think that, you know I think they're super passionate and committed but I think that our team is just, you know they just do exceptional work sometimes and I would trust that they're gonna continue doing awesome work for sports fans in the future. Awesome, thank you. Our next question is, are there types of products where gamification wouldn't work? Why? That's a great question. So I really dislike the term gamification because I feel like there's a lot of baggage behind it but I use it because it's understood. There are a lot of things where I don't think it makes a lot of sense. A lot of times the straightforward actions of a customer like I want to do X right now. Like I don't want someone gamifying bill paying necessarily. You know, it's a transactional thing for a customer, right? A customer says, I just wanna pay my bills man. I don't wanna have to worry about like achievements that I have to accomplish or whatever. And in many cases like business to business products it could be, I don't think it's necessarily appropriate to gamify that because you have a sort of peer to peer relationship, right? It's not a business to consumer situation it's more of a peer to peer situation where you're talking to fellow experts and fellow and fellow, I guess you have peers is still the right word in the industry or in your industry or even in other industries and they're not gonna want you to try to say like and if you sign up for the deluxe posting package you get a cool achievement badge. That's just not something that's necessarily needed for those kind of transactions. I think it's really appropriate in business to consumer interactions and I think it's super appropriate when you're trying to engender behaviors within a customer, right? So when you're trying to encourage a customer to take actions and behaviors where I'm most excited is when it comes to health and wellness apps. There's lots of behavioral evidence and psychological evidence that you can retrain your brain by instituting new habits, right? By saying for the next six to eight weeks I'm going to do X, Y and Z and that's going to actually change how my everyday habits or mental health affects me. It's super effective there. When it comes trying to change player behavior or comes time to give them things to do like establishing goals, it's super effective. When it's sort of transactional or peer-based I don't think it's nearly as effective. I think you could try it. I think that there's room for experimentation there but it wouldn't be my first place that I would go to. Awesome, great. Thanks for that answer. So I think that wraps up all of our questions here in the comments. So thank you so much for your time today, Alex. It was a pleasure having you on the channel. Great, thank you so much. I appreciate it. And everyone, if you have any questions you can always reach out to product school and they can put you in touch with me and thank you so much for having me. Yeah, no problem. So before we leave I want to give our audience some more information so they have the resources to become a product manager. Our product management, product leadership, coding, data analytics, digital marketing and UX design courses are taught by industry experts working at companies like Google and Facebook. And in addition to that, we offer weekly events every Wednesday and Thursday. So make sure you check out our website at productschool.com. You can also find us on social media at product school and be sure to keep up with the latest product management content at theproductlog at productschool.com. So thank you all for joining. Thank you all for joining. Enjoy the rest of your day and I hope to see you next week. Thank you so much, Alex. Have a great day.