 Awesome. Hi everyone. Welcome to this week's product school talk. Happy new year. I'm glad to be back today. We have a special guest with us. His name is be back Betty. He's the product manager for learn best. So hi, be back. How are you doing today? Good. How are you. Great. Thanks for joining us. I wanted to get started by giving everybody a an overview of your background. So can you talk about more how you broke into product management where you get started. Sure. So I was actually a double major in computer engineering computer science from workers University in New Jersey. So actually broke into product from the tech background. I was over at Goldman Sachs for about 13 years. Started my career on the tech side and moved to product did some stints at some startups and now I'm at learn best slash Northwestern mutual. Thanks. Well I know you have a presentation prepared for everyone today so I'll give you a couple seconds to set that up and start screen sharing. And guys, whenever his presentation is over of course we're going to take questions so you can type those in the comment section right underneath the live video. So just make sure you're all set and ready to go and then we'll get started. Great. Looks like I'm ready. Awesome. And the floor is yours. Great. Thank you everybody. Thank you, Cassandra. So I wanted to talk today about disrupting digital product cultures. It's actually a topic that I've been pretty passionate about in my current role at learn best slash Northwestern mutual. A little bit about culture. You know, just thinking about culture in general, it really, you know, many companies have different cultures. If you think about Apple with their culture, or even IKEA with low cost furniture, every company has different cultures. But what does culture really mean, right? And what I really think of it as how we do things around here, right, and how are we perceived internally and externally. That's what really a lot of companies are striving for is changing the way that they do things and, you know, how they're perceived and that's really become what the poster model of what culture has really become in a company. Now, changing culture is not easy. Right. It doesn't take a message from the CEO. It rarely comes from top down. It's really all about precision and sensitive nudges. You can't really convince people to change. Small changes are really what really makes the big difference. I relate this to when my actually I was getting my dad to use the iPhone, right. I told him that everybody's using iPhones. This is the smartphone technology and he was resistant to change because it was happy with his phone. But then when we got on Facebook and he wanted to see pictures of the grandkids, he came to realize that these subtle nudges and now he actually uses an iPhone. He sent me a Snapchat request and I'm not even on Snapchat. So, you know, it's really about subtle changes and small changes really make the different when you think about changing culture. It's a lot like fashion in many ways. Not that I should be talking about fashion, but, you know, if a new trend starts, it's really because of word of mouth, right. Something spreads through magazines, through circles, through celebrities, and then it cascades out. Again, it wasn't something that was kind of dictated from the top and it really spread through osmosis and through the community. There's six levels of culture that we personally use here. A fewer formal, fewer informal leadership. It's important. Again, it can't be a message from top down. You have to walk the walk. You have to talk the talk when it comes to culture. A great example is Mark Zuckerberg. Noam Well, right? He wears or anyone at Apple where the way that they dress in executive meetings and the way that they, you know, perceive themselves. That's a culture that they walk when they're in their leadership roles. Second, I'd say is clear roles. Many times the confusion and culture just boils down to, what is my role? What am I supposed to be doing? If we clearly define the roles of the various product managers, engineers, development leads, and what their roles are, it really makes for the culture migration a lot smoother and reward people, right. Obviously compensation is important, but recognition, awards, any way that you can kind of recognize people for adopting culture or some of the, you know, successes they've had really helps with the culture movement. Some of the more informal ones that are important to me is role models. You know, we have a team called our product specialist team here. Their job is literally to go out and act it, do it, right. Go back and show how we're working in a different way when it comes to our advisors. There's 8,000 in the field, by the way, or home office employees, and really kind of bringing that together. Networks, you've got to build networks and you've got to be open to change, right. Changing behavior is important as well. So how does this relate to us at LearnVest and Northwestern Mutual? Two years ago, this was our Northwestern Mutual digital experience, okay. Information overload. There was a lot of information thrown at you. It was difficult to understand. You actually needed your advisor to be with you, right, to actually explain what's happening on our digital experience. Critical experiences were missing. We didn't have a mobile presence. Not shocking for the life insurance industry, but there was no mobile presence. You had to be an actual Northwestern Mutual client to log in to any experience. If you think about companies like Mint and LearnVest and BetterMint, they all have free accounts that you can access. There was no sharing tools amongst the advisor as well as the customer and self-service was a pain, right. I remember my first, when I took on the role, it was Friday night. I don't know why I was doing this on a Friday night, but I wanted to log into my Northwestern Mutual account. I've been a policy holder for five years through my Goldman days. And I wanted to log in and I forgot my password, right. And I couldn't figure out how to log in. So a message said, call 8am Monday morning, central time to reset your password, right. In today's age, a lot of password resets, logins, changing the frequency of your bill pay. A lot of that was missing two years ago. I'm proud to say here's where we are now, right. And it's really been what we just crossed one million users using the digital experience. You can see the design, it's clean, it's friendly. We have a mobile presence and we're engaging with our clients day in and day out. As you can see here, there's a lot of areas where we've really improved it from the account aggregations, linking your external assets. We have insurance pages now that really provide details of how things are working and what do they mean for you, investment pages, really bringing in market commentary, podcasts, videos. A lot of that to really kind of educate the user on what's happening. NorthwesternMutual.com, our branding website, we recently launched that as well. It's much more consumer friendly. You'll see beautiful marketing imagery. An interesting stat here that I like to remind folks is about two months ago before we did the relaunch of our brand website. About 63% of our folks that came to the website left after one click, right. Now we have about 163% increase of folks are staying. They're staying on the site. They're consistently looking up information. And we have a mobile presence. We have Android. We have iOS. And you can do a lot of the things that you can do on the website on the mobile app as well. As well as our login registration, remember my story about not being available to reset my login. Now we have the ability to do that. We actually have text confirmations happening as well. So how did we do all that? That's a drastic change in two years if you think about it, right. Quite a bit has changed in our experience. And you know, a lot of it has been just kind of historically Northwestern Mutual as well as many big companies have been a project management orientated discipline, right. And you guys are probably familiar with this where you take requirements, you go back to your engineering team and then months, maybe even at times quarters and sometimes even a year later, you come back with the solution. So it's been very orientated where it's scheduling organization and increments. And that's the waterfall approach, right. We all know it well. We came from that discipline at some point in our career. What we decided to do is actually move more of a product approach where products sits right between the consumer, the business and the digital. And you know, looking at the product principles of being okay with failing, iterating, moving faster, really doing all of that has helped us kind of move forward and make things happen. And we iterate. We put out releases every day. A fun fact, I think I have a slide on this coming up. We went in 2015 from 85 releases to 170 in 2016 to last year to 1400 releases. That's exponential growth, right. And it's really because of our okay with being okay with the ability to fail and iterate and move faster. And you know, what it's really led to is if we kept on moving the way we were moving in the waterfall approach, yeah, eventually we would have delivered our solution, but the business and market would have changed. With our new approach, we take smaller steps and we iterate. Sometimes we're not on course and sometimes we're not even sure where the ship's going, but we keep moving and we actually end up where the opportunity actually will be tomorrow. So it's been, you know, very fascinating to see how that's come to fruition. So how are all these products built right talking about culture a little bit. First thing I'd say is, I took on the role at learn vest about two years ago running your product team, and we were acquired by Northwestern Mutual. So we are also acting as kind of their digital arm as well. Half of my team is in Milwaukee, Wisconsin, which is where Northwestern Mutuals headquartered and half the team is here in New York. And it's been very interesting bringing those two cultures together right as you can imagine in a startup besides having beer on tap and wearing jeans in New York. So the culture has been very much let's put out something fast right let's whiteboard it and let's develop and deploy it that night as opposed to Northwestern Mutual. It's been methodical right let's make sure we're doing the right thing let's get signed off their stakeholders involved, and let's make sure there's reputational risk right that accounted for so bringing those two cultures together and managing a team across of that that has been very fascinating for me right. One of the things that we've done as we blended these teams together is take the best of both I don't actually think the right answers either the right answers a little bit of both and we've come to learn that the first thing we did was she we changed the way we work. I remember my first day at Northwestern Mutual I asked who owns mobile, someone raised their hand and said yeah I kind of do but I wait on her and she waits on him. And what the immediately what came to my mind is we need clear ownership, we need small pizza pie teams you probably have heard the concept but pizza pie team shouldn't be bigger than a pizza pie boxer to conserve. And we've really moved to that, and really kind of helping break through the process. We have about 20 pizza pie teams today on the client experience team we have another 20 on our other planning experience team. They're a combination of a product manager and engineering lead a full stack engineering team a designer and a business partner. This has been very important and key. We've actually brought the business into our pizza pie teams which has been fascinating, because it really kind of reduces from overhead when it comes to going back and forth meetings and organizations. So product managers aren't actually necessary quite a bit in our new model, having these pizza pie teams has really kind of changed the way that we work, and really, you know moved it forward these some of these teams are blended right, the product manager might be in New York and the development team is in Milwaukee or vice versa. And we've really figured out a great cadence on how they're going to work together. They're having stand ups and they're, you know, talking to each other day in and day out and they work on two weeks friends. Every two weeks, something is put out. It could be a big release or it could be a combination of smaller releases but we've gotten into the cadence that we're going to put releases of features out. We changed our approach to write member I shared the story of gather requirements go away and then come back sometime later with the solution that's not what we do anymore what we do is we understand the business we spend time learning the problem. We research research research. We have an entire product research team that's focused on getting feedback. We look at data and analytics obviously for but for me equally important was balancing qualitative and quantitative. We spent a lot of time interviewing getting feedback from our financial advisors and our clients. We've created two communities. One is called the digital experience lab it's about 500 digitally savvy advisors. We show them wires and you know, tweaks of things before we release them to get their feedback. Also, we've included about 4000 clients where we interview them get them give promotions and find out how they feel about things before release so very tapped into research. So first understand what the issue is then really deep dive into the research and then break up the project. This is very key. In order for us to maintain these 1400 plus releases this year. It was important to break up the project into small bits and bytes so we can get to that two weeks cadence of putting something out over and over and then we design something. Get more feedback. Right. So we're not developing anything yet and it's really kind of geared at getting more feedback. Are we thinking about this the right way. We want to make sure what we're about to deploy or engineer still fits the business model as you guys can attest to the businesses evolving every day. So we want to make sure it's still relevant. So we get that feedback again from our clients stakeholders we tweak redesign deploy test and off you go we deploy and then we start the whole process again right. So I think with a lot of what we're doing with our pizza pie teams the work is never done. It's really kind of taking it to the next point and making it more. Taking into fruition. The last thing I'd say is we changed the way we've released a lot more show and tell. I got the best compliment that I ever got in my 17 years at an event that I was doing with some advisors. We they after the event we're presenting a lot of what we're doing in the mobile space. One of the advisors came to me and said that was a great show. Right. And that was really telling to me what I instill a day in and day out in the team is forget the PowerPoints really show show what you're about to release even if it's early phases get the feedback. If it's a button or if it's a big feature get that feedback early. So we've been doing a lot more show and tell instead of presenting and it's been very valuable and kind of taking us to the next All said and done this number is already outdated I'm sorry but yeah we went from 89 releases to 177 we just broke 1400 this year right 1405 to be exact so bringing two of these cultures together New York Milwaukee big company small start up has really really worked for us right and it's really kind of brought together the best of both worlds and taking us to that aspect. I also have a case study here that I gave in another presentation that I can also send over to the team but as you can see a lot has happened right a lot has changed what I'd say we've come really far we have crossed a million users. We have a mobile presence now we have digital loan repayment from a self service standpoint you can change your password. Yay. Your people are linking their external accounts and honestly we're just getting started. You know I was just talking to the team today and we're starting the plan for what our next 10 pizza pie teams are going to be doing as they move into next year. So that's what I have I'm happy to take any questions that anyone might have on how we've come together and two companies and made this culture splash and Awesome thanks so much feedback great presentation I know we have quite a few questions I'm sure that are going to come in soon if you could talk a little bit more about Any advice that you have as far as people that are trying to break into product management now and and what what resources you think would be useful for them. Yeah sure I mean I think product school is obviously amazing when it comes to You know breaking into products I think breaking into product is actually interesting because it's a catch 22 a lot of companies are looking for product managers that have two to three year experience. You never go to university or school for product management you actually come in through different disciplines. That's why I think companies like product school or you know just getting out there and finding a mentor. I'm You can find me on LinkedIn I'm open to always coffee chats and I do that quite a bit. I think it's really building that network and opening those doors relationships relationships relationships. I think that's important please work your network other networks to find opportunities to get the foot in the door. What I'd say about some of the skill sets that I look for in a product manager. I generally skew towards someone that has a technical background that doesn't necessarily mean you have to be a developer doesn't mean you have to code means you have to at least understand the technology. It's important for a product manager to understand the technology and also defend it. Second I'd say an amazing storyteller for me being a good product manager means you can tell the story right tell the story of yourself your products and all that that's involved. And third obviously the basics right being able to kind of write the user stories and all the things that you would learn in any schooling or curriculum and the fourth one I'd say which is the most important to me is being a balance of strategic and tactical. You know it's always good to have a roadmap and a strategy but we also have to really think about how to be tactical and get things done right that's really remove us towards that north star so those are some of the things I look for and those are some of the resources I would tap into. Awesome thank you. And as yeah as predicted we have quite a few questions so I'm going to take this first one from Andrew so VVAC do you prototype and test designs before any code gets written. Yes, so we prototype and we create wires. Usually I skipped through it very early in the presentations but they're usually black or whites right and we'll spend quite a bit of time with our those communities I talked about the 4000 clients to 3500 advisors and really kind of nitpick and make sure a lot of that makes sense before we move into development and then it's a cyclical process right once we move to the next iteration. While we're developing we're also thinking about version two of the black and whites and eventually it moves to color so along the entire process prototyping and development is going hand in hand. Okay. This next question is from Steven. How do you line all of the releases or team focuses so they relate to the key company goals. Yes, that's a good one someone actually just asked me this today we have 40 pizza pie teams how do you make sure that they're all. Do marching towards the same way right. So what we do is I'm not a big believer in process to be honest with you I think all 40 pizza pie teams have their own cadence right. We don't besides the fact that we mandate that we use a certain set of tools to communicate with each other. If you want to meet with your team four times a week great. As long as you and your dev leave had figured that if you want to meet once a week that's great right as so the way that the interaction between the teams happen is really and some of the backlog items of what they're thinking about is always really empowered by the team. Now what we do this is me personally I spend one meeting every two weeks and I get all the pizza pie team pms as well as the dev leads in a room right and this is not a meeting to present to anybody in the team about what they're working on to get by and etc. It's literally a we go around the room and every single person has about three to five minutes we actually have a timer and we go through what are the next releases coming in the next 30 days from a feature standpoint and what are some of your dependencies. And what's been eye opening in that meeting is we'll find pizza pie product manager for team a hadn't talked to pizza pie product manager for team L and they're both working on billing one for mobile one for web. Great that's a dependency that we've pinpointed in this meeting we're not going to spend time solving that here you two can go off now and kind of think about how to tackle that. And what I've noticed in that type of setup is we pinpointed so many dependencies and the team feels so valued that the fact that we're using the time wisely right we're not trying to use that time there to sell it and we're not solve it. So that's one of the approaches we've been using. Okay. Great. Let's see we have so many. Theodore. Okay so from this is from theater. I'm assuming the Phillip philosophical change from waterfall to agile wasn't an overnight shift and required a lot of like coaching champion along the way so can you speak to what strategies or tactics help you bring change to the way people thought about product development. Yeah I mean hey that is very it's a great point it was easy right it took quite a bit of time and in two years we've come quite a long way. You know I think a lot of it so I'll touch a minute on the things that I thought weren't easy and the hardest things for me personally was the cultural differences right. I think if you think about somebody who's driving a car 100 miles per hour versus someone who's driving a 10 miles per hour one's more careful. One's more dangerous one's moving faster one's moving slower. So there's a lot of that that we had to think that through right and the answer wasn't either car was driving right. The answer was to drive 60 right so we actually spent quite a bit of time thinking about examples. You know thinking about areas where does this you know I talk very quickly about the 40 pizza pie teams but a lot of it was matchmaking. It was like dating a lot of times a product manager and the development lead just didn't get along they were both great but they weren't the right fit for each other. So it was a lot of moving people around finding the right fit getting the company in the organization OK with the risk of not failing and possibly not doing the right thing right. Going out and building these communities a lot of you know trial and error so it wasn't easy and there's no you know prescriptive answer I can give you a lot of this is more of an art and a science. We spent quite a bit of time and kind of analyzing and spending that I'll tell you a lot of my time in the last two years actually hasn't been product development and it's really been kind of relationship management and looking at the organization and finding those sticky points that make sense. I'm not easy but whoever asked that question happy to think I can give you a long answer offline. OK great thing. Now this next one is from me Vita. OK. How do you do integration testing or combined testing when you have multiple teams working on the same project. Yeah so we actually have I didn't talk about this but we've embedded testing and QA into our pizza pie teams too. Right. So that's actually made that quite a bit. So if you look at our pizza pie team there's a product manager development lead engineers and developers design and QA and integration testers as well. Right. And that's actually made things quite a bit very streamlined. So right there and also the business partner I'm sorry. And it's very much streamlined the process because now while that QA representative or that tester or that business partner or that designer doesn't report into product when it comes to the organization. They're aware they have you know beat the feet on the streets of what's happening in that pizza pie team. So as they're going back to their leadership they're bringing back all of the projects that are happening on each piece of pie team and integration and testing is happening across. The product manager then kind of steps in and make sure through the testing cycle that we're looking at a day to day analysis of how things are happening and we're moving it forward. Okay we have time for probably two more questions so I'm going to take this next one from William. How do you share findings or activities from your research team with the delivery pizza teams. Yeah so the research team is not embedded into the pizza pie teams. That's actually an auxiliary function that we have. So what happens is that the product manager's job is to work with our research teams and are we have a few to be honest with you, and they're the ones that are helping draft. So remember we talked about the black and white wire frames they're the ones that are going with an idea and saying I think I want to do this and I don't not sure if it will resonate. Can you help me find the right people that will go and figure out if this is the right thing for us to do. So we'll have you'll use user testing dot com from general population will use our field will use our clients will help with our research team come up with the right prescriptive questions a lot of it are clickable prototypes where they can answer yes no like don't like what would like to see this feature I haven't we thought of this and all of that then comes back to our research team and they then put that together and almost present it back to the product manager as here's my findings right and then the product manager and the research analyst go back and forth they might do a few iterations of the testing until the product manager feels comfortable this is something we want to do right so that interaction is very tight because the research team sits within products and that helps. Okay. Um, here let's see we have another one. Okay. So one. Okay, this one here from sitter. How do you measure the success of a product and what would be your process for prioritizing features on your product roadmap. And along with that what's the biggest challenge of a product manager so a few questions there. Okay. Let me see if I can get the first one. Okay, so how do you measure the success of a product and what would be your process for prioritizing features on your product roadmap. Okay, how do I measure success. I think as a product manager to measure success is, you know, I threw up numbers, I threw up releases. For me, it's a lot about customer satisfaction. Besides the behind every pizza pie team there's a client there's a user right. How well are they satisfied with the work that we're doing and that can be measured with a satisfaction score or numbers or money that we're bringing in for that client right if it's a sales organization, or frankly just releases right. So I think a lot of product managers really need to think about how are we measuring success and equate it back to what is the satisfaction that the customer is using. That's why it's important to have the business partners in there because the offer every business partner if you're where if your business partners and operations oriented person versus a sales oriented person. Those metrics will be different and I think it's very important for the product manager to spend time understanding what is success for the business. So that's part a I think Part B remind me what was Part B. What would be your process for prioritizing features on your product roadmap. Yes, so I think the process for prioritizing for us is really kind of geared at data feedback ROI. Is this going to move the needle is this going to help more business right and that's why I think bringing that business partner in has really been helpful because sometimes we'll have ideas. Sometimes we'll come from the business and sometimes we'll actually combination of them are the right one or neither of them are right. So spending a lot of time doing use case studies we do scenarios and that's why some to the earlier question do we prototype before we go into development. Yeah, we actually even do analysis of even if we should move to prototype before we go to development. We don't want to waste development resources on something that might not be a good fit. So we spent quite a bit of time prioritizing by running scenarios and use cases. And I think the third part was what is the toughest part for the product manager right. The toughest part I've noticed for anyone on my team or even me as a product manager has been navigating and a conference room when there's differing opinions and it's your call right. So you'll have someone in one the marketing team asking for a and the tech team saying we can do a and then the operations seem saying why don't we do be and they're all looking to you to help really kind of steer that ship. And I think that's a really difficult responsibility and it's nothing you can read in a book and I think that really comes with practice on stakeholder management and really kind of bringing people together to really move the needle forward right. And I think that's a really hard aspect of product that you know takes quite a bit of time and practice to get to. Right, absolutely. Well that's all the time we have for today so thanks so much to be back for a great presentation and getting to all those questions I appreciate you being here today. Thank you very much. And thanks everyone who joined us. If you guys want more information you can find us at product school calm and we host these webinars every Thursday, as well as events at all of our campuses so thanks again everyone and have a great day.