 Good morning, everybody. So if you were in my talk yesterday, there was a bunch of technical issues. Figured out what those were, and today, we're all set. So today's talk is the experimentation mindset. Real quick, I am Doc Norton, CEO of CTO2. We do agile and leadership coaching for companies where software is key critical. You can find me anywhere as Doc on Dev, the only place that's not available to me was Google Plus because I have to use my real name. And yet, they allowed me to use Doc Norton. Doc is a nickname, but whatever. Hashtag, agile India 2016, the hashtag for this talk is experimentation mindset, which is just long enough to mean you've got to truncate your tweets a little bit more. And of course, I said I'm Doc on Dev. So it's going to do it right away. Let's talk about what is a mindset, exactly. I think we need to set the stage so we understand what are we talking about here. How many of you are familiar with the work of Carol Dweck? A few. All right, so I'm going to tell you a little bit about some of the work that Dweck has been doing around mindsets. And I'll tell you about an experiment that they ran. So what they did was they took this class of students, and I don't remember if they were fourth graders, fifth graders, but they took this class of students and they gave them a test, a math test, I think, that was a bit of a stretch, but not too much. So if they were early in the year, this is the type of material they'd probably be learning about mid-year, right? So they had enough to get through it, but not enough to have actually mastered this particular topic. Gave all the students the test, had them hand it back in. Then they randomly split those students into two groups. And what they did, when they handed back the tests, they said one of two things to these students. So if you're in group A, they said one thing to you, and if you're in group B, they said another. It was a small distinction between those two sentences, and it had a huge impact. The students that were in group A when asked if they would be interested in taking an even more challenging test, said yes, absolutely. I would be interested in that. And as they continued to treat these students in this way, reconfirming the message that they delivered to them throughout the grading period, these students' grades went up, they started mentoring other students, they started taking leadership roles in class. Group B, on the other hand, when asked, would you be interested in taking a test that's a little bit more of a challenge? They declined that opportunity. They'd rather take the same test as they continued to be basically treated the same way throughout the grading period, their grades went down. They became less and less interested in what was going on at school. They weren't taking on leadership or mentorship type roles. That difference, extremely subtle. For group A, when they handed back the test, they said to each and every single student, you did well on this, you must have worked hard. For group B, when they handed back the test, they said, you did well on this, you must be really smart. The difference is a growth or a fixed mindset. What they did was they instilled in these students a simple idea. If you did well because you worked hard, you were in control of the outcome. If you worked harder, you could do better. If you did well because you were smart, it was a fortuitous outcome that you had no control over. And if you continued to challenge yourself, you would discover the boundaries of your intellect and you would start to fail. This simple distinction created a massive difference in the way that these students operated. Now, this same study, this same experiment has been done over and over again, not just in schools, but with CEOs, the professional sports players. So it's not just about children, it's also about adults. The way that we perceive our abilities, even if that's injected into us, changes the way that we actually behave. Our mindset has a significant impact on the way that we behave. Our mindset is the ideas and attitudes with which a person approaches a situation. So we're talking about learning. Let's talk a little bit about kind of how it is that we learn, the stages of learning, if you will. And I'm gonna use the Dreyfus model for skills acquisition to talk about that. How many of you are familiar with the Dreyfus model? A few, okay. So the version that I'm showing here is just five stages. This is the original kind of writeup of the Dreyfus model. There are later versions that have six and even seven stages, but for the purpose of today's talk, five is gonna be sufficient for us. So we look at this model, you start off in anything that you're doing, in any endeavor, anything that you're trying to learn, you start off basically as a novice. And as a novice, you have little knowledge or experience. That makes sense, that's why you're a novice. In order to learn, you need rules. Think about if you have played a musical instrument, the first musical instrument you ever learned, when you started off, they probably showed you, you put your fingers here like this, it makes this note. There's no explanation of why. You don't get music theory. You don't get any of the other things behind that. You just get rote, do this like this, you will get this result. Good. So you end up focusing on how, over why, at this period in your learning. And you need a mentor or someone that is kind of close monitoring what it is that you're doing. But eventually, you move into being an advanced beginner. Now as an advanced beginner, you actually have some experience, right? And you've got the mechanics down. Now what's happening here is, you're starting to expand your knowledge, but you're not sure where to look for it. And even when you find information or when information is given to you, you're not quite ready yet to discern, is this valuable, is it not, is it relevant, is it not, is it important, is it not? Right, so everything is information for you, but you're not sure what's really valuable or what isn't valuable. At this point, you need experience in kind of limited and controlled real world situations. So we wanna go beyond, you know, press this key, play this, right? To, you know, here's, learn to play this particular piece of music in its entirety. You move into competence. Now you've actually got a mental model. Now you're actually starting to be able to do things. You can actually maybe write some of your own music. It's maybe a little bit simplistic, you can actually do this. You're starting to get to a point where you feel much more comfortable with this thing, right? Associations are formed. You can actually handle things that are unknown. You're still pretty methodical. You're still kind of following the, I'm supposed to do it this way, way of doing things, but it's starting to feel more comfortable. Eventually we move into proficiency. Now we're starting to get interested in the big picture, right? At this point, we can actually grasp and apply maxims. We get kind of ideas that early on didn't make sense to us. So if you're a software developer and you've been following kind of XP extreme programming, in the beginning somebody might have said something to you like Yadney, right? You ain't gonna need it. But what does that really mean? In the beginning you haven't any idea what that really means. You're trusting somebody else's judgment. At this point, you understand what that means. You can actually apply that concept, right? Now to move to the next stage from proficiency into expert's expertise, you need a lot of practice. And that practice needs to be unencumbered. It cannot be hindered. So it needs to be hindered as little as possible by policies, guidelines, rules. This is the point where you've really gotta just start noodling with that instrument and figure out for yourself what feels right, what sounds right, to be able to get to that next level of expert. At this point, you're a true authority, right? You've actually developed an intuition. You really understand this deeply. You've got a deep pool of knowledge that you can draw from and you can actually like interlink skills, right? But here's the interesting thing about early stage experts is they tend to be inarticulate at how they arrive at conclusions. One of the challenges when you get into early expertise is all of that stuff that you learned in the very beginning kind of seems too simplistic. And as you're making these very interesting conclusions, you can't explain to people why or how you got there, which means that experts, early stage experts, are often the worst possible teacher you can have. They know a lot of stuff, but they can't explain it. And at this point, you just continue to practice and you can actually grow your learning through teaching as you get better at it, right? So I want you to just thinking about this, the Dreyfus Model Skills Acquisition, like these five different stages. How many of you have been in your current vocation, your current role, or something similar to that for about five years or more? Okay, so for those of you looking at this continuum, in that specific vocation, in that specific skill, because this model applies to everything that you learn. So you can actually be at different stages for different roles in your life, right? But for this specific role, where do you feel like you are? You don't have to shout it out or anything, you can keep that internal, right? But just think about it, where do you feel like you are? Five years of experience been doing this. Now, the average answer is, well, I'm an expert. The reality is, in our work, most of us stay at proficient. Now, we may hit a very high level of proficiency, high enough that most people can't discern our ability from that of an expert. We haven't actually crossed that cascades, right? If you think about music, there are people who are amazing at playing all kinds of pieces. But if you gave them a piece that they had never heard before, they cannot anticipate where it is that the music is going to go. They can't play along with the new piece. An expert can actually do that. So why is this? Especially at work, why is this true? Why do we stay at proficiency? Well, it turns out that these beginning stages are all about implementation. They're all about doing the thing. Press the note like this, learn this thing. Did you do it right? Good job. Implementation, the implementation mindset, is about getting it right. It's about earning that grade. It's about knowing what the right answer is. And by the way, there is a right answer. There is a best practice for this. That's how we understand things. All of our tests, all of our exams in school, and then when we get to work even, there's a way of doing things. But to get from implementation to that next stage, we need experimentation. Remember, I said you can't get from proficiency to expertise unless you can practice unencumbered by the rules and guidelines. The experimentation mindset is about exploring different ways, but in most companies. We put rules and guidelines and procedures in place to protect the organization from the early versions of ourselves, from the novices, from the advanced beginners. But as people move up in their abilities, those rules and guidelines don't go away. And so we actually can't progress into expertise. So I'm gonna tell you a little bit about maybe a typical company. My own kind of personal experience with this. So when I was much younger, still in college, I was looking for a way to make some additional money. And I'd heard that there was a job that I could work kind of odd hours, whatever. I was in, I got into telephone sales. Now, I'm not that old. Okay, admittedly I am that old. So telephone sales. Me and my buddy Tommy decided that we actually wanted to get into this. And we were selling magazine subscriptions to total strangers over the phone. And back then there weren't computer systems or anything else. So you'd come in every day and you'd have a stack of cards that was on your desk. And you'd pull the first card off and it had individuals' name, their phone number and what their prior subscription to whatever magazine was. And then what your target cell was for them, right? And in our first couple of days we went through orientation. And they said, this is super easy guys. Everyone's gonna do great at this. Everyone does great at this because we have developed the perfect process for phone sales. We know exactly how this is supposed to work. In fact, we have a script for this. Now I said, man, this is gonna be great. And one of the ways I knew it was gonna be great was because it told me so three times in the script. So all you had to do was follow the script and you were going to be successful. So for my first week, that's exactly what I did. I picked up a card and I dialed that number and I followed the script word for word. It wasn't that great. What you're looking at here is the weekly sales ranking. You might notice that doc, me, is right below this red line. What does that mean? Well it turns out that if you were in the bottom 20% of sales, you were below the red line. If you were below the red line for three weeks in a row, you were out. If in a seven week period you were below the red line four times, you were out. And this red line wasn't some sales target. This was a strict rank and yank. It was you against everybody else in the company, every other salesperson. You could continue to improve. You could continue to get better and still get let go because they were top grading the organization. I didn't want to get fired from this job. I'd have to go back to delivering pizzas on the weekends and my car barely worked. So I thought about it for a while. I said, all right, how do I get better at this? What am I gonna do? So I looked at the list and the top of the list is this guy named Ptolemy. Well if you check out Ptolemy's desk, he's got little salesman of the month trophies. And then you notice you look on the wall and there's all these plaques, top and sales. Ptolemy, Ptolemy, Ptolemy, somebody else, Ptolemy, Ptolemy, Ptolemy, right? Salesman of the year, like this dude is the top salesman in the place. So you want to get good at this job, what are you gonna do? You watch the master. And I know what Ptolemy's gonna do because I know what the script is. Ptolemy's gonna say hello sir or madam. My name is Ptolemy Michael Acakas. That's not what he does at all. He says good day, my name is Pat Michaels. Ptolemy's a liar. What is this? Okay man, help me out. What's going on? Like the script is what we're supposed to do. This is how we get good at this. This is the best way to do this. Thomas says yeah, yeah, I know, I know, script, script. Look, half the time, I can't figure out if it's a mantel or a woman on the other end of the phone. And I gotta take a guess. And I'm gonna offend him. So if I just say good day, it doesn't matter what time zone they're in. It doesn't matter if they're male or female. I've said something pleasant. This was quite some time ago. Ptolemy discovered, most unfortunately, that if he said his name was Ptolemy Michael Acakas. The phone calls ended much sooner than if he said his name was Pat Michaels. Turns out we like to talk to people who kind of feel like us. And most of our sales were being done in the Midwestern United States where everybody's named Pat Michaels. So I thought all right, I can try this, right? I can do this. So we'll just go off script a little bit. So I went ahead and I started going off script. I tried a bunch of different things. I tried being kind of the jokester sort of dude. I tried being the forceful sales guy. I tried playing dumb. And most of the time I just tried to be sort of a nice guy. After a couple of weeks of kind of noodling around and figuring it out for myself, bam, top in sales. I'm gonna get a trophy, not following the script. So fast forward a couple more weeks. Now I'm eight weeks in. My video's gone. We got video guys? It's not a very interesting talk. So I'm eight weeks in and it's time for my review. I really need the visual for this. There we go. Almost. And I'm thinking, all right. Going to the manager's office, right? Eight weeks. First we couldn't do so good. Got to be top in sales for a while now. I've got some advice probably for these guys. You know, I mean, I'll hear what they had to say, but I've got a few suggestions for how we can improve some things around here. Starting with the copy. I really need the visual for this. Unfortunately, it didn't go at all the way that I expected. I walked in, I sat down and I got a below expectations rating. Top in sales, below expectations. You see, Mr. Norton, we don't only monitor how you're doing on your sales. We listen in on your phone calls. We record those phone calls and we audit them after the fact. And we have noticed that you do not follow the script. I don't know if we made it clear in orientation, but a lot of really important people put a lot of time into creating that script. It is our best practices for sales. I would like you to reflect on this for a moment and I would like you to think about how much better you would be doing if you would simply follow the script. As you can imagine, I decided that maybe this wasn't the place for me. So on a Friday, at the end of the work shift, I pulled my last card, I looked down at it and I knew then and there that this was my last moment. This was my last call. So I picked up the phone, I started to dial the number and I stood up on my chair. And as the phone was ringing, I stood up on my desk. I was standing on my desk in the middle of a call center, phone to ear, ready for Mr. Jones to buy one more year of field and stream. My friend Tommy thought it was pretty funny and the next thing I know, he scrambled to the top of the desk. And then another and then another. And there were probably a half a dozen of us standing on our desks at the end of the day on Friday, making that last call. I probably should have told them it was literally my last call. I finished that call. If you remember, I was supposed to sell one year. I sold three. Best practices in almost all cases are a misnomer. The very notion of a best practice is steeped in a fixed and implementation mindset. They create artificial boundaries for our growth and for our learning. That implementation mindset, right? They wanted me to get it right. They wanted me to follow the script to practice the script and get better at that script. The script was the best practice and I was violating it. This is what Chris Argyris calls single loop learning. Many, many, many organizations follow this. We hold fast these kind of underlying assumptions that we started with, such as the notion of some best practice, right? And we operate like that. We actually prioritize and decide how good someone is doing based on whether or not they're following the practice. If we do not get the outcomes that we desire, our inclination is to just get better at doing that thing. The problem with this is it leaves little opportunity for radical improvement. And with this focus on getting it right, it creates an environment where failure is actually hidden. We want to look good. We fear failure. And as we all know, failure, what we need is double loop learning. In double loop learning, what's happening is yes, most of the time we're trying to get better at that practice, at that thing. But every once in a while, we stop and we step back and we take a look at it. We say, wait a minute, if this isn't actually getting better, maybe it's our assumptions. Maybe that best practice isn't a best practice. Maybe the way that we actually achieve this end is through some other means, let's try that. So if you're on a team today, you're concerned with code quality. Someone might come in and say, oh, this is easy. Make everybody pair all of the time. Code quality will go up. And that might work really well in your environment. I've been in many environments where that works great. It's actually my preferred way. But I've been in environments where what actually works really well for them is formal code reviews. But it works for them. Or maybe you treat the actual repo more like it's open source and people issue pull requests and there's a review process and it gets a certain number of shipments or thumbs up and then it gets rolled in. What is the end we're actually trying to achieve? And what are the multiple ways that we could possibly get there? Double loop learning. That is fundamentally the experimentation mindset, exploring different ways. So I want you to think about this phrase best practices. It's probably used quite a bit in your organization. I want you to start just a teeny tiny little revolution. Anytime you're talking about best practices, I want you to say leading practices. Small change can make a big difference. Best practices are the best. That's why they have that word in front of them, right? If it's the best, you can't do better. If it's the leading practice, it's the best that we know of, right? But there's a chance that there's something else out there that actually would work better for us. I want to revisit this really quickly. Because I talked about how in these early stages it's all about implementation, get it right, but then to get to actual expertise, you need experimentation. It's not actually true. Our educational system is primarily founded on an industrialized period, where what we needed was people who could take simple instruction, get it right, comply, follow rules. It actually turns out that what we're seeing and have known for quite some time in reality is that you can get through every single one of these stages with experimentation and the outcomes are actually better. So we're seeing more and more schools, whether they're charter schools or specialized, Montessori, whatever it might be, where we're really changing the way that we're actually educating children and realizing that it's actually working much, much better. We're talking a little bit about experimentation as I've seen it in companies these days. So a simple one I think is kind of obvious is a lot of companies doing split testing or AB testing. If you have a large enough client base and you have enough transactions, you can actually do multivariate testing. The challenge with multivariate is if you don't have enough volume of activity, it can take too long to get to statistical significance and you won't know which of the things is actually the right one to go with. You can get false positives early on. So AB is basically when you test two different versions of let's say a screen against one another. You try the version with blue buttons and the version with green buttons or whatever it is that you might want to do. Multivariate is when you actually are testing multiple versions of that thing against itself all at the same time. Tracking code quality. Some of you are seeing a lot of companies now as employee engagement stuff. And I'm seeing companies experiment with this and try and figure out what actually works for them. What actually matters to them, to their people. What does an engagement survey look like that is consistent with our organizational values and views, our mores? At Groupon, when I was there, we actually started an engagement survey. We modeled it after the G12. Gartner. Sorry, I was gonna say Gallup and I knew that was wrong. So Gartner has a very extensive study that's been done for years and years and years and they've basically distilled all of these hundreds of questions down to 12 key questions that are indicators of employee engagement satisfaction. And they kind of work in almost a hierarchy of needs sort of way. But we built an entire system around that and what we did was we sent out surveys that were anonymized, the results were rolled up given to managers and they could actually see how it was that they were doing, how they were doing against the rest of the organization. And we also gave them like, look, this is where we think you should be scoring and here's where you actually are. If you're low in this area, here are resources that are available to you so that you can actually learn more about how you can help your employees feel like their opinion matters. And actually I don't like that phrasing, feeling like your opinion matters. That sounds kind of manipulative and whatnot. Here's how you can help create an environment where your employees' opinions matter. Our goal is not to make people feel like a thing. Our goal is to actually have that thing happen. Another one that kind of extends on this is team joy. We have done a few different things. I've worked with a couple different companies around this. Some of them are actually measuring joy is kind of a, can be a difficult thing to do. So in some cases it was surveys or micro-surveys. In one organization what we actually did was in Git. So they used Git as their version control system. We wrote some porcelain, which is just basically a way to extend Git. When you check in the code, they already had a standard that said when I check in the code, I put the story number or the ticket number and then some description, right? So we just altered the standard so you put the ticket number and then after that in brackets, you put a number from zero to five and then after that you put your description. That number is zero to five, zero, I hate this code, why am I here, five? This is awesome, we should open source it. Now the question we were asking the developers was their opinion of the code, but it turned out to be a very good proxy for their satisfaction with their work. Right now if we'd asked them how they felt, if we'd actually used the F word, I don't know what kind of results we would have got. We actually did some experimentation with an application that asks you a couple of times a day, how do you feel on a scale of like one to five, right? And over time we started noticing that it really had a lot more to do with what had happened in the moment right before they took the survey and not so much what had happened in the prior two hours. So by actually moving it closer to the activities that you're working on, by moving it closer to when you actually commit the code, we were getting, we felt we were getting better results. And then kind of taking all of this and even looking at, sorry, and even looking at like process design. So is this working for us, is this not working for us? And the truth of the matter is, for most organizations I don't care what the process is, what you call it, right? What I really care about is are you evaluating whether or not this is working for you and how are you actually measuring that? And how do you actually know that it is or isn't working? I see oftentimes teams that will do things like, hey, you know what, we should TDD more. Yeah, yeah, we should TDD more. All right, for the next two weeks, we're all gonna do that more. Great, come to the end of the two weeks. How are we doing? We're TDDing more. Are we doing good? We are doing good based on what? Like what is TDD providing of value? Are we, is it good because we're just doing it? Hey, we should all punch each other in the face more. Two weeks later, are we doing it? Yeah, right? Like, what value is it actually bringing? I do, however, I say I don't care what the process is, but I do, however, just want to take a little segue because I think it's really interesting. Anybody recognize this one right here? Yeah, yeah, that's a cutie, isn't it? So how many of you are familiar with Royce's paper, the actual write-up on this? Yeah, Dr. Royce is actually the, where the paper actually, sorry, is where kind of this phrasing came from and this idea came from. So it's interesting about that paper. Let's just kind of zoom in on this thing for a second. What's interesting about this paper, and this is actually Dr. Royce, this diagram appears, this I believe is diagram two in the paper. In the paragraph following the diagram, Royce says, I believe in this concept, but the implementation described as risky and it invites failure. This is like page two of an eight-page paper. He then goes on to say, so to mitigate this risk and to ideally avoid the failure, we should be doing some other things. Oh, I don't know, like prototyping and having feedback loops and communicating with each other across boundaries and the last diagram in the paper looks like this, which has an awful lot of documentation in it. He really advocated for a lot of documentation, but I understand, it was like 1954, so how else are you gonna do this, right? But if you actually turn this thing on its side, it looks very similar to what we kind of look at today, right? So I just think it's interesting, an entire methodology based on a paper that says, don't do that. All right, so back to Agile, right? How many of you know this guy? Woody Zool, or no of this guy? You don't have to know him personally, right? So Woody has recently become, recently is in like the last few years, has become one of my Agile heroes. And the reason is, and we talked about it actually a little bit this morning, Woody has come up with a couple of different things. He was in charge at Hunter Labs for a number of years and they experimented with things. They tried new ways of doing things. And out of that came the no estimates movement. How many of you are familiar with no estimates? Yeah, so Woody was one of the first to actually talk about this and how they were doing it and what was happening with it. If you look at this movement, there's a couple of different views, et cetera. Doesn't matter to me. The thing is that they were actually experimenting, trying new things, and then they were openly sharing what it was that they were doing, that experimentation is what got them there. The other thing that came out of Hunter, and we heard about it this morning because apparently Muresh hates it, is mob programming, right? And this is one of these things like, when you're early on, and you've been in software for a little bit, and then someone tells you about pair programming. You're like, what? Two people at one machine? That's a total waste. I mean, how does that even work? And then you hear about mob programming, eight people all on one machine? Impossible, it can't possibly work. Yeah, so let's make sure we not try it and talk about how much it sucks. Orientation and onboarding. Some of the most innovative and successful orientation and onboarding programs that I've actually seen have come from experimentation. So at Groupon, we actually had, some teams had their own onboarding programs, other teams didn't. So there's a general orientation for the whole organization, but there was no engineering onboarding. And so some teams were doing different things, kind of realized this was going on. Those folks got together, they started talking about what works for them, what doesn't work for them. And out of that came a volunteer group that pulled kind of all of the successful things that were happening there. They started to create a generalized onboarding for engineering and they actually experimented with it, got feedback from the people, people that went through onboarding three months after you had gone through, they asked you, did you feel it was of value? What now do you know you could have used and they adjusted it? And then after a year, they asked them the same question. So they weren't just looking at like, you leave onboarding and we ask you, how'd it go? Interest Leagues is a thing that we did a lot of experimentation with. I've actually put these into a number of organizations now. And this is the same kind of thing. What Interest Leagues are is basically communities of practice, but they're not so formal. They're not someone decided that you have to, we now have a community of practice for Java developers and everyone with this title must attend on these days. And by the way, bring your own lunch, right? These are people who are interested in a topic, getting together because they want to and talking about that topic. And they last for as long as they need to last and then they go away if they're not supposed to be there. But I've seen leagues in organizations, a speakers league that started off with just, hey, I speak publicly, I could use some help on my slides and it turned into an entire system where they were actually helping to develop the public speaking strategy for the entire organization, et cetera. Hackathons are another great place for experimentation. Sorry, I'm rolling through this quick now. Mentorship programs. Again, I've seen a number of mentorship programs where what was happening was, mentoring was happening on some teams, but not all. And actually pulling together what's working, what's not working, how are we doing and creating small localized experiments. And as we figured out what actually worked well, then growing that out, right? And I've seen mentorship programs that don't look anything like you would expect them to look, but it's because they figured out that it worked right for that organization. In mentorship, not necessarily. And the other thing is that in many organizations, you'll find that you actually have people internally that have done this and kind of know something about it. But in one, they ended up creating a very kind of formal program and they got very, very low participation. In another, it was a much more informal, hey, these folks are available, if you need help, here's a Slack channel. If you want private help, come to this group, they'll find someone to pair you with and that one worked super well. So someone experienced in creating mentoring programs may have actually structured something that didn't work for that environment. The experimentation is what really helped us get there. Companies that are doing a lot of experimentation, Valve, I think I talked with them in my other talk, and I think folks are pretty familiar, Gabe Newell, actually there's no managers there because managers institutionalize procedure and that's the last thing they want. They want people to continue to push the boundaries and keep thinking. Buffer, what I like about these guys is they have a very, very open environment and they're continuing to experiment with that openness. So they're open about pay, they're open about all kinds of stuff. You can see anybody in the organization's salary, but they ran an experiment where they actually did a completely open feedback system. So you could give redirecting or reinforcing feedback to anyone in the organization in absolutely public forum and it flopped. It turned out that nobody wanted to actually criticize anyone else in such a public way in a permanent record and so they made an adjustment and if you want to give somebody kudos publicly, you can do that, but if you want to actually give someone some critical feedback, that's a one-on-one kind of thing. They experimented with it to see what actually worked, to see what would work for them. The open pay, they still have. Yes, so the only thing that they've experimented with a bunch of different open, let's just be totally transparent, with the actual feedback a couple years ago, they went to a very open feedback system and found and they're like, yeah, it was consistent with our company culture, right? This is perfect, this is what'll work for us and it didn't work, so they made adjustments. Oh, Spotify, sorry. So Spotify I think we kind of know of their model, right? And they've been continuing. Here's the thing that's interesting about Spotify. They've been continuing to experiment and adjust and so every week I get someone that contacts me and says, hey, you know, I know you kind of do organizational stuff. We want to do something in our company and we're looking at the Spotify model and it looks really great and that's what we want to do. That's not even what they do. We're mistaking the end result as the model, the snapshot in time as the model. That's not the model. How they got there is the model. Dig deep, look at that, that's what need to be followed. All right, so wrapping up, I got basically there's four things I want you to remember as you kind of come away from this, right? The first one is know your purpose and challenge your assumptions. So we think about that double loop learning, know why you're doing a thing and then challenge the assumptions that you made about how you're supposed to do it. So I have Tesla Motors up here because not too long ago, they open sourced the majority of their patents and the reason that they did it was they looked at, they said, you know what, what we really want to do in the world is we want to lower the carbon footprint. And we will never scale to the size where we're going to make every single car in the world. It's just not going to happen. If we really want to lower the carbon footprint, we have to make this available to everyone so that all auto manufacturers can do these things and can make it better. So they knew their purpose and they challenged some of their base assumptions and they opened all their patents up. Next thing is make failure acceptable. It may not make sense to you. It sounds like it doesn't, it isn't right, but when failure is acceptable, success is more likely. Acceptable? So that's again the implementation versus experimentation, right? The implementation mindset is get it right. So if you experiment, if you run an experiment and you don't get the results that you want, you've failed, right? In some environments that's punishable. In some environments you don't get the bonus or you don't get whatever the reward is, right? You don't get an advancement. But what you're doing then is you're creating an environment where people understand that failure is not acceptable and so they're not going to try in the first place. Think big, but start small. If you want to do a mentoring program and you want it corporate wide, don't launch a corporate wide mentoring program and then complain because people aren't complying and following it. Start small, start with a couple of teams. See what works, see what doesn't work and evolve it from there. It will scale fast enough. The last one is to be intolerant of mediocrity. This is actually a core value at Groupon. It was, of their core values, the only point that I really felt like, yeah, just keep experimenting, keep trying new things. 10 minutes for questions or I'm 10 minutes over. Be intolerant of mediocrity, what do I mean by that? They are basically clicking on a different level altogether, that's what I'm thinking about. So for me, all right, so one of the things I think is interesting is in a lot of organizations, when we have the annual review, right? What do we do in those annual reviews? Say, hey, you did this well, you did that well, good job, thanks a lot. And here's three things that you need to actually be improving upon. What we're talking about typically is, here's your strengths, that's nice. Here's your weaknesses. I need you to do something about this. What we're actually doing there is we're managing towards mediocrity. What we're saying is don't put so much time into those strengths, those are good enough. But let's take these weaker things and let's move them up a little bit. Our idea is that we're improving, but actually what ends up happening is the skill atrophies, wow, these other skills don't really improve all that much, we're actually managing people towards the middle, towards mediocrity. Wow, that was so nice. Being intolerant of mediocrity is just basically saying, look, expect that things aren't always gonna work, but look for new ideas, look for exciting things to happen and they will happen. If compliance is the number one thing in the organization, in the organization, you're not gonna get those new ideas, you're not gonna get, and that compliance is fundamentally mediocrity. The compliance is mediocrity. So can I safely assume that when you said that more focus should not be put on those areas where you are weak, but more focus should be put on those areas where your strength lies, is what you're trying to say? I believe that, I believe that what we should be doing is focusing on individual strengths and we should be building teams where those strengths are coming together and we're augmenting one another. Also great. So when you've got those five people together and each of them brings a different unique strength, now we've got an amazing team. Thanks a lot. Other questions? That's it, we're out of time? All right.