 Good morning. It's it's really nice to be here in those were just fabulous opening comments and I sort of couldn't be teed up much better so as mentioned my training is as an economist and mathematician and When I started doing my research I got really interested in this field of what was called complex systems Which is really starting in the 1990s and complex systems are things like economies the brain ecosystems things like that and there's a group of us Mathematicians physicists the economist theoretical biologist who are studying complex systems and one of the things we found In our research was that diversity makes these systems more innovative more robust more interesting has been what I was mentioning and There was sort of this disconnect as he as he mentioned introduction when I would talk to organizations and businesses when you mentioned diversity There was sort of a feel-good social justice Angle to this as opposed to sort of a bottom-line performance angle So what I want to do today is I want to talk about Research some of it by own but most of it done by sort of a group of scholars That really talks about really just sort of the bottom-line benefits of diversity So the outline is I'm going to start out by talking about complexity a little bit I want to formally mean define what we mean by complexity because if You go to a decent restaurant. They'll decide they'll define the pasta is complex So I want to sort of make sure we nail that down Then what I want to do is I'm going to talk about two specific tasks that are relevant to the bank one is prediction and the other is problem-solving and what we'll do is we'll talk about why diversity actually is as important as ability in those contexts and then I'll There's a little bit of data and then talk about how we may be practice Leveraging diversity and then we'll hopefully have some time for some questions So to frame this a little bit and just sort of juxtapose The bottom-line notion of diversity with a sort of social justice notion I want to use a quote by Wendell Berry which says we have been wrong We must change our lives so that it's possible to live by the contrary assumption that what is good for the world is good for us So why do I begin with this quote? I begin with this quote because I think when we talk about diversity and inclusion We tend to think of this as the right thing to do And what I'm going to do is argue that it's not only the right thing to do But it's also the better thing to do and that's why we have to sort of change our minds So a wonderful quote with regard to this is by Astro Teller who runs Google X which is Google's sort of innovative out there You know self-driving car space and he says look people throw on the word diversity like it's a tip at a restaurant But what you really want if you're trying to be a cutting-edge innovative place is people who think in different ways Right, so that's gonna be the key to think about this And I have a new book coming out in August called the diversity bonus and the reason I put this out There's I have a previous book called the difference Which is a book about sort of how diversity makes things better and this was a book written when I was a professor at Caltech And it's a very mathematical book and I was at the IMF a couple years ago and somebody introduced me And I said this is a really good airplane book And I was sort of surprised that he said that and then he paused and he said if you're flying to Singapore Because it's really kind of tough sledding right because it wasn't really written as a business book This book is written more as a book for practitioners and it'll come out in August But my take on this and the way I come at this is much more like the way a standard economist would do it and what I do is I I come at this by sort of writing simple models to try and organize how we think about things and I think the theory is incredibly important in this particular space because It's not the case that all diversity necessarily makes us better This is something that has to be managed and understood we have to think about How do we get the right types of diversity in the room and how do we manage those things? So if we don't have a theory if we don't have an understanding of how diversity makes us better We're gonna end up with what my sons have two teenage sons They describe as a dog's breakfast, which is basically just dump a bunch of stuff in the bowl and hope somebody will eat it So let's talk quickly about complexity. So they get global financial networks They're much more connected than they were in the past if you look at how we make policy decisions in banks So this is a Chart from the World Bank But what it shows is it shows the different US banks prior to the bailout and these numbers that you see between things like this 456 and this 94 And this is Lehman Brothers not Lehman sisters here, right? These the size of the numbers show sort of how correlated stock prices are in the tails And so a big number here means if something bad is happening to AIG that means something bad will happen to Merrill Lynch so when you looked at sort of This network what you see is it's not just that we can draw lines between these different financial institutions It's that some of these lines are stronger than others and when we think about a particular financial You know entity we can't just think in terms of its own balance sheet We have to think in terms of how it affects something else So when you talk about complexity what we mean is we mean things that are connected in sort of deep ways So a couple years ago is involved in a project to try and figure out what's causing obesity and we solved it There it is. It's the simple solution now if you really obviously you can't read this That's on purpose if you drill down what you see there's different colors in here So for media social economic food activity infrastructure So everything from lack of sidewalks to large Coca-Cola's everything you can think of is sort of in this graph But the point is is that no one we look at the big picture. No one could possibly understand obesity We now have so much knowledge so much information We recognize there's so much complexity in the world that one person can't do anything anymore, right? If you really want to sort of solve obesity understand it you need teams of experts in order to do it So when we say something's complex, what we mean is that it's between ordered and random. It's not simple It's not completely random or another way we put it is to we say that it's deep It's something that's difficult to explain engineer, right or predict or evolve, right? So the brain is very difficult to explain. It was extremely difficult to evolve Right, and if you tried to engineer a brain that would be difficult So we would therefore say the brain is complex, right? We wouldn't say a coffee table is complex Because it's none of those things So if you look at economic phenomena, some of them are complex and some of them aren't so oil production isn't that complex, right? It pretty much grows in a linear fashion with economic growth oil prices because you can have inventories You can have tankers just cycling outside of ports waiting for the prices to go up or down, right? Those are complex So what we want to do is we want to say some things are easy some things are hard and some of the things that are hard that we want to use diversity So here's what I'm going to do and take two tasks The first one's going to be predicting or forecasting second one's going to be problem solving and we'll talk about why diversity is beneficial in each So I want to start with then I was mentioning group think I want to start with a book written by a friend of mine called the Wisdom of crowds how many have seen this book anyone okay, so they came out about a decade ago and Jim So wiki who writes this book he begins with this sort of amazing story of the 1906 west of England fat stock and poultry exhibition There's 787 people who guess the weight of a steer and the winner gets some prize like a cake or a pie or something Right, so there's all these guesses about the way to the steering the average guess is 1197 pounds Take 500 kilos The steer weighs 1198 pounds, which is amazing and this is a great way to begin a book But it's just an anecdote So what we'd like to do is move from anecdote to a real science of understanding like what is it that made this crowd smart? You can prove the following theorem You can show that if people are making numerical predictions that the crowd's error Equals the average error the people in the crowd well that makes sense You think the crowd is about as good as the average person in it But it turns out the crowd is actually better than the average person in it The crowd area equals the average error minus the diversity now. This is a real theorem. This isn't like a Management book theorem that like teamwork equals thoroughness plus effort plus ability plus meaning or something now This is like real math. Okay, so this is it's like the Pythagorean theorem. This is a mathematical identity It's always true any time you have a group of people make numerical predictions This will be true. So this C is the crowd's prediction. This is the truth This is just the square difference between the crowd and the truth This is the average of the people in the group So the crowd error you might think equals the average, but it doesn't the crowd There is actually less than the average and the amount by which it's less is these differences in the prediction So in Goulton's case, here's the actual data You know Goulton's been dead for a hundred years. So he happily shared the data, right? people don't always do that and So the crowd is off by less than a pound the average error Was about 70 pounds because these are squared errors So the point was these people weren't savants who could just guess the weight of a steer They weren't some sort of like geniuses from the west of England The reason the crowd got it right is they were diverse But this is the general thing if a problem is easy. This is where complexity comes in if the problem is easy Average error is going to be small, right? So the crowd can be smart without diversity But if the problem is hard Average error has to be big so the way you create a smart crowd is by having the crowd be diverse. Okay So when I talked to I spend a lot of time talking to the New York Fed and recently I've been doing some things with Some larger companies on Wall Street and one of the main things I try and get across is when they put together a portfolio of assets They think about having diversity, right? That's just one of the key things, right? You want a diverse portfolio of assets But when I have a diverse portfolio of assets, the reason I do it is to minimize risk The return I get on that portfolio is equal to the average return on those investments, right? So if one investment like the yellow investment does really well and the blue investment does badly, I get the average of those two investments With a group of predictors, I just showed you the theorem you actually get better than the average Right, that's a that's a mathematical fact. It's a mathematical fact that a portfolio you get the average payoff It's a mathematical fact with a group of predictors. You get better than the average So as diverse as your portfolio is in terms of stocks Your group of predictors should actually be more diverse, right? Because you're getting this sort of bonus hence the reason for the title of my book you get a bonus from this diversity Right, so diversity actually makes you better Okay, so that was people guessing the weight of steers a hundred years ago So you might say Interesting, but maybe not kind of where we're at now in an age of big data and hyper sophisticated models So let me jump to the completely other extreme. Let me jump to sort of what Was the most sophisticated case of collective prediction sort of this is most sophisticated contest ever run this is in 2006 Netflix Corporation released all of their data on The people who watch their movies So if you go to Netflix and you watch a show it'll give you a it'll give you sort of a gas in terms of how much You're gonna like that show. So if I type in avatar It'll say five stars and the reason it says five stars is because I have two teenage sons And this is they sort of have figured out this is what my family probably likes and it'll say five stars if you know Then was a French intellectual, right? So if he types in avastar, it may say two stars, right? Because it would be beneath him maybe to watch a show if Anybody so far as I know anyone types in rush hour three is a horrible movie It'll say one star, right? So that's a really easy movie to predict So here it is six years of data half million users 18,000 movies Largest data set ever publicly released and after they released it the US Supreme Court said you cannot release data like this So it may end up being the largest data set ever publicly released It's no it's no longer legal to do this and what they said is if anybody can beat their internal program So Netflix had this internal algorithm to predict movies. They said if anybody can beat us by 10 percent We'll give you a million dollars So what happens is and there's here's where diversity comes in So what happens is all these teams and this is these are the sort of the best and brightest Computer scientists machine learning people in the world They construct these predictive models where movies have attributes and then people have weights across those attributes What do I mean? So an attribute might be how much money didn't make in the box office, right? How many weeks was in the theater? Is it an action movie? How many days has it been available on CD also anything you can think of and Where diversity comes in is these teams had to sit around and think about what are the attributes of a movie? What are the things that might matter and then of those things that are matter Which ones can we scrape off the web because you've got to be able to you can't like code these in by hand for 18,000 movies So you got to have some way of like using reviews or using available data to scrape all this information on the med We haven't throw it into your model So the early leader in this is Belkorn. This is going to be the star of the show this guy Dr. Robert Bell who's the chief data scientist at AT&T Research Park and their team their team was called Belkorn They had 50 dimensions for each movie So he had a whole team of people working on this and this is kind of amazing So when you go home tonight try and think of 50 dimensions for a movie, right? So this is pretty impressive their best model could be to buy 6.8 percent Which is pretty good. They could so they 10% they win the million dollars six dirt 6.8 percent Now what they did which is interesting is they would like have a whole bunch of different models Where like these are the different data and they would sort of give different models different pieces of data And then to kind of average them now the reason they did this is you want to sort of Avoid overfitting and also one of the things sort of state-of-the-art in machine learning isn't to create one Supermodel that it's great a whole ensemble of models that you then average up because otherwise you end up Overfitting the data and finding patterns that don't really exist The other thing you do is you use this technique that's called boosting which is really interesting So the first thing you do is called bagging where you just sort of give give these algorithms different data So that they have so that they're diverse kind of like giving people different life experiences It's sort of like making computers like people the second thing they do is these use this technique called boosting Which attempts to create diversity. So here's what happens suppose like I've blue things are good and Positive ratings and red things are these negatives are negative ratings So you might have one model that says everything over here is good everything over here is bad and another model It says everything over here is good everything over here is bad what you see is like these things with the pluses, right? you're sort of getting wrong and So what you'd like to do then is like put more weight on these things because you're sort of confused one model saying they're negative one Model saying they're positive and so what you do then is you say okay? I'd like to get these ones right and these ones right and so then basically put more weight on these particular occurrences And you train something to figure that out. So what these boosting algorithms do is they actually train algorithms to be diverse relative to the other algorithms So what bell core does then instead of just having one model they actually create a hundred and seven different models None of them are as good as the best model, but by combining these models they can get to eight point four percent What's making them better? What's making them better is the fact that the models are diverse, right? They're systematically making diverse models So the story's fascinating. This is AT&T research park and if you read the media reports on this They're completely misleading because they're talking about these people trying to win a million dollars Okay, this is AT&T research park dr. Bell his budget is in the hundreds of millions of dollars This had nothing to do with winning a million dollars. They said everything to do with winning Right, this was a whole bunch of people competing to see who is the best data scientist? You know who's to has the best data science team in the world. This wasn't about the million dollars They were giving the million dollars to charity. Nobody cared about the million dollars Everybody cared about winning But they're two years in to this contest and they can't get to ten percent And so dr. Bell's boss calls him in and says, you know what? It's been great that you've been doing this and you've got a lot of publicity from you winning the contest But it's over you've got to stop doing this because you've got to go you have this other thing that we like to call a job and We would like you to go back to your job So he says fine fine. I'll just win this thing You know cue the chariots of fire soundtrack I'm just gonna win it right now and his boss is like, okay How exactly are you gonna do that because you've been doing this for two years and you haven't got to ten percent? How will you do it now? And he said well easy. I'm just gonna bring in some people who aren't as smart as me And his boss is like, well, who would that be and he said well anybody right because like my team is the best We've figured that out. I just need to bring in some people who are diverse Right, and we've got this theorem that says if you can bring in people who aren't as good as him or diverse They'll do better Right, so they bring in these Austrians called big chaos who aren't very good at figuring out attributes for movies But they're really good at figuring out how you weight the different models So they bring that them in they almost get to ten percent right and they do this fancy thing Or you remember I showed you that simple summation sign They take derivatives and stuff in order to figure out what these weights are using something called a Crestman weight function They still don't get to ten percent of so then they look at this team in third place fourth place And they actually take this Canadian team in sixth place who they bring in and these Canadians Weren't that good at waiting, but they're really good at figuring out funky attributes to movies Okay, so they had some these behavioral insights That never occurred to the people at Bell Labs, and this is the big one Okay, so this is a Will Ferrell movie and this is a movie called snakes on a plane I don't know if anybody's ever seen either of these movies, but here's the thing about these movies that the Canadians figured out They're a lot like margaritas And that is like right after you've had one it seems pretty good If you wait until the next morning, it doesn't seem so good So it turns out there's a whole bunch of movies that if you rate them right after you watch them You give them five stars if you wait a couple days You're like oh that was a stupid movie, and you give it two stars So there was these sort of this snakes on a plane Will Ferrell category that the Canadians figured out that had a huge effect Right, and so the same movie could get five stars or one star depending on when you rate it So they put all these teams together these three teams together now they have 800 variables for each movie, which is insane Right their best model is now 8.4 percent So there's just this bonus from this diversity because now they've got all these different variables They can get to 8.4 percent when they combine the models they get to 10 percent so they win right So they've got two different types of bonuses here one bonus is just more variables to choose from the other bonuses You combine a bunch of models so they win except for they don't win This is what's so great because the contest rules say once you get to 10 percent the contest ends in 30 days Their reaction was pure panic at this point. The reason why is here's what they knew There were 23 teams from 30 countries that they had been beating for two years But prior to this if you're an economist these other teams had no reason to work together now when there's 30 days to go They've got no reason not to so they all send all their data to Berkeley and Penn these are countries from all over the world and They combine 48 of these models hundreds and hundreds of models are sent they combine 48 of them And they take the lead this group called the ensemble takes the lead with a day to go Right, they eventually Both submit and it's a tie Belcore officially beats him on the fifth decimal point, but the contest only goes to four decimal points So the tiebreaker is who submitted first Belcore submitted 22 minutes sooner So fortunately the New Jersey public school has got the money All right, which is great, but here's the beauty of this whole contest the ensemble The people combining these models were graduate students So if you open up a thesaurus and you look up graduate student the first synonym is snarky person Right, so these graduate students on the ensembles web page when this is over they say here are the final results It's official Belcore takes second That's literally what their web page said and they're absolutely right because let's think through it I've got two models that are equally good Right so the same average error, but they're wildly diverse Right one is combining 48 models and around the globe One is complaining combining 17 models using 800 variables from three teams, so they're wildly different So if I have two models that are equally good that are diverse what has to be better Combining them Has to be it's a mathematical fact. It's like five squared plus 12 squared equals 13 squared if I make a right triangle Right this just has to be true So the challenge here when we think about Leveraging diversity is thinking about how do we get people who think differently in the room and then it's been almost mentioning How do we make them feel safe so that they share their different models because if we do we're collectively going to do better, right? So this isn't a Feel-good thing. This is just a mathematical fact now You could say these are people picking movies, right? And this is Neil Hunt the chief product officer of Netflix because this kind of cascaded on them They thought initially this was going to be we're going to find the person who's the best data scientist And instead diversity broke out, right? They didn't anticipate this at all He's like wow this turned out to be all about combining diverse ideas Not in some like touchy-feely sort of diversity is a happy thing But like the only way we could solve this problem is by putting diverse minds to it so if you look this is data from our Central Bank the Federal Reserve System and this is a regression where they compare We have something called the Federal Open Market Committee and this staff forecast if you take the 600 economists to make individual Forecast and you're on a regression on inflation just a simple OLS regression And then you say how much weight should be put on the average of the staff forecast or how much weight should we put in the FOMC Forecast do you get a weight of one on the staff forecast than effectively zero in the FMOC if you get unemployment You get a weight of one or point nine seven on the staff and minus point oh three in the FMOC on growth You actually get a little bit more weight in the FMOC that on the staff The point here is is that you've got all these in-house staff forecast They're just put in a book and they kind of read them you'd actually do a lot better just averaging the staff forecast So there's Using technology right and this was done by Christina Romer who is Obama's chief economic advisor Because she sort of believed that they were just ignoring a lot of information Okay, if you look at economic forecast in the European Union in the United States over a 40-year period So this is every base. This is 28,000 forecast by professional economists including the ones that the ECB puts out, right? But you're sort of team of economists the crowd so we know it's true The crowd has to do better than the average economist. They do 21% better Across all the state 21% better Okay, I mean mathematically. It's got to be better, but it doesn't have to be 21% better And here's the data this again. This is sort of some of these studies. There's 40 economists predicting some of these studies There's 12 economists predicting Here's a benchmark of one for just an average economist So one an average economist has an error of one if you just take two average economists you do 8% better And if you take three random economists, do you do 12% better? Here's the best economist to date Here's after the fact if you knew the best to come so let's say here's the best to date She gets 11% better if you average in the second best economist to date with the best economist to date you do 17% better Now this should be somewhat counter-intuitive right because it's like silly is the best economist I'm second best you would think let's not even listen to him Right and that would be the normal way you'd think about it But the reality is if my model is different than hers you should listen to me Because on a high-dimensional complex hard problem if we have different models that diversity helps you out Right and what you see is you see increases sort of all the way down to five six and seven, okay What about real people investing money? Well, this is a study of every single mutual fund This is an unpublished paper, but I refereed it seems right so I'll put the graph up But that's why the names aren't up here But hopefully this will become public information in a couple weeks If you look at the number of managers on mutual funds in the United States So this is 10,000 funds over a 20-year period They do 60 basis points better than funds run by one person and you can think well If that's true funds should no longer be run by one person and guess what that's true So 30 years ago 75% of funds United States were run by one person now 75% of funds are run by teams of people Right why 60 basis points, right? That's not very complicated math Okay, so the point is if you want it if you want to make an accurate prediction What you want to do is you want diverse crowds, okay? Let's go to problem-solving. This will be much quicker So here's a simple test suppose I've got a really hard problem and problem-solving difference in prediction and prediction I'm trying to come up with a numerical value for something estimate some future thing in problem solving I'm trying to find some solution to a problem like maybe some way to route my trucks Some way to design a computer chip, but I've got a bunch of variables that I'm choosing among and I want to try and find something That has really high value So what you can do is create a bunch of agents. This is what economists call people, right and Think of them as having diverse perspectives Which are sort of representations of this problem and then they've got sets of heuristics Which are methods they have of trying to solve them and all these agents are smart So now I create a group of the 20 best agents the best individuals and I compare them to a Random 20 agents all the agents are smart So I'm taking the very best ones versus a random one and I have them just kind of work collectively right on this problem So I can think of this as the following I've got one group of people who are really really good and one group of people who some of them are good and some of them who aren't that good right and What you'd think in this setting is that the group of people who are good should be better than the group of people Some of whom are good and some of them aren't that good it turns out though If you sort of do the math on this that the diverse group almost always outperforms the other group if you sort of Reasonable size groups like groups of size 10 or 20 The group of high ability does better if you have groups of size one because that means you've got the best person against a random person Right, but once you start having in 10 or 20 people this flips And here's why thinking of people in terms of ability on the problem kind of misses the point Right, that's kind of your score. What you wanted is you want to ask what tools do they have? Well, suppose this is an economic problem Maybe that all of these people have econ PhDs and they've taken a lot of statistics and mathematics So the third person doesn't know anything that the second person and the first person doesn't know So if I got this alpha group if I think in terms of toolboxes and Ted of scores I've got a whole bunch of people with that exact same skill set Right Where's this diverse group? I may have someone who's actually studied political science Somebody knows linguistics someone who's taken sociology someone in psychology So Andy Haldine who's the chief economist of the Bank of England? He was in Ann Arbor a few months ago and he was talking about how the Bank of England is now hiring Psychologists and I'm like wow is it 2017 already, right? I mean the point is right you want to think about really enlarging right who you bring in to discuss these policy things And Andy's great and he's really been a leader in trying to think about how do you diversify? The skill sets of the people who work in your research team Now the thing is that model this toolbox model is a simplification When there's a famous quote by George box that says all models are wrong And so I'd the paper the monologue just showed you where diverse groups do better than random groups was written by myself And Lou Hong is a mathematical economist at Loyola University in Chicago So I was visiting Cornell and there's a guy John Kleinberg is a canal who's much smarter than I'm he's a Certified genius by the MacArthur Foundation, right? He's won a genius award. He's a brilliant guy So he decided to look back at the my paper with Lou and rewrite a different model Sort of like if a statistician wrote the model so he went out on a pretend I'm a statistician right the same model so our model was we've got perspectives and heuristics We have people sort of coming up with representations of the world, right? So what some of our agents might have used Cartesian coordinates like x and y and some would use polar coordinates right with the theta and r stuff like that and What we showed is that like a people who sort of use represent problems the same way and solve it the same way gets stuck at the same points It's the formula. We call that sort of getting stuck at the same local optima So in our case what had to be true is the problem solvers had to be smart You had to be drawing from a diverse set of people and the key condition was the problem had to be hard So when we figured out what was true it was Smart people who are diverse hard problem, right? So John's model what she wrote with my Thorego as a grad student of his they said here's our model's totally different We're gonna assume that people are just distributions of solutions. So if I ask Celia What do you think she just dumps her ideas on the table? I dump my ideas on the table and then we sort of like Combine them or do something with them and come up with some best team idea And so the team value is then gonna be some sort of function of all the things we dump on the table That's their model So you can think of them this sort of a function of the following sort so you could have a linear function or an averaging function where you just sort of like average the ideas if that's the case if it's a Simple problem then the best team consists of the best people But as soon as it becomes nonlinear as soon as you do something interesting with the ideas that throw in the table Then the best team doesn't consist of the best individuals. So they get the exact same result that we get Completely different model But the exact same result and the exact same result is as soon as the problem becomes complex The best team doesn't consist of the best individuals Why because the best individuals tend to be similar and what you really want on hard problems is diversity Now what they go on to show and this is why Kleinberg is a genius and I'm not is that Because he uses words like sub-modular and super modular that just means nonlinear But what they basically show is this is that not only does the best team not consist of the best people on that problem There's no test that you can come up with Such that the people who score highest on that test Are the best team? So there's no single hiring criteria you can use That determines the best team Now there's a third model that Lu Hong has that basically shows This holds generically for any function that's nonlinear in a broader class of problems So as soon as it becomes nonlinear There's no test Right what you can't use the same test and then pick the best people what you need to do is you need to have diverse criteria Okay, so again this portfolio of asset thing doesn't make any sense Right because portfolio we get the average and the groups of problem solvers. We actually typically get better than the best Right, so if I'm in a room and I have crazy ideas most of the time You can ignore them most of the time and if everyone's in a lot of a great idea Then you get to take my great idea that rare moment when I have the great idea, right? I don't get averaged in otherwise. I would be like investing in Kodak Right where you'd lose all your money because my average return might be very very low Right, but with a group of problem solvers, you don't care about average return You care about the best and you care about how can we combine and interweave best answers And so again if you think just sort of at a heuristic level What that suggests is you really want to have a lot of diversity in the room Okay, let me do a quick aside on identity diversity except I've been talking entirely about cognitive diversity and see we talk about some of this as well But how does identity diversity relate to this right because when we talk about diversity we think about where I started We think about things like gender race You know culture sexual orientation that sort of stuff and here's the funky thing and this is work by Catherine Phillips This is a survey article, but if you survey hundreds and hundreds of articles What you find is diverse groups Based on identity are actually more creative or innovative than groups that are homogeneous, right and Catherine and I are good friends and so we like to joke about this She says she's basically when I just spent the last 30 minutes telling you is kind of obvious That if you have cognitive diversity you should do better, but what's less obvious is that identity diversity actually makes you better and The story behind this kind of has to go as follows right it has to be the case that identity diversity somehow maps to Cognitive diversity which leads to better outcomes. So the operative thing has to be cognitive diversity. The question is how does identity diversity? Get you to this cognitive diversity. This isn't very complicated either, but what you've got it is you've got to kind of unpack it So the stories the mathematical models aka stories that I've told you about how diversity makes you better Boiled down to cognitive things like different information sets different knowledge different heuristics different representation different models right different ways of thinking These things are identity diversity gender race age sexual orientation ethnicities social class neighborhood right religion these sorts of things So the question is how do these things map to those things or do these things map to those things and that's a very complicated map but people used to sort of say well There must be sort of differences these things must map to those things So in my opinion the best evidence that these things map to these things is that Google knows where you search right, so they know what information you grab Google can tell you these things with an error rate of less than 1% So if you go to Google and ask who am I they will say to me You're a 55 year old white guy who grew up poor in rural Michigan and you're Protestant And I'm like that's right They and they know all sorts of other things that they won't tell me that I wish I knew about myself Right so the point is right that Who we are affects the information we gather the knowledge we acquire that sort of stuff and like and that can be predicted with a high degree of accuracy So if you can predict this from this it's incontrovertible that these things influence those things The other last thing and I'll talk very quickly about some data and then some actions is there's indirect effects So this is some work by Sheen Levine where they created groups of people trading and this looks at sort of market efficiency And these were homogeneous groups and these are diverse groups And one of the things that you find is you get more group think and homogeneous groups And you get in diverse groups because of the fact in diverse groups Because they're sort of more questioning and less trust which we see as negative things Actually can sometimes be positive things because people will be saying wait a minute why are other people doing this, right? But in order for them to actually question it in this case it works and actually because it's no market You can bet against other people you've got to have sort of an open inclusive environment Okay, so I've given you theory. Let me talk quickly data and then quickly practices So the data is kind of amazing. So this is in computer science average number of authors for papers gone from one and a half to five Medicine it's gone from one and a half to six Brian Uzi has looked at this in gory detail. He says here's why If you look at team authored papers versus solo authored papers, they're four and a half times as likely to get a hundred citations This by the way is on a data set of 18 million papers. This is every paper ever published So teams are if a hundred citations is like a threshold for an important paper Which is what most universities use you're four and a half times as likely to get one of those if you do it as a team If you unpack this, this is a baseline paper. What causes these team author papers to be better This is the baseline paper. This is if the bonus you get if you cite other important work This is the bonus you get if you cite works that haven't been cited together in the past very often So if I cite psychology and economics together This is kind of a diversity bonus and this bonus is the bonus you get is if you cite these papers and these kind of papers So good papers and diverse papers you get this bonus plus this bonus plus that bonus And again, this is over sort of the 18 million papers Richard Freeman has looked at a smaller set much smaller It's only 11 million papers So he's science and he finds which is interesting the number of email addresses the number of references and the number of past papers All increase your citations and the impact factor the journal it's in and homophily Which is do you work with people who come from the same ethnic group based on last name? They code last name hurts you So Richard says and it's absolutely right the really good way to write a b-plus paper is to work from someone Basically who's at your same university you cite the same stuff over and over you haven't read many papers And that person's from the same ethnic group with you which is what unfortunately most of us do Right the really good papers are working with people other universities different ethnic groups and citing lots of stuff if you go back and unpack in a different way This is Melissa Schilling from NYU. It turns out this is an odds ratio So adding authors each time you add an author actually cost you in 20% 20% in terms of citations But this 15 means you get a 15 times the chance of writing one of these papers with a hundred citations This atypical connect is the same thing that Uzi get it if you connect papers that hadn't sort of been connected in the past So it's really driving this is not more authors. You might think oh you just get more citations You get more authors. It's not more authors. It's more ideas more authors actually hurt you Accounting for everything else. It's the more ideas that helps you Lada Adamic has looked at this with patents What you're seeing here is this is the paper data This is the patent data. This is on this axis is sort of how good the patent is or are good the papers This is the top. This is the top 1%. This is the bottom 60%. This is a measure of proximity Which is how similar the authors are who worked on this in terms of their research What you get is that again in the B plus area are all these patents and all these papers that are high proximity which means that there's not a lot of diversity in the things you're citing and The very best ones and the very worst ones are diverse And this is the management challenge and this is something that if you look at the management literature It says the same thing if you look across hundreds and hundreds of study the best teams are diverse The worst teams are diverse the homogeneous teams are the B pluses, right? And this is why people no longer just talk about diversity. They talk about diversity and Inclusion right because it's about practice. So how does the practice work really quickly? Then we'll have some like 10 minutes per question So first thing you want to do you want to think about casting wide net So my friend Sheen Levine says he has this wonderful phrase he uses called the siren call of sameness like he who looks like me is smart Right, you've really got to sort of avoid that Google has worked really hard at this Google gets over three million job applicants a year Which is a lot right and so because they get so many job applicants They've got really good data on this and what they found is that problem-solving abilities sort of the main criteria They use for hiring and they look for sort of different ways that people solve problems People tend to think that Google only hires from like elite the elite institutions Well here's Ivy League Cornell's the highest ranked Ivy League, which is actually part public They're seventh Harvard is 10th and Penn is 19th the public schools like California, Michigan right each Wisconsin, Washington, Illinois, San Jose State, Berkeley They actually have much more from public schools from the elite privates, right? But the the main thing to notice is they're hiring from tons and tons of schools and that's because they want this diversity When you think about it as a manager though once you've got you want to think about Inclusion is starting really on Sunday night because the key message I have in some sense is that diversity helps you on these hard problems so I don't feel like so in My world everybody is a sort of outlook calendar, right? So you sort of look in your week and you say good This is what's ahead of me for the week What you need to do I think is think about okay, what are the decisions or what are the meetings? I have during this week where these are really hard problems, right and on the meetings that are really hard problems You want to think about okay? Who should I have at that meeting? How should I prep those people at me at those meetings? Right because it shouldn't necessarily be it's harder. It's costlier. It takes more time to manage a diverse group So you don't want to have a diverse group if you're sitting around deciding what food you're gonna have at the reception Because that's just gonna eat up a lot of time. No pun intended, right? But what you do want to do is have a diverse group when you're thinking about Really complicated policy decisions, but you also have to have some forethought in terms of thinking about who should be at those heart at those those meetings All the evidence suggests in order to make people feel safe in order to really credit inclusive environment You've got to make agendas available beforehand and actually make sure people had time to read the materials beforehand Especially because people who come from less safe places or don't feel as secure or aren't in the majority group Need to make sure that they're informed before the meeting starts Some organizations and I've got the companies at the bottom that do this just to sort of give them credit for this at a place Like Boeing when they have a meeting of a new meeting of a group the person in charge of the meeting will explain Why each person is at the meeting and if you can't explain why I'm at the meeting then maybe I shouldn't be there All right, it's that because the answer shouldn't be Scott looks like me. So I wanted him at the meeting, right? That's not a good reason a Lot of places like Ford and AB in bed which I've worked with which have a culture of doing things fairly quickly What they'll do is they'll do relatively fast direct reports This is how they sort of like leverage diversity without wasting time They sort of speed it up the direct reports and then people can just basically raise a flag and say here's where I'm having trouble Right, here's where I'm getting stuck and then and you focus everybody's attention on the things that are hard Places like Google the New York Fed does this to some extent. They'll require that everybody speaks at a meeting Because you're not there as an observer now This can get a little bit into sort of culture of fear But the thing is once people sort of accept it right then it can kind of work And then the last point I want to bring out is that in this space of diversity I think this is the hardest thing to work on is that Efforts to be inclusive Can also sometimes not take people as seriously as they should so this is Kim Scott And if you can read her name at the bottom, she's this wonderful book out where she talks about radical canner She tells this amazing story about she was working at Google and she presented to Larry and Sergei And and they said your unit is doing great and they're gonna give her unit all this extra money to do more stuff And she walks out of there and she's like all excited, you know, everything went great in her boss at the time Who was Cheryl Sandberg who obviously everybody knows right that? CEO Facebook and Cheryl chases her down and says You said I'm a lot at the meeting and Kim's like whatever and Cheryl's like no no no you said I'm a lot Kim's like, yeah, okay fine, whatever and then Cheryl's grabs her by the shoulders and says You're gonna think you're facing a glass ceiling, but you're gonna face an um ceiling And I'm gonna send you to an um coach on Monday I know a really good um coach and you're going to the um coach, right? So Kim wrote this book on going to the um coach and she has this great two-by-two chart This is just brilliance right so she says look when you think about managing you have to care about people personally And you also have to challenge people directly if you challenge people, but they don't think you care about them You're in this obnoxious aggression box But the danger with inclusion is if you care if you create this inclusive environment You're caring about people personally, but you're not challenging them you end up in the ruinous empathy box That's where you don't tell the person right that they've got an um ceiling So we've had I was I was working with a company and they had someone who had been this was a minority female This is in the United States and she wasn't coming to work on time right, so her value they're thinking of promoting her and her evaluations were all great except for in one category right come to work on if she wasn't coming to work on time and They almost weren't gonna promote it, but then they decided well Let's ask her about this let's at least give her the benefit of the doubt and it turned out You know she was managing West Coast accounts Working on the East Coast and her boss had said doesn't matter if you show up on time her first boss but then Her other bosses never you know she kept never showing up on time for seven years and the other bosses Nobody ever called her on it, and she's like you're kidding me. I'll get it promoted if I show up on time no problem Right, I mean she's handing billions of dollars in accounts But she had been in the ruinous empathy box Everyone was so happy to have her there that no one was challenging her directly about showing up and then her performance was suffering Right, so what you have to do I think in order to have a really effective inclusive policy Is recognize that these diversity bonuses these benefits and diversity come from challenging people's ideas, right? deeply engaging their ideas But the only way you get the ideas to engage them is by caring about people personally I'd be creating a place where people feel safe and included and then you end up in this sort of what she calls the radical Candor box, which is I think where this good stuff can happen. All right. Thank you very very much And I think we have about 10 minutes Okay, so you have 10 minutes and we have a microphone