 Do the wave or something, I don't know, we'll see. Welcome to Moneyball on the Keyboards, Gattling Talented Developers. And I really do appreciate that you're joining me at this time slot. I know there's some awesome talks, and so I appreciate you being here, because without you, there's no talk. I'm Adam Jonas, I'm the Managing Director of Engineering at the Flatiron School, where our primary goal is to inspire people to fall in love with code. We train some of the best junior developers around, placing them at awesome companies like Kickstarter, Etsy, Intel, and the New York Times. I am not a teacher. My team has been working on transforming our internal learning management system into a full-fledged education platform. It's called learn.co, we're really proud of it. Love you to check it out. But it's that work that really inspired me to start putting together these slides, because I spent a lot of time thinking about how we could create some sort of metrics around how we could possibly surface talented students. And so, as I often do, I fell back to a sports analogy, and so today we're gonna talk about baseball. We're gonna use baseball in the ways that talent is identified in baseball as an example of how we can improve our own mental model for thinking about talent in software. And so why am I in particular talking about baseball? It's because, thanks to the Flatiron School, software is actually my second career. I worked in a few different capacities in scouting player development for the Brewers, the Twins, and then ran an academy and spent most of my 20s doing this, living in and around Latin America, and specifically the Dominican Republic. I moved back to the US in 2010 and tried my hand at founding a company that helped drafted players better improve their bargaining position and make more informed decisions about their futures. And while this venture went terribly, horribly wrong, it did pique my interest in code, which eventually landed me at the Flatiron School. And so here's what we're gonna go over today. I'm gonna tell you a little bit about the Moneyball Philosophy and why it created such a shift in baseball. Then we're all gonna go to Scout School together and see how professional baseball talent has been traditionally scouted. And after that, we're gonna see how talented students are identified at the Flatiron School. And finally, based on what we've learned, we're gonna discuss three ways to better understand developer talent and how to better evaluate it going forward. If you're still awake at that point, we'll hopefully do some Q&A at the end. Cool, one more thing before we dive in and there's lots to cover, but I think it's natural to sort of think about these sorts of conversations, specifically when thinking about hiring. But we evaluate talent all the time. We evaluate our coworkers because it has a lot to do with compensation. We evaluate our coworkers because it has a lot to do with clout on our team. We evaluate open source contributions and pedigrees and everything that comes across on a resume. We evaluate speakers and whether they're any good. We put value judgments on a lot of things. And so I think we're wired to do that and that's totally okay. But I challenge you to just not limit your thoughts on this subject to just the interview process because it's a lot bigger than that. Let's go on with the show. So you might be familiar with Moneyball. It was a book written in 2003 by Michael Lewis and then turned into a movie with the always handsome Brad Pitt in 2011. And the essence of Moneyball, while specifically focusing on baseball, is really about objectifying what was previously thought as subjective. And so the subject of Moneyball is the Oakland Athletics. I can see one of those hats right there, awesome. And the fact that they have smaller revenues and constrained resources, against other teams they're competing against on that right side of the graph, the New York Yankees, the Boston Red Sox, et cetera. And it's because of those constraints that Oakland is forced to search for undervalued players in the market and they end up turning to these underutilized tools of statistical analysis. And so what did we learn from the Moneyball approach? We learned that industry outsiders who had not been indoctrinated with the traditional way of doing things could identify inefficiency in the old ways. And we learned that we could use better talent analysis tools to determine a player's objective value. So for an example, what's the difference between a player who gets to first base via single and a player that gets to first base via walk? Intuitively, we favor the person who took the action, the person that swung the bat and earned their way to first base. But from a team's contribution perspective, it's about the same. You got a guy on a first base, that's about all it is. And so when we think about these contributions, we often will overvalue what we think is being earned or somehow has more merit to it. But, and this is exactly what was happening in baseball. We saw players with high batting averages were being overvalued and players with high walk percentages were being undervalued. And so the Moneyball approach took advantage of this, this gap between perception and reality. And it was more accurately able to value a player's contributions to the team. So this is Moneyball, it's this objective view, this data-driven view, this high-level view, this detached view. But it's not perfect because humans are complicated. And so let's turn to a little bit more of a humanistic approach by all heading to Scout School together. Scout School is a very real thing. It's sponsored by Major League Baseball and I was lucky enough to attend in 2005. And you have to be invited by a team and what you do is you spend two weeks spending the mornings going over the theory of scouting and the afternoons literally applying what you learned in the field, writing up reports and seeing different levels of the game. And so the first thing you learn is that all players are evaluated in the five following categories called tools. Hitting for average, hitting for power, running, fielding, throwing, this is it. This is what everything everyone is evaluated on. And players that possess elite talent in every tool, well that's pretty rare. Willie Mays stands out as sort of the prototypical five-tool player and that he could do it all. But most players have strengths and weaknesses and it's these strengths and weaknesses that help determine their role and the ways that they can contribute to the team. There's rubrics for all of these tools as well and in Scout School they give it to you in this neat little binder, only it was that easy. But let's go over the running tool in particular and these grades range from two to eight as they do in all categories. And it's measured by the time that the bat hits the ball on contact to the time that they arrive at first base as measured by a stopwatch. Now you can imagine there's some complicating factors here. First of all, a righty is actually physically further from first base than a lefty. Also, you would imagine that baseball players being the people that they are are not gonna run 100% every single time to first base. And so we need to look for the right situations to actually take down that time. Lazy Pop flies, gimme singles to left field, home run trots, probably not the right time to do that. And so while traditional scouting has this reputation of being purely subjective and sort of antithetical to the moneyball approach, you can see how there's some standardization here. You can see that there's possibly some consistency and I think you'd be surprised with how consistent the grades are from experienced scout to scout. So the only way that this could work is if it were written down all the time. And baseball has done an amazing job of leaving this robust audit log for us to examine decisions over the years. This is an actual scouting report. You can see the rubric is there on the left. It's not particularly hardened because you're actually talking about all the tools. You also have the five star categories that we just went over, as well as some other categories to give you a little bit more context. So this lets us know why we're hiring this person or why we're passing on this person. It also lets us know where the candidate we see as we evaluate the candidate, where are they today versus where they could possibly project to the future. Same player, a little bit more exposition. And this provides a ton of data, not only data on the candidate themselves, but data for us. This is not only about how we see the person that we're evaluating, it's also about the evaluator. And so this means that we're a lot less likely to succumb to hindsight bias where we claim that we saw it all along. At the end of the day, a scout's job is to predict the future, and it's hard. It opens scouts to a ton of criticism, but leaving a record can be your proof and ultimately your redemption for naysayers. I wanna focus on three particular statements of this. And this is the good face, this is the first one. So the good face, pretty subjective, right? Why is that even in a scouting report? And the reason is that scouting, like a lot of other things, is pattern matching. And so this scout felt that he had seen faces like this in the past. They could possibly be on an all-star poster or the face of a franchise. And so he thought that this was data that he would include in the report. We'll be in the big leagues by age 21. Pretty bold. You're gonna go out on a limb for, this happens to be an 18-year-old, and say that they're gonna make it to the highest level possible by the time they can legally drink. All the basic tools to be an outstanding short-stop prospect, best-raw tools of any position player I've ever scouted, should only get better with maturity and experience, tools and makeup to be a star. This is in fact a report on Derek Jeter who was drafted sixth overall in 1992, and he in fact did debut at age 21, and he was a 14-time all-star, which clearly he fulfilled the promise that was predicted for him by the scout. And so if you think that this Derek Jeter could possibly grow up to be this Derek Jeter, then you're gonna want credibility for that. You're gonna want someone saying that I wrote this down, this is what I put myself out there all those years ago, and guess what, I was right. Let's look at three other examples. Trevor Hoffman, a future Hall of Fame pitcher, Albert Pujos, the best hitter of his generation, David Ortiz, cultural icon, and three-time World Series champion. All these players were missed. Trevor Hoffman was selected 288th overall as a short-stop, and then converted to a pitcher after two seasons of proving that he could in fact not hit. Albert Pujos was drafted 402nd overall, meaning that every single team had 13 opportunities to select the best player and failed. He was so good in fact, he was the fastest player of his draft year to make it to the major leagues, and he went on to win Rookie of the Year at age 21. In 2002, which is the season that Moneyball takes place, David Ortiz was let go by the Minnesota Twins for no compensation, and he was picked up by the Red Sox the following winter, and hit almost all of his 500 home runs for them. And so how did this happen? How did everyone miss these guys in an industry so focused on evaluating every single player? It's because they weren't seen in the proper context. Their talent was hidden by their circumstances. Hoffman was at the wrong position. Pujos was at a junior college that had never produced a major league, Ortiz had a myriad of nagging injuries which disguised his potential. And so let's take a step back and acknowledge that talent and context are inextricably late. There's in fact no such thing as general talent. Derek Jeter earned more than $400 million in his 20-year major league career, but it's not like you'd let him touch your MRC file or do your taxes. His talent, your talent, my talent, everyone's talent, they only exist in the context in which they're applied. I want to extend this a little bit further. Within the game, players are expected to contribute in ways determined by their roles. And so because of the defensive demands of a shortstop, fielding, throwing, and running are considered the most important tools for prospects. If we look back at Derek Jeter's report, you can see he comes in at a 65 on fielding, a 60 on arm strength, and a 70 on running, and is hitting in power great out to 50s, major league average. And so he's the exact profile for what the scout is looking for in a shortstop, and you can tell by his gushing report. Then there's the first baseman. The first baseman only works 100 feet away where the defensive demands of that position are just not as critical. And so they're expected to contribute in other ways, namely to carry the offensive side of the coin. Power, hitting, that's what they're known for. And so it's really determined, your value is only determined by the role in the way that you can contribute to a team. And these are generous. These are people that are expected to contribute in multiple ways. There's also the specialist. In this case, the specialist is the pitcher. We only care about one tool of theirs. That's their arm. How hard can they throw? How well can they throw? How many outs can they get? This is a picture of Jim Abbott. And Jim Abbott had a 10-year major league career. He threw an O-hitter for the New York Yankees, and he won a gold medal for Team USA. What makes Jim Abbott a little bit different is that he was born without his right hand. And so if you met Jim Abbott on the street and tried to shake his hand, it might be difficult to imagine that he is a professional baseball player, let alone had this highly successful career. But it's because of the specialized nature of his role and his unique abilities that he was able to have the sustain success. And so given that context is so important, what's your context? Are you looking for a generalist to work on a team? Are you looking for individual contributors to work alone? Is this a small startup where flexibility and adaptability are valued or a large organization where optimization is most important? And so given that talent doesn't live in a vacuum, getting a handle on your unique set of circumstances and the circumstances that your team faces is critical to understanding what talent really is. Let me tell you a little bit about my context and the way that we identify students at the Flatiron School. And I think I sort of imagine this as a Petri dish, you can tell by my awesome graphic. The Flatiron School has contact with thousands of students a year. We have a 6% acceptance rate and so over the course of the last three years we've accepted hundreds of students and all of them that have been job seeking pretty much to a person has in fact gotten a job. And so we have lots of conversations with employers and when I started talking about this to the admissions department, I think I was sort of overwhelmed with the parallels to my time in the Dominican Republic where a lot of players had very little training and how do we know that they have talent before they really understand how to play the game? And so when we're tasked with trying to understand who has talent and who doesn't, it's really difficult because you have very little context to go on. And so what's their ceiling? How far can they go? That's hard. Remember the five tools of baseball? These are the five tools of the Flatiron School. Hireability, technical background, aptitude, passion and culture. And so before anyone has written a line of code this is what we're looking for. I wanna go over these really quickly. Hireability, our school's designed for a very specific outcome and so when I say a successful student I'm talking about a student that can go from no experience to employable in a really short period of time. They can also get their first job at a high starting salary and they're promoted quickly and their employer of course is giving us a lot of positive feedback on them. And so to be able to deliver on this promise that we're making to our students we need to know that that outcome is actually something that they want. And so that's a big part of the admissions process. We also need to know that they can hold a conversation, that they can talk about times that they work together in a team, that they can project enthusiasm and energy because all job interviews have some of those components in them. Technical background, it's exactly what it sounds like, how much have they done in the past. Aptitude, and you might see how some of these tools can sort of bleed together, sort of like a fast runner that can get down the line, that might boost their hit tool because they can beat out some ground balls. And so how well do you think that this person can learn is what we're actually after an aptitude? How well can they integrate new information? We happen to use tic-tac-toe as our code challenge and that's a highly Google-able challenge. Lots of our students are just pulling down solutions from the internet and so part of what we do is suss out the difference between what they actually wrote versus what they pulled from somewhere else. And I think that's okay. I think that's okay because that's how I work. Not every line of code, not every way I attack a problem is from inside my own head and so we should be using the tools and the approach that we expect to be using after they graduate. And so what we're looking for here is pattern matching. Can they see sameness and differences? What's the tiniest next step that they could take to improve? Here's our rubric. It ranges from one to five, starting with low effort and little comprehension, going all the way up to clean object-oriented code that's tested and clearly demonstrates that she's mastered the problem. We have exemplars for all these answers. And again, like baseball scouts, I think you'd be surprised by the consistency from instructor to instructor in their grades. Passion. So there's a sort of sycophie in nature to what we do because when we run into code, sort of the learning never ends. We push the boulder up to the top of the mountain and then we think we've got it and then we realize we're just at the foot of the next mountain. And so we find that our successful students really enjoy the process. They even enjoy, I guess it'd be generous to say the mental gymnastics. It's really the mental punishment of learning new things. And so we need to see that students have overcome difficult things in the past and they've come out on the other side that they're really willing to push through those challenges. Culture. This is a tricky one. We don't admit students, we admit classes. And those classes need to be balanced, of course, in terms of gender and ethnic diversity, but they also need to be balanced in terms of background and diversity of perspective. If we have a bunch of finance folks who pass all of our criteria, we may ask some of them to defer because they just sort of come from the same place. And we've seen that having a monoculture really impedes learning, motivation, and ultimately collaboration. I'm gonna throw this in, but it's a little anecdotal. Somewhere around 80% of our top students have some sort of creative outlet, whether that be professional or as a hobby. And so these are poets, these are musicians, these are photographers and painters. And it doesn't seem to really matter what the medium is, but they just seem to approach problems differently. And so this is something that we've discovered. We're still working on it, but it seems to exist. So here are some actual ratings. These are the actual ratings of our last class, of course, without the names. And so by a show of hands, what do you think was the best predictor of success in terms of speed-defining a job, starting salary, and employer satisfaction? Was it culture fit? No people. Was it passion? Some people. Was it technical background? Two brave souls. Higher ability, awesome, and aptitude. Cool. It was in fact passion, and it was by a large margin. Passion was the best predictor of whether a candidate was going to succeed, and it wasn't even close. What didn't matter, technical background. Technical experience had almost no correlation with how fast a student was hired, or what their starting salary was, or how fast they were promoted. Passionate students eventually caught up with the experienced ones, and then even surpassed them. And so it would be great if this rubric sort of worked for everyone, but unfortunately we aren't all operating within the same context. And so you need something that's tailored to your situation. And so what's important to you? What are you thinking about? And there's some places that you might look to determine your tools. Performance reviews, although those can really be a minefield if you guys aren't doing them right. Mission statements, value statements. Sometimes those are updated, sometimes they aren't. One-on-ones and retros. What's your team actually thinking about and talking about? And so ultimately you can't be successful in evaluating talent without knowing what you value and what's important to you. Okay, everything we've been talking about has been pretty abstract. Let's discuss three things that will help you better think about talent starting today and maybe discover some of those diamonds in the rough. Admit we have no idea where talent comes from. Control for sample size bias and rethink cultural fit. Okay, so Trevor Hoffman was drafted as a shortstop. He's converted to this pitcher. Is it outside the realm of possibility that you might come across someone who was in a generalist role and if they could just focus on the one thing they did particularly well, they could excel? What about Albert Pujos? Albert Pujos got drafted from Maple Woods Community College. If you got a resume from Maple Woods Community College, how would you feel about that? Is that something that you would give a lot of attention to? Maybe the physical injury analogy doesn't really work here in David Ortiz's case. But what if someone had a number of personal circumstances that were affecting their performance? What if they were going through a divorce? What if they had a three-month-old who was conducting some sort of twisted sleep deprivation experiment on him? This is my guy. He is really messing with me, but I do miss him. Anyway, so we're so bad at separating context from personal attribution that there's a term for it. It's called fundamental attribution error. And so baseball is really bad at this too. There's 600 players playing in the major leagues at one time and 72 of those players play in the All-Star game. So that's the top 12%. Only a third of those are the slam dunk first round prospects that everyone saw coming. And so in reality, if a prospect, whether that be in baseball or in software, is projected to be a star, guess who else knows? Everyone. And if you're Google and if you're Facebook and if you're Amazon, you can afford to throw a ton of money at everyone and hope that they develop. But for the rest of us that need to live in reality, we need to be looking for candidates that fall through the cracks. And they may fall through the cracks for a number of reasons. But ultimately, they've been viewed through the conventional lens where we judge their talent and that might emphasize their weaknesses and obscure their strengths. And so this is really what Moneyball boils down to. It's looking where others aren't. The second thing is sample size bias. So we're wired to make snap judgments. It's no matter how thoughtful or kind we think we might be, we still do it. And our goal is to create as many touch points as possible without necessarily dedicating more time. It took baseball a really long time to come to the realization that one game was just a snapshot. And because the software industry is much, much younger by comparison, I think we're still conceited enough that we can, we think that we can judge someone's talent based on one interview. But let's admit that we're worse at this than we actually think we are. So creating a fuller picture of the process, creating a fuller picture is essential part of the process. And this is how we currently do it. We got whiteboard challenges, we got phone screens, we have open source commits, we have looking at their blogs and resumes. I personally like looking at blogs and open source commits and any sort of evidence that demonstrates consistent long-term behavior. Pairing has also been mentioned as one of the better methods since it's actually simulates the job that someone might be looking to get or working to work in. But ultimately these are all data points and they need to be treated as such. And in my opinion, what we're really after is how much does she love the work? Is he creative and will she make your team better? Traditional interview screens can do this. We just have to be asking the right questions. As can whiteboard challenges or any other data points that we use as long as we're taking the macro view as opposed to this sort of one game snapshot perspective. Cultural fit, cultural fit is hard. And I think it's become a little bit of a mess. What we're starting to screen for is much less of a match with organizational values and much more of a match for personal fit. And so I think it's natural to bond over shared experiences and pedigrees and if you went to the same college you can play the name game and that helps us feel connected. But cultural fit has become this new form of discrimination all in the name of employee enjoyment and fun. And so what are we actually looking for here? Are we looking for someone that we wanna have a beer with or are we looking for someone that we wanna work with? To turn this around we're gonna need some serious self-reflection because the research says that more diverse teams outperform less diverse ones. For jobs involving complex decisions and creativity more diverse teams outperform less diverse ones. We're more confident in homogeneous groups because that makes us feel comfortable but they just don't perform as well. And so to make this happen we need to confront our biases head on. It's scary. It's really scary to admit that we might have personal biases or institutional biases but the data is likely there if you have the courage to take a look at it. We're comfortable with what we know for what we've seen in the past. Faces like ours, backgrounds like ours, skills like ours but is comfort really enough? Is that good enough criteria? This is Jackie Robinson signing his first contract to become the first black player in the major leagues. Now Jackie was signed by the Dodgers for one reason and one reason only and that was to win. He of course went on to have a great career, a Hall of Fame career but from a hiring perspective the Dodgers got access to a talent pipeline that no one else was willing to take advantage of. And so yes it was the right thing to do from a social consciousness perspective but it was also a huge competitive advantage. Hiring people from non-traditional backgrounds takes a lot of guts but it's one way to take advantage of a developer hiring market inefficiency. These players offered incredible value to the organizations that took chances on them. And yes, high price folks might be worth their resources but you just can't afford all the most expensive players all the time unless you are the New York Yankees or Boston Red Sox. And so talent needs an opportunity from you. Being more objective by writing things down, establishing rubrics for rating guidelines, introspecting on our own scouting abilities, not letting cultural screens be prohibitive. Just generally getting away from going with our gut means that we're gonna be a more inclusive industry and ultimately push us more towards a more complete meritocracy. Thanks. How else can we look at technical background? Yeah so actually my team, we have a different set of tools. I knew this was gonna come up. It is communication, ownership, adaptability, velocity and quality. And so with some, given the small team that we have and the speed that we're moving, those are the things that we look for. We have our own rubrics and I do grade people out when I interview them. But this is what's worked for the school and again, it's really based on your context. Yeah, oof, I don't, no, no, I don't. I think we should, I think it's sort of built in because a lot of, we do hire quite a bit out of our own school too because we, they get to go through the program and then we know who awesome developers are because they just graduated. But in terms of hiring from the outside, yeah I think that's a reasonable way to approach it and it's not something that I currently do. Sure, so the question was, when evaluating culture fit, how different is this person than the rest of the organization? Yeah. Yeah, I mean I think we just have to be really careful of making culture fit like this prohibitive screen. Like there's, I mean we use Greenhouse as our software within the school and the culture screen is one interview and we also do the sort of gladiator style like yay or nay thing which I think works out pretty well. But it's really up to the person doing the screen. Oftentimes that's our recruiting manager for my team in particular. I think I have, it's up to me to make those calls but when I'm looking to round out my team, a lot of it is thinking about what we don't have and so is this a person that is gonna make our team better? Are they creative? How much do they love this and how much are they gonna develop? But going back to this sort of present versus future, like passion just drives everything and so if that's the case and they really wanna get better, they're gonna get better. Yeah, it's somewhere in between, right? I was a student in the first class of Flatiron and they didn't really know if it was gonna work. It was sort of just like 20 people in a loft and the founder just telling us stuff. It worked for me, so that works but now we have an actual system and we're using the product that I'm working on to sort of scale that out and we've seen things like before we had the system in place versus after they can consume like twice the amount of material and so we're able to cover some things like that we just didn't cover before and so we give them much more of an education. But yeah, I think it's buyer beware, right? Bootcamps are a thing. They weren't really when I was going through the process but they have changed a lot of people's lives so that seems to be good but people that are getting oversold, like did they do enough research or could they not get into a more prestigious program? It's hard. Yeah, we have about as long term as me. I was in that first class so yeah, we're up to a little over three years. We do. We're really proud to have an audited jobs report where we go to every single student and this is a third party consulting party that we pull in and actually they contact every student, talk about their salary, talk about their prospects. As far as I understand, we're the only school that's doing that so yes, we're taking that data very seriously. Oh, in terms of the grading. Yeah, I don't know if that's true but given my answer to the previous question is then I use a different rubric for hiring on my team. I think it really is context dependent and so if you're talking, if we're putting someone at Intel, well, they're gonna clearly have a different hiring standard than we would for screening our students. Thank you everyone. I really appreciate being here.