 I guess we'll go ahead and get started. It's 1050, welcome. This is the code-free developer interview. My name is Pete Holliday. You can find me on Twitter as at too much Pete. Feel free to tweet at me during the presentation. I'm sure I'll get the notifications up here and we can have a nice little conversation while the talk is going on. I'm currently a software engineering manager at CallRail. CallRail is a company in Atlanta, Georgia. We make a call tracking analytics platform. And CallRail is one of the companies that I've used these particular techniques at and have a lot of success and had a lot of success with. We've hired, over the years I've hired dozens of engineers using these techniques without having ever seen a single line of code from them before they started. So my goal for this session is, it's a very humble goal, it's a very modest goal, and that is to eradicate all coding interviews. So, thanks Jennifer. But no really, I'm kind of kidding. A more realistic goal is to convince you today that there's a better way to evaluate your potential engineering candidates than making them code for you. But I guess before we get into the meat of it, what even is a coding interview? My definition is a coding interview is any interview practice that requires you to see code that your candidate actually wrote. So, reviewing their code samples are GitHub profiles, which is probably one of the most boring things that I've ever done as a hiring manager. Code tests like codility and the like, code challenges where you give somebody, oh go build this to-do list or whatever, and bring it back to us. And live coding or pair programming, those can be incredibly stressful, even for people who don't normally have anxiety issues. But these aren't the only things that you can do that I would classify as a coding interview. It's really anything that relies on seeing code a candidate has written to make an evaluation about their fit for your company. So, why do we do these in the first place? Why do we have this whole list of things? And it's because they're replacing some less great interview techniques. So, whiteboarding, the thing we all know and love. Clever riddles, like how many tennis balls fit on a 747. I'm really not sure why anybody ever thought that would be a good idea for an interviewing question, but people have used it, so. And then language trivia, like, what's the difference between a subclass and an interface? And the reason that these are generally not great is because they're bad. They don't tell you what, they don't have a good ability to predict whether a software engineer is gonna do a good job at your company. They don't really map very well. So that tells us what we came from, but why are we here? Whoops, what is, why interview with code in the first place? The theory is good, right? The idea is that if you're gonna hire a software developer, you should see them develop some software before you give them a job. You should kind of make sure that they can do what they say they're gonna do and what you think they can do on the interview. But the big lie of coding interviews is that they don't actually replicate real work. It's a good theory, but if you think about the way these coding interviews typically go, if we treated our coworkers the way coding interviews tend to treat candidates, we wouldn't have many coworkers left. They're really stressful and they don't map particularly well to the real work and why is that? Well, for one thing, they disadvantage people without a lot of free time. So if you've got a take home test, if it's codility, if it's check out my GitHub, look at your open source contributions, that disadvantages people who might have a really stressful day job or maybe they're taking care of a sick parent at home or maybe they have, maybe they're a single parent and they just don't have time to do your coding challenge in the evenings after work. Live coding and pairing is stressful. It's really difficult to get a good feel for whether or not somebody can do the work when they're constantly worried about the evaluators sitting over their shoulder making sure they don't make a typo. I've been in the industry for 25 years and even I get a little nervous and fidgety when I'm being, somebody's looking over my shoulder while I'm coding during an interview. This is sort of the hidden one. They're difficult to develop and maintain. It is really hard to develop a take home code challenge for example or a pairing session that accurately and accurately evaluates a senior's skill. It's easy to develop a code challenge that will tell you if somebody can program at all but then how do you modify it to allow a senior to show you what they can do in a reasonable amount of time. And then once you have these code challenges you have to maintain them. You've got to make sure that they continue to work that you don't put your candidate into dependency hell on accident or on purpose. Don't do that. And then finally we're in an industry where hiring is hard. It's hard to find good candidates. And if you're trying to poach from companies where people are mostly happy they're just not gonna do your code challenge. And if they do it may take them twice as long to get it done because they've got other stuff going on. And so in the end all of these things make live coding and pair programming and all of these other coding techniques they sort of fight your goals of hiring good people as quickly as possible. Excuse me. So what should we do instead? Now I hate to be the bearer of bad news here but the solution to this problem is to talk to the candidates. I know it sounds a little snarky and it kind of is but I wanna emphasize that the techniques that I'm about to talk to you about they're not the only ways to go. These are not the only things you can do. These are the three that I've had success with but really the core part of this is that you wanna have a conversation with the candidate about technology, about the technology they've built, about the technology that they're interested in and you wanna get to know who they are and what kind of software they build through any means necessary. And so these are just the options that have worked well for me. You might come up with something completely different and if you do I'd love to hear about it so hit me up on Twitter. But before we get to that point and this could be and in fact was earlier an entire talk. What are some general interviewing tips? This will apply to anything, it'll apply to your coding interviews, if you continue to do those, it'll apply to your code free interviews, it'll apply to really anything. Define what you are looking for ahead of time. This is one of the hardest parts of hiring people, deciding what you're looking for in a candidate. But if you walk into an interview and you haven't decided what the criteria are for the hire that you're making, you're going to be hiring almost entirely on your own personal biases. And that works out sometimes and a lot of times it doesn't. Ask all of your candidates a consistent set of questions. This is actually much easier to do when you've defined ahead of time what you're looking for. They flow naturally into one another. And then your interviewers should write down their thoughts before they talk to anybody else. So after they've had their session after they've talked to their candidate, just have them go back to their desk write down what they think, put it on a notepad or what have you. And the reason for this is that once you get in the room and just kind of put yourself in their shoes, once you get in the room, if you haven't written anything down, the first person starts talking, and they say, yeah, well, I didn't think they were very strong technically. Even if you previously thought they were strong technically, your opinion starts to shift and you start to see things from your coworker or your colleague's perspective. If you've written down, actually, I think they're pretty good. By the time that person is done talking, you might agree with them, but you still have some points to make about, no, well, actually, I thought they might have actually been better than you maybe thought they were because of X, Y, and Z. And that helps you get everybody's real perspective on what it was without the biasing that happens from listening to somebody else talk. And the final point, and this is a big one, interviewing is a skill. It is not a skill that all software engineers have inherently by virtue of the fact that they know how to program. And if you don't practice, if you have 100 engineers on your team and they do one interview a year, they're not probably gonna be very good at it. And so this is something that you have to work at, something that you have to practice, that you have to get better at, and then you have to think about why you're doing what you're doing and how you do it. In the end, if you do that, if you put the work in, you will get better candidates and you will get more of them because I think the thing that people often leave out of talking about interviews and interviewing is that a poor interview process or a poor interview generally is going to cause candidates to think disfavorably about your company. So you think about interviews that you've been to if you thought the interview was kind of weird or you didn't get all the information that you needed or the interviewer asked you kind of some weird questions and you didn't understand how they're related. You're kind of mentally marking off points from the company that you're talking to. And so having a bunch of skilled interviewers can actually help your attraction rate and get more candidates in there. So what are the ways? So we've talked a lot about how and talked a lot about why and sort of how to get started with interviewing in general. But so what are the three techniques? The first one is one that you probably already do to one degree or another. You wanna dig into their experience. Look at their resume, find something on there that you think is interesting or even better once they get in just ask them, hey, can you tell me about a project that you've worked on recently that was particularly interesting or particularly challenging and let them direct you towards the thing that they think best represents their own abilities. That allows you to sort of highlight or allows them rather to highlight what they think is best. If you pick something, be aware that there's a risk that you might pick something that looks great on paper, but then actually it wasn't that great. If you run into that situation, it's fine. Redirect either pick something else from their resume or ask them. And so a lot of this is kind of impromptu like finding the path as you go, but definitely have read their resume before you come into the meeting into the interview. That's a big one. And having some idea what you're gonna ask about once you get there is good. So what are some of the questions that you can ask? Obviously you wanna ask what their role was on the project. I wouldn't put too much weight on this. One of the reasons you might be getting a senior candidate interviewing with you is because they're not being given the opportunities at their current company, but it is good to know like, did you build the entire thing? Were you a code reviewer? Did you help with the architecture and somebody else implemented it? It's good. It colors the rest of the conversations you have. The next thing that I like to do is I like to figure out how it works myself. So act like I'm doing an architecture review or that I'm onboarding onto their team and I have to figure out what is this thing and how does it work if I'm gonna be working on it. So that means asking a lot of really detailed questions. How does the system talk to that system and how do you store that data? And what do you do for backups? And what about the lag time between these two things? Do you have real time or is it just an, or is it a eventual consistency? Really dig in. And you might run into places where people don't remember or they don't know and that's all fine. But really try to get as good of a picture as you can about the project they're working on. And then these next two are ones that I've kind of come up with over the years that I've gotten some really interesting answers to. Where's the worst technical debt in your project and how would you fix it? And why hasn't your team fixed it yet? The last one is interesting because it kind of gives you an idea of whether or not they understand the business concerns of refactoring and technical debt. That's a really important quality for a senior to have in my opinion. And so it's one of the things that I like to figure out from people. The final one that I like to get in is, have you had any outages in production or bugs, big bugs that you want to talk about? Why did it happen? What could you have done ahead of time to catch it? How did the team fix it? How did you respond to it? And this gives you an idea, especially from your more senior candidates, of what's it like when the rubber meets the road? How do they respond when there's problems in the system? And what did they learn from their production outage or their bug? So that's kind of a good way to, it's probably something you're already doing. It's worth linking that up with the questions that you already, with that, the section of going through and figuring out what your criteria are. Link that up with their experience. Try to figure out how you can ask questions that identify those qualities that you're looking for in your software engineers. The next one, and this is if you, excuse me, if you caught the asterisk on the title slide, you might be wondering why there's a qualification there and have them do a code review. So this is not necessarily a code-free interview. It's more of a coding-free interview. But one of the things that I like to do is give them a small amount of code and role-play them reviewing my code. What you get out of this, I'm not in this case trying to replicate them reviewing a 2000 line pull request into one of our major systems. What I'm trying to do is figure out how do you talk to other engineers about their code? What questions do you ask? Are you gonna look at something and be like, oh, that's a dumb way to do that? Or are you gonna say, why did you choose that particular method to calculate that information? And so one of the hard things when you start thinking about having your candidates do a code review is selecting what code to review. My big number one rule is do not have them review your own production code. And this is not for security reasons, actually. It's really because they will not have the context necessary to really review that code well and it's gonna be super overwhelming for them if you drop them into a 500 line Rails model that controls your entire application. Work as you're coming up with this project or this code sample to review, work to reduce the complexity of it. It really make it as simple as you think you can make it and then make it even a little bit more simple because you don't really need to test whether or not they can read the most complex code in your system, probably. You'll wanna include realistic bugs but do not make this a bug hunt. It is really hard to find bugs in code without being able to run it. It's really hard to just look at code and find a bug especially when it's the first time you've seen it and you're on the spot. But including a couple of realistic bugs in the code will give you an idea occasionally of a candidate who has a particular eye for detail. So I think of these kind of as bonuses but we include a couple and if they don't find them it's no points off, it's fine. So what kinds of things meet that criteria? Completely contrived pull requests into completely contrived repositories. One option if you're currently doing like a take home code challenge, turn that take home code challenge into your code review, open your own pull request into it, have your candidates review that instead of writing the code themselves. Or find an open source repository and fork it and open a pull request into it that has the qualities that you want to see in a code review. The trick with that one is make sure that it's easy to understand what the open source repository does. You don't wanna have to spend 10 minutes of your 20 minute code review time explaining them, okay, well this application does this really complicated thing. So pick something that you think you can convey in a sentence or two so that they understand the larger context. We tend to go with completely contrived but open source repos also can work in a pinch. So now once you've gotten the code review you know what code they're gonna review. What kinds of things are you looking for in this code review? And this can also influence what you put into the code review in the first place. One of the things that we like to have in there is we like for the pull request to contain some kind of bad commit habits, some bad pull request habits. So short commit messages or pull requests that don't have any data in the, don't have any explanation in the pull request. Maybe commits where one of the commits has a syntax error in it and the next one fixes it but those weren't squashed down into a working commit. Whatever your personal company's commit habits are that you like to have look for those in the developer or in the reviewer. You might look for non idiomatic Ruby code. You might look for things that are not the Ruby way or whatever coding framework you're in. I might give you an idea of whether this person's experience is in Ruby or in something else and that's fine. It might be totally fine to you if they aren't Ruby experts but it's a good thing to point out during the interview or to learn during the interview. I would avoid or I wouldn't avoid. I would look for a reviewer that can find overly complex code and other code smells like confusing variable names, overly architected classes. Any of those code smells are good candidates to put into your code that's being reviewed to see if the reviewer picks up on it and then like I mentioned, actual bugs but be careful. Now, you really wanna run this as best you can like a role play. You want to, this needs to function as if they are talking to you, the developer so that you get a good feel for how they actually talk to somebody who's code they're looking at. We typically tell them this is going to be a role play. Pretend like this is my code and that usually works for most people. They can usually get into character pretty quickly and if they don't, you can redirect. So you can say, okay, well, you know, they say, oh, well, this person's code. You're like, okay, let's pretend like it's my code. What would you say to me if it were mine? And the main reason to do that is because you get a really good feel for how they communicate with the code reviewer. This can highlight, if somebody who does really well on this portion, it can highlight that they might be a really great ad to your team, even if the other technical indicators didn't put them in your top two. They might have been your third best technical candidate but they have amazing code review skills and they communicate so well with their colleagues. That person might be a better hire than the two more technical people just because of what they bring to the table in other areas. So that's the code review portion. I encourage you to play with this and run these. All of these, I encourage you to run them internally with people that you know as you're practicing to see if they'll work for you. And this brings me to my favorite one. Collaborative system design. This is not only a really effective way to find out somebody's technical level but it's also fun if you do it well. So what is a collaborative system design? You hypothetically build something so you just talk with the candidate. We're gonna build the tool or a platform or a project or we're gonna build something. We're not gonna write any code, we're not gonna write any pseudo code. You might write on the whiteboard, you might let them write it on the whiteboard. What I like to do is I write on the whiteboard while they talk so I can sort of sketch out what they're describing, how the system works and that helps me make sure that I understand and it helps them make sure that I also understand that's a big part. And the collaborative part is important. So once we get, we'll get a couple more slides and you'll see that there are going to be some places where there's not a right answer necessarily. There are many potential right answers and going back and forth with the candidate, letting them kind of choose the path helps you learn how they think and what they think is important. The big thing that I love about this method is that once you get good at it, you can figure out whether somebody is junior, mid or senior in 15 minutes. It's really not that, it really gets right to the heart of things pretty quickly. And it's not super intimidating for most people. At least that's been my experience. It is a little bit at the beginning like oh gosh, I've gotta do this perfectly but once the conversation gets flowing, people tend to forget that they're in an interview and they're just talking to another technical person about code and that's a really great place to get with your candidates. So again, we're back to a situation where you have to figure out what this is gonna be. It should be easy to understand. It's excellent if your company makes software that everybody understands. So if you work at a company that everybody kind of knows about, you might use internal, you might use your own company's product as the thing that you're gonna try to rebuild. If your company doesn't necessarily isn't that well known or the use case is kind of more complicated, you might have to do something else. And try to make it related to the skills that you're hiring for. So if you're hiring a back end engineer, maybe building some Rails app or building some kind of back end is great. If you're hiring a front end engineer though, those skills might not be as applicable. And so when we interview front end engineers, we're an Angular shop, we actually show them a screenshot of kind of a pared down page on our internal site and say like, hey, how would you, what would be the components in an Angular system? Like how would you build this out? How would you architect it? And I'm not, I don't know very much about Angular, but the people who do our front end interviews say it works really well for them. But tailor it to what kind of thing you're testing. And when in doubt, if your system, if your own thing doesn't do it, pick something generally well known. Like a social media network, Twitter, Facebook, Instagram, what have you. It's something that most people are gonna know about. And if they don't, for some reason, know what Facebook is, you can kind of explain the concept to them in a couple of minutes. If they don't know what Facebook is, they might look at you crazy when you explain to them what Facebook is, but it's okay. So now once you've picked a project, how does this work? Like I said, this is really quick. You're not gonna build all of Facebook in 20 minutes. And so let them know ahead of time. It's up to you whether or not you tell them how much time they have. We generally just say like, hey, we're gonna start going through this process. We're not gonna get through all of it, but let's just get as far as we can through it. And that gives them an idea that like, if you cut them off at 18 and a half minutes because you've seen enough, it doesn't make them anxious that like, oh God, I was doing so poorly that we just canceled the whole thing. Pre-populate the boring stuff. So the boilerplate sections like auth that you might not find as interesting. So avoid going down rabbit holes. And let the candidate lead. When you have an option, a fork in the road, let them do that. And then the big key, the big key for this one is if the candidate is moving fast or they seem really comfortable with the thing that they're building at this particular moment, that's when you increase the complexity. And that's what really brings value to this particular technique is that it allows you to dynamically scale the challenge level of the interview to the candidate. And that's where you determine whether the person is more junior or more senior. So let's talk about building Facebook. What would we do? CallRail, we really value people's contributions to product and sort of thinking about what our users need. And so we ask them, for starters, like, hey, if we're gonna build Facebook, what's our MVP? What's included here? And so we set them up with, okay, we've got users and they can log in already because we don't wanna spend 20 minutes talking about how you design auth. So what's included in the Facebook MVP? Well, friending, status updates and news feed. When you start getting into things like likes and comments and reactions to likes and reactions to things and pokes and whatever else you have, be careful because that, is that really an MVP? But instead of saying, oh, no, no, no, let's not do that, just ask them why, like, okay, why are likes a really important part of this to you? And don't worry about it because you're not gonna get to them anyway, probably. So then once you have sort of the list, where do we start? What's the thing? Okay, so we're gonna build friending and we're gonna build status updates. What do we build first? So, status updates, let's go there. Okay, so when you implement status updates, you really want to, usually people are going to have a pretty good idea, okay, we're gonna add a table or we're gonna add another model that has status updates in it. Once you get that, like, dig a little deeper. How are you gonna put privacy controls in this thing? Are they just going to just add a field to the status update column or the status update row? And once they're there, well, what about granular privacy controls? What about I want my post to be visible to all of my friends except John because he's a jerk? And then once they have that, they've probably chosen some method of handling privacy which is not perfect. And so keep digging. What performance problem does that raise? What method of operating with that does that solve? So if they just have a bunch of fields on the end of the status updates that says who can see it and who can't, well, that's gonna make a really complicated query when we go to build the newsfeed. So friending is the same thing. It's a slightly different situation. But how do we implement friendship status? It's bi-directional. You've got some, do we put one row or two in that join table? How do we implement the friend requests and what happens when somebody rejects it? How do we add blocking? We can kind of go down that rabbit hole. And then once you've got both of these in, if the candidate has done really well, you can put them together and start talking, build, draw that system out on the whiteboard and see how it goes. So should you ever resort to a coding interview? And the answer is, yeah, sometimes. If your interviewer skills are lacking, this can be a temporary solution while people get up to speed using coding. Determining whether entry level candidates can program at all. So this is the Jeff Atwood coding horror versus Buzz use case. It actually works pretty well for that. And then when a very specific programming skill might determine seniority. So if you're kind of on the border between mid and senior and how good they are at Angular, really is the thing that determines whether they're across that line. Sometimes after the interview is over, you might ask them if they could do a short coding example. Another big one is that if your organization mandates pair programming every day, you might wanna see if they can do that or if they're gonna like it or hate it. And then finally, the big one is, now that I've convinced you all that pair programming or that coding interviews are no good anymore, you'll probably wanna do them for a little bit longer as you transition to this method so you can calibrate your situation. I really think that we'll come to see you over the next couple of years coding interviews as an anti-pattern. I hope that I've convinced you of that. I really appreciate your patience and your attention and thank you for coming. I'll be up here to answer questions. Thank you so much. Thank you.