 Yeah, both for a teacher and a teacher like that, or like a water painting. Shouldn't it be at least this kind of thing? Yeah, yeah. Sorry, you're safe. Alright, so we'll still have prizes next event. I don't want to close this up. Spain has ideas for prizes. There's a lot of books around the world, because we all know that it's a KCD. Um, not the programmers. Other options. Record your screen. I think we're good. Good evening everyone. Thank you for joining us here at Go To Supply. Thank you for Justin and Colin and John for kind of pushing me to give this talk. It's a topic that I think we've all had a little bit of experience with if you've worked in the tech industry. And it's just something that, something I felt like talking about. So first of all, I'm going to go over kind of an, kind of an overview of, I'm going to go over an overview of what some of the challenges we face and things like that. And we'll talk about a couple of ideas to help solve some of those challenges. So first of all, it's pretty universal. Job interviews are a stressful experience regardless of what industry you're in. If you're in finance, if you're in accounting, if you're, um, if you're, I don't know, if you're a rocket scientist, they're universally stressful. In fact, and so not only that, being under stress, like I am right now giving this talk, negatively impacts performance on skill-based assessments and anything where there is kind of a cognitive element to it, much like coding interviews. In fact, there's a NASA study where, where test subjects were shown to significantly slow the evaluation of math problems when placed under some form of, when placed under some form of stress. This is my opinion that coding challenges and technical interviews are not accurate nor are they unbiased. And the reason for that is because a lot of them either assess for skills that are either not required or in some cases are completely orthogonal to the, to the job that you're being hired for. Imposter syndrome is a very real thing. We're going to talk about it a little bit later, but the short version is it's this irrational fear that one day you, you may be uncovered as a fraud or, or something along those lines. But before we talk about that, let's talk about coding interviews. First of all, for those of you that aren't in tech, we should define what exactly a code interview is. A code, a coding interview is essentially a time skill assessment where a job candidate is, is asked to write code in real time either on a whiteboard or through a shared editor or through some other kind of automated tool of some sort. In some cases, the job candidate may even be remote and this could, and they could be communicating by Google Hangout, Skype, phone call, what happened? Now, the second question that many people outside of the tech industry might wonder is, why are these done? The answer is pretty simple. Companies want to make sure that they're hiring qualified candidates. I mean, if you drove here today, you likely have a driver's license and in order to get a driver's license, you have to pass a road test. So, you know, it's, it's, so companies apply the same logic to coding interviews. However, coding interviews are hard work for everyone involved, not just, not just for the candidate, but also for the interviewer as well. In fact, many developers find live coding to be a very stressful experience for a variety of reasons. Candidates are exhaustively interviewed, sometimes about orthogonal topics. For example, if you're, if you're aspiring to be a full-stack engineer of sorts, a job interviewer may start asking you questions about, about the document object model or DOM. And more specifically, they might want to ask like internals and things like that. And the problem is, while having this knowledge is good, it's not necessarily a good indicator of that person's, of that person's capability or their performance on the job. It only demonstrates whether or not they were curious enough to look or that they've had some experience working with it. In my opinion, live coding is inherently flawed. What does it actually measure? Like, does it measure my ability to solve a problem in real time? It's my opinion, live coding is a distinct skill set separate from, separate from programming in and of itself. That, and it's a skill set that some programmers may not have. It doesn't mean that they're bad programmers. It just means they're bad at live coding. And that's okay because really, who live codes all day anyway. So, so given the proliferation of, of, of technical interviews and some of the background that some people have with them, a lot of people have put out books and online guides and things like that to help, to help people prepare for coding interviews. While these are helpful, it's a sign that there's a problem. In conjunction with the growing industry of boot camps, of coding boot camps, I'm sure we've all seen a whole bunch of these. There's, I've also noticed there is a parallel industry of, there's a parallel industry revolving around technical interview preparation and even a some degree technical candidate assessment. Maybe this is why tech interview practices haven't changed much. Let's, let's talk a little bit about like what the world would be like if other industries hired the same way that tech does. For example, auto mechanics. Let's say that you've been wrenching on cars all your life as a hobby and you recently decided that, okay, I'm going to get a job as a mechanic. So you take your interview, you walk down the street to Joe's garage. You go inside and you speak with the manager. The manager brings you into a room where this is waiting for you. And your interviewer says, here are all the parts to assemble a BMW V8 engine. It must turn over, start and run. You will not be given any tools. You are not permitted to use any reference guides. You have one hour. Good luck. Meanwhile, what your actual day-to-day job is, is you're changing oil, you're rotating tires, you're checking people's breaks. Never in an ordinary work day would you be expected to put that together and much less in less than one hour. I mean, I don't even know what the torque specs for all those bolts are. And I'm just an enthusiast. Meanwhile, anecdotally, Mercedes-Benz AMG technicians, it takes a single technician a full eight hours to assemble one engine and that is with complete reference guides and tooling. Another example is a plumber. Let's say that you decide one day that you're going to become a plumber and I'm not sure what the process for becoming a plumber is like, so I'm not sure what kind of training or certifications are required there, but if it's like a lot of those professions, there's usually some kind of an apprenticeship component to it. But anyway, imagine if you walk into a plumber's office one day for a job interview and you're brought into the break room where that's waiting for you. And they say to you, well, this sink is clogged. You can't snake the drain. You can't open the trap. Here's a plunger. Good luck. I mean, just the fact that you don't even have the right tool to do the job or permitted to take the appropriate steps in this case shows that this is kind of flawed. What about locksmiths? Here's a paperclip. Go. You've got to open that in 60 seconds. Meanwhile, your day-to-day is copying keys, installing new locks. Never would you be expected to open a lock in 60 seconds for the paperclip. I mean, you would have the toolbox. Tax accounts. You have a client who needs a tax return in the next 30 minutes and is asking you to find all possible deductions. This book that's right here, which is the 2006 complete internal revenue code, you're not allowed to open that book. You will only be given their financial statements. Meanwhile, if you're in the real world, a tax accountant usually puts all that information into this fancy contraption called a computer that spits out a tax return in a couple of seconds. What do all of these interviews accomplish? Each one, without exception, puts the interview candidate under unnecessary stress. In the case of a tax accountant or mechanic, it causes careless mistakes that they might not otherwise make. People can become mentally blocked when they're under stress. I've certainly had that happen to me. And for people that may end up getting the job, they might think that, oh, man, I wasn't really that good at it. And they end up with imposter syndrome. We're going to talk about that a little more later. But first, let's talk about what these don't accomplish. They don't accurately assess people. They don't provide dependable and repeatable results. They don't verify that the candidate has the actual skills. Think of it like a street artist, one of these people that draws caricatures and everything else. It's more performance-orange than anything else. What do all of these professions have in common? They're all skilled professions. I mean, I most certainly could not be an auto-mechanic. I certainly couldn't be a plumber. But all the practitioners in that field have to have a lot of rigorous training, certification. And in some cases, like accountants, life's the same requirements. And usually, failure to perform under pressure in those fields usually, though not always, does not result in loss of life. Question is, is good performance under pressure really necessary for software development? Maybe. Maybe not. I don't know. But we can talk about some professions where good performance under pressure is absolutely required. Cave diving's a good one. Imagine if you are 500 feet away from your nearest air surface, you're underwater, and you need to do task loading, like running reels, running lights, and keeping track of references. PS, I'm certified to do this, but I realize this isn't necessarily an occupation. It's just an hobby. However, pilots are a definite example of a profession where good performance under pressure may be a requirement. Another example, surgeons. You better believe that. And nuclear reactor operators. What do these have in common? The jobs typically come with high cognitive load. I mean, consider a surgeon. I mean, consider keeping track of all those of your patient's vitals. In the case of a nuclear reactor operator, keeping track of a whole bunch of gauges and values of where they're supposed to be set, they all come with a low margin for error. For example, nuclear reactor operators, surgeons, they can sometimes be a stressful work environment. Consider an operating room. Alarm's going off at you or you're trying to, you know, you're sweating and trying to perform your operation. And in these professions, mistakes can cost lives, not just dollars. Does this describe what good software engineering jobs are like? I mean, not taking into account workplace culture or other organizational stresses. While software engineering does require a high cognitive load, mistakes do commonly occur. I mean, how many of us have checked in a bug or broken a code base? However, there's one thing that's different. We've gotten very good at catching the rigorous testing, continuous learning, and self-improvement. Performing under pressure, it's pretty well known that cognitive tasks such as programming are more difficult to perform under pressure. Recollection of important details get lost. People make careless mistakes. It doesn't correlate well with what we actually do. Can software defects be fatal? Yes, believe it or not, they can. Consider the Therax-25 radiation therapy machine or the Ariane 5 rocket disaster. Both incidents were caused by software defects. However, in response, several industry standards were adopted. However, none of those standards make any reference to developer skill levels and are more oriented toward organizational control and processes and procedures. However, let's be honest. If I check in a code that brings down your Ruby on Rails app, is somebody gonna die? I don't think so. So, with that in mind, let's talk about coding assessments. But first, what is a coding assessment? Usually, it is a take-home test that a job candidate will complete on their own time. Some companies call them take-home tests, some people call them coding challenges, some people call them coding exercises, whatever. It's all the same thing. Are they a good idea? Yes, they are. And the reason why is because they provide a way for a candidate to evaluate the problem on their own with their own knowledge and expertise. They allow a candidate to think creatively and independently to use the workflow and the tools that they're most comfortable with. Now, I can hear some people saying, well, what if the candidate didn't actually do the work? What if they got their friend into it? Well, you can ask them questions about it. However, scope and complexity are sometimes a problem. For example, someone wants a very simple exercise, such as taking put data A, mutate it into format B. Others are more involved. Yet others place the unrealistic constraint of expecting a candidate to implement both a complete front-end and back-end in 48 hours in whatever frameworks they choose. Participation bias is introduced. For example, they favor people who usually don't have a lot of out-of-work obligations. So people who don't have children or are taking care of other people. Also, consider the case where I may be already employed and I have 24 hours to complete this. That's basically asking me to take a vacation day or work over my weekend. Sometimes the requirements for these things can be really, really, really unclear. And even in the event that they are clear, some of them, especially when sample data isn't given, provide the opportunity for additional bias decrepitment. For example, did it solve the problem? Yes, was it the most elegant solution? You use short variable names, whatever. Now, I've talked a little bit about why or about how things currently are. And what I haven't talked about yet is the rationale behind why we should even think about these things. And here's why. Mental health is a growing problem in the tech industry. The fact that sites like DevCrest, Barnard IO, DevRant, and the Mental Health channel on code and supply slack indicate this is a growing problem. And also, given that in recent years, the software industry has grown exponentially. This problem will only continue. It will only continue to grow until it's addressed further. I found a study that was conducted in 1990. So please take this with a very big grain of salt that suggests that 32 software engineers that they studied suffer from depression. Now, I can't entirely trust this study, but it is a statistic worth throwing out if only for the fact to encourage additional discussion. So let's talk a little bit about what imposter syndrome is. Wikipedia defines it as a psychological phenomenon in which people are unable to internalize their accomplishments. The thing is, imposter syndrome is not an actual psychological diagnosis. It is not present in either DSM-4 or 5 or whatever the current revision is. While it's not an actual medical condition, it can exacerbate pre-existing mental health conditions. Another interesting statistic I found was that according to the International Journal of Behavioral Science, 70% of millennials have experienced imposter syndrome in one form or another. Now, consider when you look at a lot of tech companies, a lot of the folks that they're hiring are millennials. In a mentally healthy person, imposter syndrome can just be a temporary annoyance. Consider when you first start a new job. Consider how for the first three to six months, you feel like, oh, I don't know what I'm doing. I don't know that I quite fit in, whatever. If you're mentally healthy, this subsides, and in some cases can be a motivator for you to learn. If you're already, if you're not mentally healthy, in some cases, you know, if you already suffer from depression or anxiety, you know, that anxiety can get in your way in the middle of coding. And if you're depressed afterwards, you may feel like your soul is crushed. If you're stuck in a toxic work environment and you can't, and you're having trouble passing coding interviews, it can become a snowball effect. And in some cases, this feeling of, you know, feeling like you don't know what you're doing, whatever, whatever, can act as a very powerful motivator for you to burn yourself out. Studying, just relentless study. While studying on its own is a good thing, studying until the point that you burn out and are dissatisfied with your job and career choice is a problem. So it's one of these things, like, if you're in a job interview and you're given a task to perform, you subconsciously compare yourself against not just your interviewer or your interview panel, but also the company and other people you know that work in that field. Comparison and judgment can be very distressing. In fact, one of the best pieces of mental health advice is learning to be comfortable with who you are and more importantly, to stop comparing yourself to others. It's easier said than done. It really is. And it's even harder to do after a particularly grueling tech interview. In fact, this is what tech interviews are all about, comparing your technical abilities to the person that's sitting next to you. This says it all. We are our own worst enemies. But, you know, I can't talk about all of this if we don't discuss a couple of ways that we might be able to fix things. So for example, let's start and we'll talk about what companies can do. So first of all, admit it's broken. A lot of companies still do whiteboard interviews and put people through a ridiculous battery of interviews, coding assignments, whatever. You know, there's a bunch of other organizations I'm aware of that have bizarre requirements and assessments for potential recruits. They're called college fraternities. Companies should be transparent with their candidates. They should explain how they're going to be evaluated. They should explain what they're looking for and how they will be assessed. Not only that, they should also be transparent about the time frame. For example, instead of saying, well, we'll get back to you next week. Instead, you should hear from us by the end of Thursday. If you haven't, please let us know. Companies should be respectful. It shouldn't have to be said. But really, if you're interviewing someone, they took time off from their job to come in and interview with you. You really should pay attention to them. I've heard stories. I've actually experienced it where an interviewer was sitting there on their smartphone the whole time. It's a really off-putting thing. Secondly, interviewers sometimes, interviewers, and I've certainly experienced quite a bit of this in the tech industry, sometimes have a condescending know-it-all attitude and it is incredibly off-putting regardless of whether or not you get the job. And lastly, if you do coding assessments, if you do take-home coding assessments, why don't you ask your interview candidate about it? Don't just throw it away. Companies should collect and monitor interview metrics and statistics. So for example, what do your interview questions actually measure? Compare that against what you're actually looking for. Are you keeping track of how many people get through how many people don't? But more importantly, are you keeping track of why? And last question there is, if you're not doing these things, why not? Applicant tracking systems, you know, there's a whole bunch of them that operate under a software as a service model. Get one and use it. Companies always need to close the loop. In this day and age, you'd be surprised how many companies don't just say, don't just say, sorry, you're not a good fit. Once again, as I said, with modern applicant tracking systems, there's really no excuse for this. There really isn't. Technical interviews should have same scope and complexity lines. For example, if you're hiring a web application engineer, why are you looking for Hadoop? Wouldn't it be better if they knew more about Ruby on Rails, Node.js, Django, or any other web application framework instead? If you're looking for a DevOps engineer, do they really need to have all the Unix system calls memorized? Do you really, does your job really require you to memorize the entire, the entire Donald Knuth series of the art of computer programming? Really, does it? Technical interviews should not include live coding from scratch. I know that this is probably going to be a sore point with a lot of people. Think about it this way. If you're hiring an artist or a graphic designer, you want to see what their work style is. You want to see if their style, if their work is something you might be interested in. So what do you do? You ask to see their portfolio or a sample of their work, but you don't ask them to create new work on the spot. Why? Because it's certainly known that, so say for an illustration or something along those lines, it takes quite a bit of work and quite a bit of time to do that and to get it just the way that you want it. Programming is no different. Programming is exactly the same. You have to wrap your head around the problem and figure out a way to make it work. And in a live coding situation that it is incredibly hard to do that. It really is. Pair programming should actually be pair programming. If you're going to share an editor, it should be a collaborative effort. Don't just sit there and just ding the candidate. You should actually be programming with them. So for example, if you gave them a coding assessment, why not pair with them and help them make it better? At the worst, you're going to teach someone something who may not end up getting the job but who may end up going on to do better things because you actually taught them something. And mind you, it doesn't have to be their code. It could be an open source bug that their company doesn't depend on. It could be any number of things. But pair programming should actually be pair programming. Coding assessments need to have same time limits. As I said before, 24 to 48 hours is not enough time, especially for candidates who are presently employed. Like I said earlier, this assumes that a candidate will take at least one vacation day or give up their weekend to complete your coding assessment. What if they have a family? As I mentioned before, children that they have to help with homework and make dinner for and everything else. Sometimes people who are very talented, they end up having a lot of these to complete and they may decide, well, I'm not going to do that one. It's my opinion that if you're going to give me a coding assessment to complete, you as an interviewer should complete it as well. You don't necessarily have to complete it in order to get hired. But before you start talking with me about it, you should have completed it as well. The reason why is this brings familiarity with the problem into... It brings familiarity in. And so, not only that, how do you know whether your coding assessment is actually at the appropriate difficulty level or if it's even solvable? I remember I had one where they gave me sample data and nothing I could do could reproduce that sample data. I eventually sent them an email back and said what was happening here? And they were like, they took a look at it and they realized your code's correct. Our sample data's wrong. It does happen, believe it or not. More importantly, upon successful completion of the coding assessment, I should be able to see other people's answers, including my interviewers. They need to have clear and concise wording. Ask a non-technical person to review the wording or to see if the requirements are clear. If they're not clear, rewrite your coding challenges. Remember, even though programming is inherently very math-oriented, not everyone has a background in mathematics. As I mentioned earlier, they should include sample input and output data. If the object is to mutate data set A and output B, you should provide your sample input and what your expected output is. Good candidates will just use this as a guideline to get things started. Excellent candidates will use this to set up unit tests. Coding assessments and their solution should be open source. Too many companies hide behind non-disclosure agreements for interview candidates. Too many companies hide interview questions. Why? Because your interview question is better than everyone else's or because you just want to stress people out. The candidate should be able to, but not required, but not required to put their solution on GitHub or some other social coding platform instead of uploading it to Google Drive or any other kind of shared file sharing service. Interviewers should examine a candidate's prior work. If I'm an applicant somewhere, I likely have a resume, I likely have a LinkedIn, and I also likely have a GitHub that has a lot of commit history associated with it. With that in mind, why don't you ask them to select a piece of code from it that they can use as the basis for their prior programming interview? There is a bit more time involved there, but you could certainly do that. In fact, an even better practice might be if your company has their own GitHub account. Why not put your coding assessment there and the candidate can complete it and then submit it with their application just to save you the time? That way they can complete it on their own schedule. Candidates should be able to use the tools that they're the most familiar with. As I mentioned in the Plumber example, you're given a plunger when really what you needed was a wrench to open the trap. Candidates find coding assessment tools sometimes to be off-putting. I mean, how many event users do we have in here? I like my keybibings. Thank you very much. The important thing here is that while learning new tools is a good thing, a job interview is not the appropriate time or place to do it. If you expect me to write code for any portion of any kind of skill assessment, I should be able to use the tools that I'm the most familiar with. Automated coding assessments shouldn't be used. The reason why is because it's automatic whiteboard coding. Tools like code wars, codility, true ability, they often have an unrealistic time constraint associated with them. Problems are either purposely confusing or at least I find them in some cases to be purposely confusing. They offer no opportunity for clarification or even collaboration. In fact, Codility's FAQ specifically prohibits that and there's another question in there that says, well, what if I misunderstood the assignment? It's up to the recruiter. Those are two things that in the real world, I can just say, hey, I just want to make sure I'm on the same page with you. Candidates should be given detailed feedback about why they did not receive an offer. This is kind of a sore point. And the reason why is, and so companies are very opaque about it. Now, I realize there are laws in place so that in some cases you assume some kind of liability for telling someone why they may not necessarily be a good fit at your company. However, if you're assessing someone in a technical context and you have all these dimensions that you want to assess them, you have the data. Why not share it? I mean, wouldn't it be better if instead of saying you're not a good fit, you've got something more complete such as? Your coding skills were good. You certainly knew your data structures, but your analysis of algorithms needs more work is specifically space complexity analysis. Candidates in general should feel empowered to ask very candid questions about interview processes. So for example, I found this thing called the Joel test for interviews. And I'm just going to read out maybe one or two of these from here. So for example, one of which is, do you have a qualified technical person conduct a non-screening call to explain the hiring process? Do you commit and adhere to a realistic schedule? Do you allow candidates to take home assignments on their own, on their own time? Lastly, we need to make programming fun again. Most of us got into programming because we find it fun and rewarding. Like many of you, there is an inexplicable joy I get from watching lines of output scroll past as a computer does respond to my wishes and commands. If you want a real-world example of that, watch your unit test pass. That green is kind of exhilarating sometimes. However, you know, doing all this prep for a job interview, it makes it really not fun. We need to end the stigma of mental health. I should be able to say, I have a therapist appointment with no shame. The same shame that I might say, oh, I can't go out to dinner. I have a dentist appointment later. And remember, each and every one of you are awesome. And each and every one of you deserves a fulfilling tech career. Thank you.