 Hi, my name is Ted Stressenreuter. I'm the CTO at SecretSource and today I want to talk to you a little bit about code reviews and pair programming as part of our recruitment process. At SecretSource, when you, as part of our recruitment process, you may be asked to participate in a pair programming exercise either as a candidate or as an employee. The purpose of this exercise is to establish what level a candidate is on on our professional ladder. That's it. If you are unfamiliar with our professional ladder, please consult some of my prior videos where I explain what the professional ladder is. Basically, it's a matrix that allows us to identify what level you are on an objective scale. The pair programming exercise combined with the professional ladder turns a subjective exercise into an objective one. In the ladder, we quantify knowledge and skill. During the pair programming exercise, we're looking for evidence of behaviors and knowledge that represent a specific level on the ladder. Note that this is slightly different from a test. In a test, normally, there is a pass-fail element that is not present in our pair in our pair programming exercise. Basically, in the pair programming, we're simply looking to see where on the scale are you. Are you at level eight? Are you at level nine? At level 13, we're literally trying to identify where candidates fall on a scale from eight to 13. And by the way, eight to 13 is a mind hack. We didn't want to use level zero and then only go up to level five. We wanted it to be something that you don't necessarily associate with better or worse. So don't pay too much attention to the actual numbers. At each level of the ladder, we provide examples of behaviors that represent the level in the given area. For example, for ownership, we provide examples of poor or low ownership, for example. I just do what I'm told. You don't take any responsibility for the work. What follows is a detailed example of how we analyze a sample HTML snippet. When searching for evidence, we are looking for the most salient features that represent an example of any given level. For example, right now, we're going to look at the HTML part. This comes directly from our professional ladder. These are the hints or guides on how we analyze what we consider to be levels nine, ten, eleven, twelve, thirteen with regards to HTML. So basically in our professional ladder for the technical parts, we have four levels, eight, nine and ten, eleven and twelve, and then thirteen and up. These are guidelines. They're not written in stone or on a blockchain, just a Google sheet. The following slides provide some examples. So let's have a look. So here on this particular slide, you can see that the HTML is not very semantic. It's an incorrect or potentially incorrect use of the span element. It should probably be a label element. It's missing a form element. This is supposed to be a form. And we see this a lot in people who are JavaScript developers who have never done anything outside of a JavaScript library. They don't necessarily know that a form element actually even exists or that it should be used and why. And I can't state why. This is an example of what I would say is a level eight HTML. Level nine, ten, you see code that is somewhat semantic but not necessarily HTML five. There's still some reliance on JavaScript for things that are supported natively across all major browsers and have been for quite a while. There's perhaps an overuse of wrapper elements. Wrapper elements aren't necessary if you have something other than div elements because you can target different elements in different contexts using specificity and CSS. There's an unnecessary use of data attributes, for example. It could just be a hidden field. Levels 11 through 12, you start to see mostly semantic and valid HTML, no incorrect attributes. Everything is closed properly if required. There's some use of wrapper classes, but it's less and it's only for specific cases. And there's even some use of advanced attributes such as placeholder and autocomplete, unrequired. At level 13 and above, you see highly semantic HTML with what I like to call an intelligent embedding of elements. So the input element wrapped inside of a label element means you don't have to use the for attribute because it's obvious what it's for, right? I've never actually tried embedding multiple input elements inside of a label, but it might be an interesting experiment for you to try at home. Also, there's a minimal amount of HTML and that's a good thing. We want to keep code, we want to write as little code as possible when we're programming. The technical ladder covers about 70 topics on web development and general programming. Topics include planning, DevOps, object-oriented programming, HTTP, CSS, common programming language features such as functions, constants, variables, TDD, debugging, clean code, logging, MVC, and most recently some functional programming concepts in addition to several dozen other topics. This document exists so we all mean the same thing when we say the candidate has level 11 knowledge of unit testing and level 13 knowledge of CSS. Without this document, if I ask one of my co-workers what level does this candidate have, they may say, oh, they're great at JavaScript because during their interview, they communicated and they connected on a specific topics, maybe testing, for example. But I could ask a different person about the same candidate and they say, oh, they don't know anything about high-order functions or about using context properly. And both of them may be true and may be accurate, but you get different results. So with our professional ladder and with this using it properly, we get an objective result and we're all speaking the same language. While doing the code review and pair programming activity, it is crucial that you also make note of evidence from the other four areas of the ladder. For example, if the README doesn't exist, then that would be an example of poor communication, level 8 or 9. If the README does exist but is outdated or missing some critical details about how to run the project, for example, then that would be a 9 or a 10. If it is mostly up to date and includes working examples of the code or how to use it, that could be an 11 or 12. If the README also provides links to supporting documentation, contribution guidelines, links to supporting libraries that are heavily used, etc. And if it is well written, free of typos and thoughtfully presented using Markdown, for example, that would be a 13 in my book. Note that just because a candidate writes a lot, it does not mean they write well. Quantity is not equal to quality. You need to think about your audience. Although I use the README in my example right here, there are ample opportunities for finding evidence in these other areas. Does the candidate use Slack or email to ask questions or provide updates while working on the project? Are the commit messages clear and well written? Is there a development plan in the README or elsewhere that explains what order things will be done in? Are the estimates provided? Are they even close to accurate? And if they're not accurate, do they explain why? There's lots of opportunities during the whole interview process for us to understand where you fall on a ladder, on our ladder, where a candidate falls on a ladder. Often candidates will talk openly about what they learned while doing the project. It's important to engage in those conversations and sometimes I like to even ask, did you run any formal experiments? So instead of just trial and error while I tried this then I tried that, did you hypothesize about what the outcome should be given a specific input? Basically, are you doing test-driven development? Leadership can be a little bit difficult to judge during the interview process because there's not much opportunity. However, a good leader will always have a clear understanding of success and keep the moment's goals and objectives in the forefront of the conversation. Make a note, it's important to make a note of when the candidate steers the conversation, whether steering away from something that's not relevant or steering towards something that is. This is, in general, our hiring process as it stands today. The parts in green are the parts that involve the pair programming exercise. And you'll notice that as soon as the candidate passes our technical screen and is invited to the pair programming activity, we set the date for the pair programming activity immediately. Even though the candidate hasn't done the project and we haven't reviewed their code yet, and we do that to sort of put a stake in the sand and get the ball rolling. We can always reschedule if we have to. If the candidate for some reason can't finish the project in time because we're all busy people, many of us have full-time jobs and family. And it's difficult to find the time to finish a project in the amount of time that's scheduled. But we'd like to have that date set so that everybody knows what's coming up. Our internal developers, the code reviewers, have time to schedule set-aside time for doing the code review. They can be active in the Slack channel communicating with the candidates if they have questions. That sort of thing. The key takeaway here is that pair programming session is set as soon as the candidate is invited to do the test. So as soon as the candidate notifies us that they are finished with their project, we analyze their code, storing our notes in a document template that I've designed for doing code review for recruitment. This should not be an unfamiliar experience for reviewers as it is very similar to doing a code review for a PR. Once the programming session is complete, once the pair programming session is complete, the reviewers should meet briefly to finalize their report and findings. A final communication indicating the candidate's level and ideally an indication of whether or not the reviewers want to work with the candidate should be sent to the hiring manager and a copy to HR for our records. And it's really, I just want to say one more time that it's important to understand that when you're doing the code review and the pair programming session, you're not asking yourself, do I like this person? You're asking yourself where on our professional ladder does this person fall in the different areas and what evidence do they have they provided us with in this whole process? And that is basically our code review and pair programming process in recruitment at Secret Source. Thank you very much.